Recently (ok november of last year; so relatively recently ;) i had a linode that was infected by malware. i was first alerted by the excessive amount of egress traffic being used on my account; however after missing the email notification(s), and therefore exhausting my monthly transfer limit, linode’s NOC (SOC?) shutdown my instance. no fault to them; that’s exactly what i would’ve done as well. after contacting support and having a very helpful, responsive, and friendly conversation i was able to gain access to the halted instance.
first, i used this incident as an opportunity to (finally!) learn how to migrate a linode to a vm, but more generically dd via ssh. My notes are as follows:
ssh [email protected] "dd if=/dev/xvda" | dd of=/dev/sda
grub2-install --boot-directory=/boot /dev/sda --force
This allowed me to get a local infected vm for further research. Luckily (for me ;) the malcode persisted and I immediately started seeing a large stream of outbound traffic (don’t forget to use bridged networking in your test lab/cyber range [or not as the case may be ;]). Further, it seemed to be a different host from before which told me it was highly likely the malcode had checked into C2.
ok, so the hunt is on to find out what process. a quick look through netstat showed that ls was sending udp traffic. clearly; not normal. further, by inspecting /proc i quickly found the malcode in /boot. after copying what looked like a mktemp(3) filename off and checking it on virustotal.com i found a couple of shocking revelations…which brings me to the second lesson/teaching moment for this exercise.
anti-virus software is not only ALIVE AND WELL; there is very little AV that actually catches ITW malcode! ok, so this isn’t a shocking revelation in and of itself; nor is it surprising which vendors caught this trojan: basically the same players we’ve all seen recommended time and time again however, the surprise (for me at least) came ~2mo later when i re-analyzed
so the real story here is that only ~26% of vendors caught a known trojan 2+ months after it was publicly disclosed to exist! further, some of them are just re-bundles of the original 6 that have been “tuned” for performance.
my take away’s from all this:
–183r space cowboy
1: ok so i didn’t actually test this cli. instead i followed their docs (ref: https://www.linode.com/docs/migrate-to-linode/disk-images/copying-a-disk-image-over-ssh) copied to an image file first, (WHICH TOOK FOREVER!) and then i had to copy (again!) all 24GB to the new vm created in step1. hopefully the above works and saves me (and you!) some time in the future :) ↩
2: I will continue to use ESET as it has a crazy low system utilization (Kaspersky at similar settings just owned my current and previous macbook pros) ↩
3: Further, Kaspersky still doesn’t handle scanning exceptions properly; where ESET NOD32 works exactly as I would expect.↩
4: Moreover, ESET claims to have never (?) missed an ITW threat; which as seen above is also my experience. While, Avast is a good free product (that works on MacOSX and Windows) as is Sophos, (free only for MacOSX)↩
5: APT? http://blog.malwaremustdie.org/2014/09/mmd-0028-2014-fuzzy-reversing-new-china.html↩
6: I like to run Sophos’s excellent and free unified threat mangement system (formerly astaro security gateway [http://www.sophos.com/en-us/products/free-tools/sophos-utm-home-edition.aspx] as my core router and therefore prefer a more robust endpoint solution; see previous comments.↩
7: I use AlienVault products as they are Open Source based but also offer commercial support and improvements.↩