tl;dr

  1. Get a Kaspersky free trial.
  2. If it doesn’t slow down your system and you can live with broken exceptions (e.g. don’t scan your malware research directory) buy it immediately!
  3. Else, get a trial of ESET (and purchase it at the end of said trial :)
  4. If you’re too cheap to purchase AV (hey, no judgments :) then checkout Avast! or Sophos.

Recently (ok november of last year; so relatively recently ;) i had a linode that was infected by malware. i was first alerted by the excessive amount of egress traffic being used on my account; however after missing the email notification(s), and therefore exhausting my monthly transfer limit, linode’s NOC (SOC?) shutdown my instance. no fault to them; that’s exactly what i would’ve done as well. after contacting support and having a very helpful, responsive, and friendly conversation i was able to gain access to the halted instance.

this is when i learned two very important lessons

first, i used this incident as an opportunity to (finally!) learn how to migrate a linode to a vm, but more generically dd via ssh. My notes are as follows:

  1. new virtual machine based on your linode specs (i used vmware fusion pro to create a new vm with 1GB RAM, 24GB disk NOT split into multiple files; not sure this is needed, but AWS import made this a habit)
  2. boot into rescue disc (i used the latest kali iso)
  3. ssh [email protected] "dd if=/dev/xvda" | dd of=/dev/sda [1]
  4. mount /dev/sda /mnt
  5. for i in /dev /dev/pts /proc /sys /run; do mount -B $i /mnt$i; done
  6. chroot /mnt
  7. update /etc/resolv.conf (i used 8.8.8.8 and 8.8.4.4; which it turned out so did the malware! N.B. DNS is often overlooked, so be careful and don’t forget you have your local ISP; not to mention OpenDNS, etc.)
  8. yum install kernel grub2 (I was using CentOS; adjust for your package manager)
  9. grub2-mkconfig -o /boot/grub2/grub.cfg
  10. install grub grub2-install --boot-directory=/boot /dev/sda --force

This allowed me to get a local infected vm for further research. Luckily (for me ;) the malcode persisted and I immediately started seeing a large stream of outbound traffic (don’t forget to use bridged networking in your test lab/cyber range [or not as the case may be ;]). Further, it seemed to be a different host from before which told me it was highly likely the malcode had checked into C2.

ok, so the hunt is on to find out what process. a quick look through netstat showed that ls was sending udp traffic. clearly; not normal. further, by inspecting /proc i quickly found the malcode in /boot. after copying what looked like a mktemp(3) filename off and checking it on virustotal.com i found a couple of shocking revelations…which brings me to the second lesson/teaching moment for this exercise.

anti-virus software is not only ALIVE AND WELL; there is very little AV that actually catches ITW malcode! ok, so this isn’t a shocking revelation in and of itself; nor is it surprising which vendors caught this trojan: basically the same players we’ve all seen recommended time and time again itw-malware-virustotal-first however, the surprise (for me at least) came ~2mo later when i re-analyzed itw-malware-virustotal-second

so the real story here is that only ~26% of vendors caught a known trojan 2+ months after it was publicly disclosed to exist! further, some of them are just re-bundles of the original 6 that have been “tuned” for performance.

my take away’s from all this:

  1. dd via ssh — very cool!
  2. Kaspersky and ESET NOD32 are quantified as #1 and #2 in my book for endpoint protection.[2][3][4]
  3. The little guy is exposed, can, and will be attacked; even on modern, monitored platforms.[5]
  4. Castle theory is dead; long live the castle mentality!
  5. Defense in Depth is more relevant than ever. (Embrace a more modern protection methodology/approach such as immunology or policing.)[6]
  6. Use a SIEM framework/solution; if possible.[7]

–183r space cowboy

Footnotes

1: ok so i didn’t actually test this cli. instead i followed their docs (ref: https://www.linode.com/docs/migrate-to-linode/disk-images/copying-a-disk-image-over-ssh) copied to an image file first, (WHICH TOOK FOREVER!) and then i had to copy (again!) all 24GB to the new vm created in step1. hopefully the above works and saves me (and you!) some time in the future :) ↩

2: I will continue to use ESET as it has a crazy low system utilization (Kaspersky at similar settings just owned my current and previous macbook pros) ↩

3: Further, Kaspersky still doesn’t handle scanning exceptions properly; where ESET NOD32 works exactly as I would expect.↩

4: Moreover, ESET claims to have never (?) missed an ITW threat; which as seen above is also my experience. While, Avast is a good free product (that works on MacOSX and Windows) as is Sophos, (free only for MacOSX)↩

5: APT? http://blog.malwaremustdie.org/2014/09/mmd-0028-2014-fuzzy-reversing-new-china.html↩

6: I like to run Sophos’s excellent and free unified threat mangement system (formerly astaro security gateway [http://www.sophos.com/en-us/products/free-tools/sophos-utm-home-edition.aspx] as my core router and therefore prefer a more robust endpoint solution; see previous comments.↩

7: I use AlienVault products as they are Open Source based but also offer commercial support and improvements.↩