The dangers of stealthy malware

Some malware attacks are so blatant you can't miss the fact that you've been victimised. Ransomware programs lock out all access to your computer until you pay to have it unlocked. Social media hijackers post bizarre status updates on your social media pages, infecting anybody who clicks their poisoned links. Adware programs litter your desktop with popup ads even when no browser is open. Yes, those are all quite annoying, but since you know there's a problem you can work on finding an antivirus solution.

A totally invisible malware infestation can be much more dangerous. If your antivirus doesn't "see" it and you don't notice any untoward behaviour, the malware is free to track your online banking activities or use your computing power for nefarious purposes. How do they stay invisible? Here are four ways malware can hide from you, followed by some ideas for seeing the un-seeable.

Operating system subversion

We take it for granted that Windows Explorer can list all of our photos, documents, and other files, but a lot goes on behind the scenes to make that happen. A software driver communicates with the physical hard drive to get the bits and bytes, and the file system interprets those bits and bytes into files and folders for the operating system. When a program needs to obtain a list of files or folders, it queries the operating system. In truth, any program would be free to query the file system directly, or even communicate directly with the hardware, but it's vastly easier to just call on the OS.

Rootkit technology lets a malicious program effectively erase itself from view by intercepting those calls to the operating system. When a program asks for a list of files in a certain location, the rootkit passes that request to Windows, then deletes all reference to its own files before returning the list. An antivirus that relies strictly on Windows for information about what files are present will never see the rootkit. Some rootkits apply similar trickery to hide their Registry settings.

No-file malware

A typical antivirus scans all files on disk, checking to make sure none are malicious, and also scans each file before allowing it to execute. But what if there is no file? Ten years ago the slammer worm wreaked havoc on networks worldwide. It propagated directly in memory, using a buffer overrun attack to execute arbitrary code, and never wrote a file to disk.

More recently, Kaspersky researchers reported a no-file Java infection attacking visitors to Russian news sites. Propagated through banner ads, the exploit injected code directly into an essential Java process. If it succeeded in turning off User Account Control, it would contact its command and control server for instructions on what to do next. Think of it as the fellow in a bank heist who crawls in through the ventilation ducts and turns off the security system for the rest of the crew. According to Kaspersky, one common action at this point is to install the Lurk Trojan.

Malware that's strictly in memory can be purged simply by restarting the computer. That, in part, is how they managed to take down Slammer back in the day. But if you don't know there's a problem, you won't know that you need to reboot.

Return Oriented Programming

All three finalists in Microsoft's BlueHat Prize security research contest involved dealing with Return Oriented Programming, or ROP. An attack that uses ROP is insidious, because it doesn't install executable code, not as such. Rather, it finds the instructions it wants within other programs, even parts of the operating system.

Specifically, a ROP attack looks for blocks of code (called "gadgets" by the experts) that both perform some useful function and end with a RET (return) instruction. When the CPU hits that instruction, it returns control to the calling process, in this case the ROP malware, which launches the next scrounged block of code, perhaps from a different program. That big list of gadget addresses is just data, so detecting ROP-based malware is tough.

Frankenstein's malware

At last year's Usenix WOOT (Workshop on Offensive Technologies) conference, a pair of researchers from the University of Texas at Dallas presented an idea similar to Return Oriented Programming. In a paper titled "Frankenstein: Stitching Malware from Benign Binaries," they described a technique for creating hard-to-detect malware by piecing together chunks of code from known and trusted programs.

"By composing the new binary entirely out of byte sequences common to benign-classed binaries," the paper explains, "the resulting mutants are less likely to match signatures that include both whitelisting and blacklisting of binary features." This technique is much more flexible than ROP, because it can incorporate any chunk of code, not just a chunk that ends with the all-important RET instruction.

How to see the invisible

The good thing is you can get help to detect these sneaky malicious programs. For example, antivirus programs can detect rootkits in several ways. One slow but simple method involves taking an audit of all files on disk as reported by Windows, taking another audit by querying the file system directly, and looking for discrepancies. And since rootkits specifically subvert Windows, an antivirus that boots into a non-Windows OS won't be fooled.

A memory-only, no-file threat will succumb to antivirus protection that keeps track of active processes, or blocks its attack vector. Your security software might block access to the infected website serving up that threat, or block its injection technique.

The Frankenstein technique might well fool a strictly signature-based antivirus, but modern security tools go beyond signatures. If the patchwork malware actually does something malicious, a behaviour-based scanner will probably find it. And since it's never been seen anywhere before, a system like Symantec's Norton File Insight that takes prevalence into account will flag it as a dangerous anomaly.

As for mitigating Return Oriented Programming attacks, well, that's a tough one, but a lot of brainpower has been devoted to solving it. And economic power, too – Microsoft awarded a quarter of a million dollars to top researchers working on this problem. Also, because they rely so heavily on the presence of particular valid programs, ROP attacks are more likely to be used against specific targets, not in a widespread malware campaign. Your home computer is probably safe; your office PC, not so much.