This chapter is from the book
This chapter is from the book
This chapter is from the book
2.5 Camouflaging the HTML Body
HTML (Hypertext Markup Language) is a language that describes how text and objects should be displayed. HTML also controls the interaction between the user and the message. Most, but not all, email is formatted using HTML. This is a boon to users and businesses, which benefit from the ability to display attractive fonts and images as part of an email message. But it is also a boon to spam email senders because HTML-capable mail programs are liberal in what they accept, and that allows spam email to be easily disguised.
In this section we first describe the use of HTML comments as a means to disguise content. Then we describe two forms of encoding often used to disguise content: character-entity encoding and URL encoding.
2.5.1 HTML Comments
Consider this HTML comment:
<!-- this is a comment -->
An HTML comment begins with the four-character sequence <!-- and ends with the three-character sequence -->. Comments may span multiple lines and may enclose and hide arbitrary text.
Early in spam history, spammers discovered that words could be broken up with HTML comments, making those words appear normally when displayed. Here's an example:
V<!--THYME-->ia<!--ONION-->gr<!--ALMOND-->a
Here, the word "Viagra" is hidden among the names of various foods. But because the comments are ignored by HTML-capable mail programs, this text is displayed like this:
Viagra
More recently, spammers have discovered that any unknown HTML keyword in angle brackets acts just like a comment. [6] Thus the comment technique has been modified by spammers and now looks like this:
V<xyxxx>ia<ftmmwe>gr</gvwquu>a
Clearly, to eliminate comments from HTML you must now possess a full understanding of valid HTML keywords. For example:
<font> Yes, a keyword
<fonts> No, not a keyword, but instead a comment
Similarly, spam-screening software needs to know that <p>, <b>, and <i>, for example, are HTML keywords, but <q>, <t>, and <c> are not.
To make things more complicated, any words that begin with an ! or a ? and any non-HTML keywords that begin with a / character are also treated as comments:
<!xxx> Also a comment
<?xxx> Also a comment
</xxx> Also a comment
</font> A keyword, not a comment
Be aware, too, that intervening newlines may also be included in comments. Consider, for example, the following:
V<<OICE GIMM>>I A<<LIVE ABLO>>G R<<IVER ITUN>>A
Here, we show the effect of both unbalanced angle brackets and a newline between the matching angle brackets of the comment. When viewed by HTML-capable mail programs, the preceding lines will look like this:
V<>I A<>G R<>A
In short, spammers can play many tricks to disguise HTML. Newlines may be included in comments. Extra angle brackets may become part of revealed messages.
As an aside, one trick that may be useful when you attempt to match angle brackets is to scan from the left to the right and look only ahead, never back. To illustrate, examine the following HTML fragment, in which we have indicated matching angle brackets using bold font:
<xxx<yyy<zzz<>www> <aaa<bbb>ccc>ddd>eee>
Reading from left to right, note that the first left angle bracket starts the xxx comment. The following right angle bracket (even if there are intervening left angle brackets) ends the comment, revealing www as clear text. The next left angle bracket (that is not part of the comment) begins the aaa comment and ends just before ddd. The result, when viewed on HTML-capable mail programs, will look like this:
zzz> ccc>ddd>eee>
2.5.2 Character-Entity Encoding
Character-entity encoding provides the means to include special characters in HTML documents. It can be invoked in either of two forms. Both begin with an ampersand character and end with a semicolon character. In between are either a keyword, such as lt, or a literal # followed by a three-digit decimal value. For example:
< A Keyword that produces a < character.
< # and three digits produces a < character.
Without character-entity encoding, the following special characters (to list only two) would not be possible:
© Produces the © symbol.
Ÿ Produces the Ù symbol.
The HTML standard allows all characters, and not just special characters, to be encoded in this way and provides a handy way for spammers to hide otherwise obvious information. Consider this example:
<a href="mailto:bob@exam ple.com">
After decoding, the obscured email address is revealed to be an easily recognized address:
<a href="mailto:bob@example.com">
Clearly, it is necessary to decode character entities before attempting to parse HTML for clues about spam. In Section 11.4 we show you the C language code you can use to decode character-entity-encoded text.
2.5.3 URL Encoding
A URL (Uniform Resource Locator) is a web reference. Every time you click on a link with your web browser, you are invoking a URL. The URL standard assigns special meanings to special characters. For example, a ? character is used to separate a web reference from its arguments:
www.example.com/cgi-bin/search?name=bob
To prevent confusion, such special characters, when not in their special role, must be URL-encoded. To URL-encode a character, first prefix it with a % character, and then convert the character itself into its two-digit hexadecimal ASCII value. [7] To illustrate, consider a search program whose name includes a ? character:
big?search.pl
To use this program name as part of a URL, the ? character must be converted into its hexadecimal value, 3F, which must be preceded by a % character:
www.example.com/cgi-bin/big%3Fsearch.pl?name=bob
But as we have shown, almost anything will be misused by spammers to hide message content. Consider, for example, the following obscured web reference:
href="%77%77%77%2E%65%78%61%6D%70%6C%65%2E%63%6F%6D"
Clearly, before you can screen a message for spam content, you may need to URL-decode all such URL-encoded expressions. [8] The example, when decoded, will look like the following:
href="www.example.com"
In section 11.5, we show you the C language code you can use to decode URL-encoded text.
2.5.4 The Order of Encoding
Character-entity decoding is performed by a web browser before it displays the text on the screen, but URL decoding is performed later, as a result of an HTML action, such as clicking on a web reference. Thus, web references may be first URL-encoded and then character-entity-encoded, and the result will be understood by HTML-capable mail programs. When decoding email to examine it for spam content, always character-entity-decode first and then URL-decode.