snf_engine.xml

<node/> - <gbudb/> - <training/>

<drilldown/>

The drilldown feature allows GBUdb to automatically learn to ignore IPs as infrastructure by matching message source candidates in received headers with specific patterns.

The <drilldown> section of the configuration file contains a list of pattern definitions that describe received headers that will contain IPs that should be ignored. Each pattern is represented by a <received/> entry.

Ignoring IP Blocks

The GBUdbIgnoreList.txt file does not understand CIDR or wildcards. That's because GBUdb doesn't understand them either - each GBUdb record represents exactly one IP address.

If you want to ignore an entire block of IPs without making a specific entry for each one then an efficient way to do that is to add a drilldown entry using a partial IP pattern for the block of IPs you want to map as infrastructure. For example, if you want to ignore any message sources in the block 12.34.56.0/24 then you could create a drilldown entry like the following:

<received ordinal='0' find='[12.34.56.' /> 

The result would be that whenever a server delivers a message from that IP block, the top (ordinal 0) received header would contain that pattern and the IP from that received header would be automatically be added to the GBUdb with the Ignore flag.

Note that if the pattern is found in a different Received header nothing special will happen. This way forged Received headers containing the pattern won't have any effect (presuming the header directive is coded properly).

Ignoring Friendly Servers by Reverse DNS

Large ISPs with multiple IP blocks full of servers can be a problem. Most of them spew as much spam as legitimate email becuse their accounts and end users are either compromised or are configured to forward messages to other accounts (perhaps accounts on your system?). The result of such forwarding, for example, is that any spam that arrives at the large ISP mailbox is retransmitted by that ISPs servers to the new mailbox.

If a reputation is generated for the IPs of the large ISPs servers then that reputation will represent the averaged behavior of all of the message sources that have to be handled by that ISP. These days such an averaged behavior metric is only slightly better than that of a spam bot!

It would be nice to skip past all of the ISPs servers in the received headers, but those ISPs seldom (more like never) publish the IPs of their servers and even if they did it would be a big management task keeping up with the constant changes.

Drilldown provides a mechanism for clearing up this mess by allowing SNF to learn to see past the large ISPs servers to the original message source. Most of the time, the large ISPs PTR records will contain their domain or something very predictable that will appear in the Received headers of the messages they send. Using drilldown header directives you can automatically train SNF to see past these servers and drill down to the original source for the message. This gives your system more accurate statistics, so that more messages can be blocked based on these IPs without penalizing the ISPs servers.

Note: Some ISPs don't provide received headers for the original sender's IP. Thankfully some of those ISPs do provide the IP in another header. We have a <source/> feature for those cases.

Suppose some large ISP uses the domain mixed-source.net, then you might see received headers similar to:

Received: from out56.mixed-source.net [12.34.56.78] by my0wn1.bestfilterever.net
    (1.2.3.4 / 5.6.7.8) so forth and so on;
Received: from inside34.mixed-source.net [210.1.2.34] by outside56.mixed-source.net
    (and so forth) and so on;
Received: from border124.mixed-source.net [210.1.2.124] by inside34.mixed-source.net
    (and so forth) and so on;
Received: from ugly-spambot-customer.dyn-dsl123.eviltown.cpe9.example.com [99.88.77.66] by
    border124.mixed-source.net (and so forth) and so on;

Then your <drilldown> section might look like this:

<drilldown>
    <received ordinal='0' find='.mixed-source.net [' />
    <received ordinal='1' find='.mixed-source.net [210.' />
    <received ordinal='2' find='.mixed-source.net [210.' />
</drilldown>

The top received header (ordinal 0) created by your system would match the first drilldown header directive and so 12.34.57.78 would be added to GBUdb with the Ignore flag set. SNF would keep looking.

The next received header (ordinal 1) created by mixed-source would match the second drilldown header directive and so 210.1.2.34 would be added to GBUdb with the Ignore flag set. SNF would keep looking.

The next received header (ordinal 2) created by mixed-source would match the third drilldown header directive and so 210.1.2.124 would be added to GBUdb with the Ignore flag set. SNF would keep looking.

Finally, the next received header would not match any header directives. The previous received headers have all been made ineligible as message sources. As a result the IP 99.88.77.66 will be treated as the source IP for this message.

The end result is that messages flowing through the mixed-source ISP will be evaluated based on their content and the original source (not the ISPs servers).

Please email support@armresearch.com with any questions.