Message Sniffer.FAQ.SubmittingSpam
From ARM-KB
This page is no longer maintained and may contain information that is out of date. We have left this page in place to provide a historical reference and to provide assistance to folks who may have not yet upgraded from Version 2 to Version 3. EVERYONE should upgrade to the latest version if they have not done so already.
For the latest information covered on this page, please see the following pages on our web site: http://www.armresearch.com/support/articles/procedures/spamSubmissions.jsp
Home -> Message Sniffer -> FAQ -> Submitting Spam
Guidelines for Submitting Spam
To report Spam:
- Please send an email from your registered address or alias to spam@armresearch.com. ( NOTE: messages sent to support@ will be filtered out and not seen.)
- Forward spam one message at a time in it's original form (as if it were originally sent to us and not you) and/or include headers if possible. The following are a prioritized list of how to forward spam to us:
- The best way we can get spam submitted to us is from a redirected spamtrap. If you have one or more of these then please let us know and we'll set up a special collection point where you can redirect these messages.
- The second best thing you can do is to redirect the message to spam@.
- The next thing, and most common, is to simply forward the message to us at spam@ in as much the original form as possible. Some email clients forward messages more or less intact,... others tend to completely rewrite and/or "defang" messages that are forwarded. The former is preferred.
- The last option you have is to include it as an message attachment but only one message at a time please, and if possible please include the original headers. We don't always use these since most submissions to spam@ are suspect, but if we do dig deeper on a particular submission then the extra detail can be very useful.
- Please be sure to submit the messages to the correct address. If you do not, it is likely that your email will be filtered out.
SEND ONLY SPAM MESSAGES TO THIS ADDRESS, OTHER MESSAGES WILL BE IGNORED! WE *MAY* ATTEMPT TO UNSUBSCRIBE RECIPIENTS FROM MESSAGES THAT ARE RECEIVED AT OUR SPAM ADDRESS!
POP approach for submitting spam
For spam submissions, we are moving to a POP approach because it is more secure and more scalable. In general, spam can be redirected or forwarded to an account on your system and we can pop those messages from there. If you have any clean spamtraps that you would like to share with us then we would pull those messages from a different pop account. (We treat clean spamtraps differently than user submitted spam.)
Anyone can switch to this method at any time. Our current policy is to ask anyone who will be providing any significant submissions to use the pop3 method. We will probably always have the ability to accept spam at out spam@ submission address.
The only instructions are that if you want to provide us with spam samples from user submissions or from your own spamtraps, please:
- 1. Set up a pop3 account on your system that our Trapbots can access.
- 2. Provide us with the email address (login info), password, and the host name of the pop3 server.
- 3. Tell us how the messages arrive at this address. For example, if you can provide "clean" spamtrap data that is purely from harvested email addresses then we would like to keep that separate from spam that is submitted by users or perhaps derived in some other way.
When in doubt it is always ok to submit spam to our spam@armresearch.com address.
What spam do you want?
It is useful for us to get any spam that does not get caught by SNF. You never know when your submission might be the first one we've seen - your submission might be the one that gets the filters updated just a bit faster ;-)
[#top | Back to Top]]
How do you handle spam submissions?
If a message is submitted as spam, we consider the source and the content. We ignore questionable submissions, those that look like errors, and those that are likely to cause others false positives. (If you have a chronic local problem and want a specific black rule then contact support@).
Messages sent to spam@ are assumed to be spam, or potentially hostile. We filter all of these messages through our rulebase, then each message is pulled up on our "Spam Hud" for review. The SPHUD gives us extra information about the email and helps us avoid errors and capture important patterns more easily.
Each message that survives is manually reviewed and considered for filtering based on our current policies and procedures. Each source that we can identify may be treated differently based on the assumed risk or probability of error that we have while reviewing those messages. For example, we have some folks who seem to be subscribed to everything there is on the 'net - and they submit every message they get - so we must avoid many of their submissions. We have others who only send us definite spam and so we take a hard look at every message submitted.
Of the messages submitted to our spam@ address, there are few that we don't add to the core system. However, If there is any doubt about the "spamminess" of the message based on the current trends of our subscriber base then we will hold off adding the content to the core rule base.
There are some cases where we will attempt to unsubscribe an address from submitted spam. These are generally cases where spam has been submitted from an end-user on one of our client systems through an alias and where the unsubscribe process appears to be legitimate.
Our research teams continuoulsy go through all of the messages that reach us through our spam@ address and our spamtraps and evaluate the messages to see how they can be filtered, and in some cases *if* they should be filtered. In order for us to generate filtering rules for our standard database we must determine among other things that:
- The content is likely to be viewed as unwanted by the vast majority of our subscriber base. Ultimately our definition of "spam" is dependent on the preferences of our subscribers - not our own opinions.
- The message contains content that can be clearly filtered using our current pattern matching technology with a minimum risk of producing false positives.
It is common for one system to receive some spam that other reporting systems and our spam traps won't see. The more spam we have submitted to us the more comprehensive our filtering system will be. We continue to improve as our user base grows. All incoming spam is filtered through the current core rule base so that we can concentrate only on any new messages that aren't being tagged.
All of the heuristics we generate live in our rulebase forever and compete to be part of the active rulebase system. If you submit spam to our spam@ address for filtering it is very important for you to also submit your log files so that the rules we create will see some activity. When no activity is shown for a period of time rules can be made inactive to improve speed and reduce rule file sizes. Any rules generated for spam that is unique to your system will eventually become inactive if your log files are not submitted because our system will not be able to track the activity of the applicable rules.
Can I auto forward spam to you?
Yes. You can set up a script on your system to auto forward spam not caught by Message Sniffer.
How can I be sure that my spam submissions were received?
In general, if you've not received an error during delivery, we most certainly got your message... it may have even made it to the queue (if it wasn't already filtered by new rules).
One way to be sure we receive your spam is to create a pop3 box on your system for your spam submissions and provide us with the login data (email address (as login), password, FQDN of the pop3 server -- further instructions below). This way, if the mail in that box gets deleted you know one of our bots has pulled it in and added it to our queues.
Do you respond to spam submissions?
No. Since spam@ submissions are potentially hostile - we do not respond to any messages we find there - we only consider them to be potentially useful spam or hostile content. Any notes or comments that claim to be from our customers are explicitly ignored if they come in via the spam@ address.
Spam submissions are always treated as anonymous for security reasons and also because of the volume. During the day we are processing up to 5000 spam per hour. At those rates it is not practical to respond to each submission.
Advanced features near V4 (some time in the future) will allow us to handle some spam submissions specifically for a particular license ID--- so there are some plans for this later on. However, for the short and medium term all spam submissions will remain anonymous.
If you have a chronic spam for which you would like a local black rule added then you should send a zip'd copy to support@armresearch.com along with your requests. We will help you adjust your rulebase accordingly. For example, some relatively closed systems are able to use broad rules for certain character sets, file attachment types, or other features to eliminate messages they simply will never see in practice.
I am in ISP. What is the best way to forward user spam submission?
For spam submissions, the best thing to do is set up a pop3 box where we can have our robots pull messages. Since they are being submitted by your users we will classify that content as a "usertrap". If you have any clean spam traps (collecting spam only from harvested addresses) either now or in the future then we would need to use an alternate pop3 box for that.
- Once you've set up the mailbox please send us:
- The pop3 server address,
- The email address (login for the pop3 box),
- and the password.
We will have our robots collect the messages in this box, classify it, and add it to our queues for processing.
If I set up an account for you, can I have all of my users forward "spam" to that account?
You can do that, but you may want to review some of it - depending upon your system policy.
Users often have a habit of submitting anything they don't want anymore as spam --- that can include everything from personal emails containing jokes they don't like, to trade subscriptions, to direct advertising or even order confirmations from companies they do business with.
We classify sources that receive user submissions as "usertraps" for precisely this reason and we use different rules for processing the messages we receive there. We will skip these kinds of submissions when we see them, but I wanted to make sure you were aware of it ;-)
What is a virtual spamtrap?
What we do use from time to time are virtual spamtraps. In a virtual spamtrap scenario, you can submit spam that reached a very high (very low false positive) score but did not fail SNF. Generally this is done by copying the message to a pop3 account that can be polled by our bots.
We treat this kind of submission as if it were a usertrap, so we are very careful about what to code. The advantage to this methodology is that the detection of new spam is timely. Also, since we code rules speculatively for entire campaigns and message structures, SNF will often end up capturing instances of the campaign that did not score highly enough on other tests to get into the trap, and also preemptively captures future versions of the campaign that are not yet seen.
All that said, the biggest benefit of this kind of synthetic spamtrap is that if you now see something before we do, we will see it and code for it faster.
How does Message Sniffer deal with spam in foreign languages?
We attempt to create reliable rules no matter what language we see, however it is more difficult to do that in foreign languages. We are working on upgrades to our internal systems and procedures to address this. None the less there are usually things within these messages that we can tag and if we can identify them then we do create rules for those items.
I keep seeing a specific piece of spam. What's going on?
If we've seen it repeatedly and we haven't filtered it out then there is a reason --
- It could be that the spam cannot be safely filtered in the core rule group due to false positive reports by other systems. In that case we will work with you to come up with a suitable local black rule.
- It could be that the spam is of a type that constantly changes and that what you are actually seeing are the few that get by each time a new variant is created. Here again, your system policies may allow for more aggressive rules than we can use in the core rulebase, or you may have noticed something about it that we didn't notice. Either way we will help find a solution.
Examples of things that can be done locally but not in the core rulebase:
- Some systems have local black rules to capture foreign character sets.
- Some systems have local black rules to capture specific words or phrases (one notable example does not allow messages with "the F word" to go to or from one of it's domains.)
Forwarding Spam Samples From Outlook Clients
Outlook does a poor job of passing the headers of an original message when the FORWARD feature is used. One work around is to use:
- TOOLS->OPTIONS->EMAIL OPTIONS…
- Set the “WHEN FORWARDING A MESSAGE” value to “ATTACH ORIGNAL MESSAGE”
OR another strategy that might work for you is
If I've read this correctly, you could have folks who are interested in submitting install this and submit the messages to your local spam submission box. (The tool does not require that the submissions go to spamcop.)
Once you've checked over the submissions to make sure you agree (for example, making sure folks aren't using the spam submission process too liberally) then you could move the messages to a collection mailbox where our bots can come to collect it.
We are usually more interested in the content than the headers (though we do look there occasionally to backup our research) - so if the submission's content is unaltered then we're usually in good shape.
We are seeing an increase in these days. We are already forwarding them to spam@. Is there anything else we can or should be doing?
- Check to make sure you account has not expired.
- Check that your updates are run automatically when you receive an update notification.
- Check for any errors in your SNF log.
- Check that you are uploading your log files (so the bots can check stats).
W/ Regard to forwarding spam to spam@sortmonster.com - that's fine on a small scale.
If you have a lot of spam to submit then please set up a pop3 account on your system where you will forward your spam and provide us with login information (email address (login), password, and FQDN of the pop3 server). We will have our bots pick up your spam there.
If you have any clean spamtraps then please set up a different pop3 account on your server so that we can have our bots collect data from there. We keep user submitted spam and clean, automatically generated spamtrap data separate.
If you have a particular spam that you see chronically then you may be receiving that because we've not seen it, or because there is some conflict in system policies which has caused us not to code rules for it. You can submit "chronic" spams to us by saving the entire contents of the message to a file, zipping the file, and attaching it to an email to support@armresearch.com with the words "chronic spam" in the subject. We'll look into it and work with you to devise a solution for your system if possible.
Sometimes individual systems can incorporate custom black rules that are not appropriate for everyone... for example, blocking certain file extensions, character sets, words, etc.
Chronic Spam Problems
If you find you have a chronic problem with a particular spam that you have submitted to us then please zip up a few examples of the spam and send a note to us at support@armresearch.com with the subject title: Chronic Spam.
When we receive this email, we will contact you and work with you to help resolve this problem.
