[117342] in North American Network Operators' Group
Re: Repeated Blacklisting / IP reputation
daemon@ATHENA.MIT.EDU (Nick Feamster)
Thu Sep 10 10:20:11 2009
In-Reply-To: <28880400.269001252443527011.JavaMail.root@zimbra>
Date: Thu, 10 Sep 2009 10:19:04 -0400
From: Nick Feamster <feamster@cc.gatech.edu>
To: Tom Pipes <tom.pipes@t6mail.com>
Cc: nanog@nanog.org
Errors-To: nanog-bounces+nanog.discuss=bloom-picayune.mit.edu@nanog.org
Hi Tom (and NANOG),
You may be interested in an alternative approach, motivated by the
very problem you are facing (see below). Our system, SNARE, develops
IP reputation automatically based on a combination of network
features. We'll discuss the pros and cons of this approach at MAAWG.
The additional information that SNARE provides might be helpful.
-Nick
Detecting Spammers with SNARE: Spatio-temporal Network-level Automatic
Reputation Engine
Shuang Hao, Nadeem Ahmed Syed, Nick Feamster, Alexander Gray, Sven Krasser
Usenix Security '09, Montreal, Canada, August 2009
Users and network administrators need ways to filter email messages
based primarily on the reputation of the sender. Unfortunately,
conventional mechanisms for sender reputation -- notably, IP
blacklists -- are cumbersome to maintain and evadable. This paper
investigates ways to infer the reputation of an email sender based
solely on network-level features, without looking at the contents of a
message. First, we study first-order properties of network-level
features that may help distinguish spammers from legitimate senders.
We examine features that can be ascertained without ever looking at a
packet's contents, such as the distance in IP space to other email
senders or the geographic distance between sender and receiver. We
derive features that are lightweight, since they do not require seeing
a large amount of email from a single IP address and can be gleaned
without looking at an email's contents -- many such features are
apparent from even a single packet. Second, we incorporate these
features into a classification algorithm and evaluate the classifier's
ability to automatically classify email senders as spammers or
legitimate senders. We build an automated reputation engine, SNARE,
based on these features using labeled data from a deployed commercial
spam-filtering system. We demonstrate that SNARE can achieve
comparable accuracy to existing static IP blacklists: about a 70%
detection rate for less than a 0.3% false positive rate. Third, we
show how SNARE can be integrated into existing blacklists, essentially
as a first-pass filter.
http://gtnoise.net/pub/index.php?detail=3D14
On Tue, Sep 8, 2009 at 4:58 PM, Tom Pipes <tom.pipes@t6mail.com> wrote:
> I am amazed with the amount of thoughtful comments I have seen, both on a=
nd off list. It really illustrates that people are willing to try to help o=
ut, but there is an overall lack of clear direction on how to improve thing=
s. =A0Most of us seem to adopt that which has always just worked for us. Do=
n't get me wrong, I'm sure there are a lot of improvements/mods going on wi=
th RBL operators in terms of the technology and how they choose who to bloc=
k. =A0I'm also certain that most of the carriers are doing their best to fo=
llow RFCs, use e-mail filtering, and perform deep packet inspection to keep=
themselves off of the lists. AND there seems to be some technologies that =
were meant to work, and cause their own sets of problems (example: =A0allow=
ing the end user to choose what is considered spam and blacklisting based o=
n that). =A0As was said before, it's not the "WHY" but rather how can we fi=
x it if it's broke.
>
> The large debate seems to revolve around responsibility, or lack thereof.=
In our case, we are the small operator who sits in the sidelines hoping th=
at someone larger than us, or more influential has an opinion. =A0We partic=
ipate in lists, hoping to make a difference and contribute, knowing that in=
a lot of cases, our opinion is just that: =A0an opinion. =A0I suppose that=
could spark a debate about joining organizations (who shall go nameless he=
re), power to the people, etc.
>
> It seems as though a potential solution *may* revolve around ARIN/IANA ha=
ving the ability to communicate an authoritative list of reassigned IP bloc=
ks back to the carriers. =A0This could serve as a signal to remove a block =
from the RBL, but I'm sure there will be downfalls with doing this as well.
>
> In my specific case, I am left with a legacy block that I have to accept =
is going to be problematic. Simply contacting RBL operators is just not doi=
ng the trick. Most of the e-mails include links or at least an error code, =
but some carriers just seem to be blocking without an error, or even worse,=
an ACL...
>
> We will continue to remove these blocks as necessary, reassign IPs from o=
ther blocks where absolutely necessary, and ultimately hope the problem res=
olves itself over time.
>
> Thanks again for the very thoughtful and insightful comments, they are gr=
eatly appreciated.
>
> Regards,
>
>
> ---
> Tom Pipes
> T6 Broadband/
> Essex Telcom Inc
> tom.pipes@t6mail.com
>
>
> ----- Original Message -----
> From: "Tom Pipes" <tom.pipes@t6mail.com>
> To: nanog@nanog.org
> Sent: Tuesday, September 8, 2009 9:57:58 AM GMT -06:00 US/Canada Central
> Subject: Repeated Blacklisting / IP reputation
>
> Greetings,
>
>
> We obtained a direct assigned IP block 69.197.64.0/18 from ARIN in 2008. =
This block has been cursed (for lack of a better word) since we obtained it=
. It seems like every customer we have added has had repeated issues with b=
eing blacklisted by DUL and the cable carriers. (AOL, AT&T, Charter, etc). =
I understand there is a process to getting removed, but it seems as if thes=
e IPs had been used and abused by the previous owner. We have done our best=
to ensure these blocks conform to RFC standards, including the proper use =
of reverse DNS pointers.
>
> I can resolve the issue very easily by moving these customers over to our=
other direct assigned 66.254.192.0/19 block. In the last year I have done =
this numerous times and have had no further issues with them.
>
> My question: Is there some way to clear the reputation of these blocks up=
, or start over to prevent the amount of time we are spending with each cus=
tomer troubleshooting unnecessary RBL and reputation blacklisting?
>
> I have used every opportunity to use the automated removal links from the=
SMTP rejections, and worked with the RBL operators directly. Most of what =
I get are cynical responses and promises that it will be fixed.
>
> If there is any question, we perform inbound and outbound scanning of all=
e-mail, even though we know that this appears to be something more relatin=
g to the block itself.
>
> Does anyone have any suggestions as to how we can clear this issue up? Co=
mments on or off list welcome.
>
> Thanks,
>
> ---
> Tom Pipes
> T6 Broadband/
> Essex Telcom Inc
> tom.pipes@t6mail.com
>
>
>
>
