Upgrading and Custom or SARE rulesets

[Deon de Villiers - Hetzner Africa]

Hi

I am about to upgrade SA for the first time (from 2.61 to 2.64). What I
am wanting to find out about is how do check whether rules which I have
from SARE, for example, are now included by default in SA, or where
default scores have changed significantly.

Are these an issue or is it safe to just go ahead withe upgrade. I will
obviously be doing significant testing, but have been wondering about
these 2 particular issues.

Thanks and Regards

[Bob McClure Jr]

Post by Deon de Villiers - Hetzner Africa
Hi
I am about to upgrade SA for the first time (from 2.61 to 2.64). What I
am wanting to find out about is how do check whether rules which I have
from SARE, for example, are now included by default in SA,

None have, to my knowledge.

Post by Deon de Villiers - Hetzner Africa
or where default scores have changed significantly.

Dunno. I'll let others more informed advise you.

Post by Deon de Villiers - Hetzner Africa
Are these an issue or is it safe to just go ahead withe upgrade. I will
obviously be doing significant testing, but have been wondering about
these 2 particular issues.

That's not a big jump of versions. It should go pretty painlessly.

If you are using blacklist.cf, blacklist_uri.cf, or bigevil.cf,
don't. Instead, install Mail::SpamAssassin::SpamCopURI after you
upgrade SA, and follow the directions here:

http://www.surbl.org/

Post by Deon de Villiers - Hetzner Africa
Thanks and Regards
--
Deon de Villiers
Hetzner Africa

Cheers,

[Robert Menschel]

Hello Deon,

Monday, August 16, 2004, 3:36:45 PM, you wrote:

D> Hi

D> I am about to upgrade SA for the first time (from 2.61 to 2.64). What
D> I am wanting to find out about is how do check whether rules which I
D> have from SARE, for example, are now included by default in SA, or
D> where default scores have changed significantly.

When will you be doing the upgrade?

I'm currently in the process of updating the General Subject, Header, and
HTML rule sets to identify/eliminate the 2.64 and 3.0 duplications.

In addition to files 0 through 3 (0 being most conservative/safest rules,
3 being most aggressive/risky), I'm creating
* "eng" files, for rules that work well in the English language but should be avoided by systems that get many emails written in other languages,
* "x30" files, for rules included in 3.0.0 distribution
* "x264" files, for rules included in the 2.64 distribution
* "arc" files, for rules that no longer hit any spam

So when you migrate to 2.64, you'll want to avoid the x264 files.

D> Are these an issue or is it safe to just go ahead withe upgrade. I
D> will obviously be doing significant testing, but have been wondering
D> about these 2 particular issues.

Our scoring is conservative enough that it probably isn't an issue to
begin with, and will be less of an issue when these updates to SARE are
completed.

After these three, ratware.cf is my next to review for duplication,
unless some other SARE Ninja beats me to it.

Bob Menschel

[Deon de Villiers - Hetzner Africa]

Thanks for the info Bob. I can wait! :)

What sort of time frame do expect to be working towards (won't hold you
to it ;)?

Thanks
Deon.

Post by Robert Menschel
Hello Deon,
D> Hi
D> I am about to upgrade SA for the first time (from 2.61 to 2.64). What
D> I am wanting to find out about is how do check whether rules which I
D> have from SARE, for example, are now included by default in SA, or
D> where default scores have changed significantly.
When will you be doing the upgrade?
I'm currently in the process of updating the General Subject, Header, and
HTML rule sets to identify/eliminate the 2.64 and 3.0 duplications.
In addition to files 0 through 3 (0 being most conservative/safest rules,
3 being most aggressive/risky), I'm creating
* "eng" files, for rules that work well in the English language but should be avoided by systems that get many emails written in other languages,
* "x30" files, for rules included in 3.0.0 distribution
* "x264" files, for rules included in the 2.64 distribution
* "arc" files, for rules that no longer hit any spam
So when you migrate to 2.64, you'll want to avoid the x264 files.
D> Are these an issue or is it safe to just go ahead withe upgrade. I
D> will obviously be doing significant testing, but have been wondering
D> about these 2 particular issues.
Our scoring is conservative enough that it probably isn't an issue to
begin with, and will be less of an issue when these updates to SARE are
completed.
After these three, ratware.cf is my next to review for duplication,
unless some other SARE Ninja beats me to it.
Bob Menschel

--
Deon de Villiers
Technical Manager
Hetzner Africa
Tel: [phone]
Fax: [phone]
http://www.hetzner.co.za/index.php?id=245

[ * Awarded Top 50 ICT Company in South Africa for the period 2003/4 by
the Corporate Research Foundation]
[ * Named National Top 300 High Growth Companies by DTI for the period
2004/5]

[Robert Menschel]

Hello Deon,

Tuesday, August 17, 2004, 1:33:08 AM, you wrote:

D> Thanks for the info Bob. I can wait! :)

D> What sort of time frame do expect to be working towards (won't hold you to it ;)?

I expect to have the General Subject and HTML files done this week.
Header files will be done this week or next. Ratware next week or the
week after.

For those interested, the duplications and overlaps identified so far
are:
- SARE_RECV_IP_FRMOIP1 now duplicates RCVD_DOUBLE_IP_SPAM in 2.64 and 3.0.0
- SARE_MSGID_ALL_CAPS now duplicates MSGID_SPAM_CAPS in 3.0.0
- SARE_RECV_FORGE_OUTBLZ now duplicates FAKE_OUTBLAZE_RCVD in 3.0.0
- SARE_HEAD_XORIP_NOTIP now duplicates X_ORIG_IPNOT_IPV4 in 3.0.0
- SARE_SUB_FTC_PORN now duplicates SUBJECT_SEXUAL in 3.0.0
- SARE_SUB_PAREN_NUM greatly overlaps SUBJ_2_NUM_PARENS in 3.0.0

I'll publish the full list once this analysis is complete.

Bob Menschel

Post by Robert Menschel
Hello Deon,
D> Hi
D> I am about to upgrade SA for the first time (from 2.61 to 2.64). What
D> I am wanting to find out about is how do check whether rules which I
D> have from SARE, for example, are now included by default in SA, or
D> where default scores have changed significantly.
When will you be doing the upgrade?
I'm currently in the process of updating the General Subject, Header, and
HTML rule sets to identify/eliminate the 2.64 and 3.0 duplications.
In addition to files 0 through 3 (0 being most conservative/safest rules,
3 being most aggressive/risky), I'm creating
* "eng" files, for rules that work well in the English language
but should be avoided by systems that get many emails written in other
languages,
* "x30" files, for rules included in 3.0.0 distribution
* "x264" files, for rules included in the 2.64 distribution
* "arc" files, for rules that no longer hit any spam
So when you migrate to 2.64, you'll want to avoid the x264 files.
D> Are these an issue or is it safe to just go ahead withe upgrade. I
D> will obviously be doing significant testing, but have been wondering
D> about these 2 particular issues.
Our scoring is conservative enough that it probably isn't an issue to
begin with, and will be less of an issue when these updates to SARE are
completed.
After these three, ratware.cf is my next to review for duplication,
unless some other SARE Ninja beats me to it.
Bob Menschel

[Robert Menschel]

Tuesday, August 17, 2004, 7:17:52 AM, I wrote:

RM> I expect to have the General Subject and HTML files done this week.
RM> Header files will be done this week or next. Ratware next week or the
RM> week after.

The General Subject rule updates have been published to the SARE site.

As documented at http://www.rulesemporium.com/rules.htm#genlsubj

http://www.rulesemporium.com/rules/70_sare_genlsubj0.cf -- Rules that hit
spam and no ham. Best file of this family.

http://www.rulesemporium.com/rules/70_sare_genlsubj1.cf -- Rules that hit
ham but have S/O of 0.900 or higher. Also rules that used to be in file 0
above, but no longer hit significant spam. Also rules that look
promising, but hit only a few spam and no ham. Suitable for most systems.
If used, should be used in conjunction with file 0 above.

http://www.rulesemporium.com/rules/70_sare_genlsubj2.cf -- Obfuscation
rules that should never hit ham, but that also don't hit any spam.
Appropriate only for systems with lots of resources.

http://www.rulesemporium.com/rules/70_sare_genlsubj3.cf -- Rules that hit
ham, with S/O below 0.900. Suitable for aggressive systems.

http://www.rulesemporium.com/rules/70_sare_genlsubj.cf -- One file that
contains all four files above. Aggressive systems can pull this one file
instead of all four individually.

http://www.rulesemporium.com/rules/70_sare_genlsubj_x30.cf -- New file,
containing the rules that duplicate or overlap distribution rules in
SpamAssassin 3.0.0 -- should be used by systems that have not upgraded to
3.0.0

http://www.rulesemporium.com/rules/70_sare_genlsubj_eng.cf -- Rules that
hit well in systems that deal strictly with the English languages, but
that might cause false positives on some systems that receive a lot of
emails in other languages.

http://www.rulesemporium.com/rules/70_sare_genlsubj_arc.cf -- Rules that
used to be in one of the files above, but that no longer hit any spam, or
that have too poor an S/O to be worth while. Should not be used by anyone
but the most aggressive systems with plenty of resources. SARE will
review these regularly and revive any that start hitting spam again.

Mass-checks on these final versions should be published by Friday
morning for those interested in the statistics.

Bob Menschel