prefork: server reached --max-clients setting, consider raising it messages

[Ron Culler]

After upgrading to 3.1 from 3.0 we are starting to see the following
error messages in our logs "prefork: server reached --max-clients
setting, consider raising it"

I have looked through the list archives and tried the suggestions that I
have found. (adjusting the max children numbers) But still no luck.

Here is our config:

3 inbound qmail servers doing rbl look-ups to block spam
They deliver to 6 qmail servers running qmail-scanner with spamc support
enabled. Those 6 servers look up against 4 spamd servers for scoring.

All the scanners and spamd boxes are P4 3.2GB 1M cache machines with 2GB
of ram. The spamd servers have a second disk for the bayes db file.
( haven't moved to SQL bayes yet) We are using Mysql for per user
actually per domain white\black list and default score options.

The inbound servers deliver via a dns round robin lookup to one of the 6
scanners. They in turn do the same dns round robin lookup to a spamd
server for scoring.

Spamd startup options - "-d -q -m 50 -x --
siteconfigpath=/etc/mail/spamassassin -i xxx.xxx.xxx.xxx -A xxx.xxx.xxx.
-u spamd"

Spamd servers are also running a caching dns server as well.

localdb.cf contents
required_hits 5.0
report_safe 0
use_bayes 1
lock_method flock
use_auto_whitelist 0
bayes_path /home/spamd/.spamassassin/bayes
bayes_auto_expire 1
skip_rbl_checks 1
loadplugin Mail::SpamAssassin::Plugin::Razor2
loadplugin Mail::SpamAssassin::Plugin::DCC
loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold

# TextCat - language guesser
#
loadplugin Mail::SpamAssassin::Plugin::TextCat

# WhitelistSubject - Whitelist/Blacklist certain subject regular
expressions
#
loadplugin Mail::SpamAssassin::Plugin::WhiteListSubject

ok_languages all
ok_locales en

user_scores_dsn DBI:mysql:spamassassin:xxx.xxx.xxx.xxx
user_scores_sql_username username
user_scores_sql_password password

bayes_ignore_header X-Habeas-SWE-1
bayes_ignore_header X-Habeas-SWE-2
bayes_ignore_header X-Habeas-SWE-3
bayes_ignore_header X-Habeas-SWE-4
bayes_ignore_header X-Habeas-SWE-5
bayes_ignore_header X-Habeas-SWE-6
bayes_ignore_header X-Habeas-SWE-7
bayes_ignore_header X-Habeas-SWE-8
bayes_ignore_header X-Habeas-SWE-9


Spamc switches - "-c -s 200000 -d round-robin hostname -u domainname"

We average between 50k to 75k messages scanned per day via this
system.

I was wanting to find out if there are any additional things that can be
done to better tune the spamassassin performance and get rid of these
messages. We normally see that message when a large burst of mail is
transmitted. Is there anything that can be done to either spamd/spamc or
both that would handle this better?

Thanks

[Mike Jackson]

Post by Ron Culler
Spamd startup options - "-d -q -m 50 -x --
siteconfigpath=/etc/mail/spamassassin -i xxx.xxx.xxx.xxx -A xxx.xxx.xxx.
-u spamd"

There's a disconnect between the spamd man page and what the error log
reports. What you're looking to change is the -m option in your startup. The
man page lists that option's long name as "--max-children", though the logs
refer to it as "--max-client".

You might want to run a few messages through in debug mode to see what's
causing the bottleneck. 50 children should be plenty for that many messages,
unless your mail traffic is incredibly bursty. I'll bet some step in the
checking process is tying up the spamd children for longer than necessary.

[Andrew Donkin]

Post by Ron Culler
After upgrading to 3.1 from 3.0 we are starting to see the following
error messages in our logs "prefork: server reached --max-clients
setting, consider raising it"

Short version: try --round-robin on spamd.

We scan about 100k messages a day balanced (with -H) between two spamd
hosts. Traffic is bursty, and during the bursts a lot of spam leaks
through unchecked because spamc reaches its 120s timeout.

The really annoying thing is that a spamd child would continue to chew
on its message for a further few hundred seconds before classifying
it, only to find that spamc had already given up. That child could
have been working for another spamc. I wonder if there is a way for
spamd to catch SIGPIPE or some other message from the client, and
abort.

So I added --round-robin and things improved markedly. The logging
isn't nearly so good ("grep prefork:" without --round-robin draws you
a great load histogram) but far less spam is leaking through.

One theory is that spamd doesn't spawn children quickly enough to cope
with rapidly-ramping load. I was thinking of ripping out spamd's
one-new-child-per-second throtting to see if it improved matters, but
that experiment is way down the task list now.

Try --round-robin. Scale it up until your spamd hosts are maximising
the use of their RAM.

Note that your spamd hosts should be similarly capable - spamc will
split the load evenly between all of them, even when all children are
busy on one.