Re: multithreaded environment ? (better now: multi-core environment)

From: Dipl-Inform. Frank Gadegast <frank_at_powerweb.de>
Date: Fri, 2 Dec 2011 08:54:24 +0100 (MET)

>
> On Thu, Dec 01, 2011, Frank Gadegast wrote:
>
> > Thats what I mean, opendkim is only running on one CPU.
>
> How did you check that? If you have multiple connections going on
> concurrently, libmilter will create several threads (as Murray
> explained).

Sure it will create several threads ... but thats not the point.

sendmail already runs in a forked model.
apache and spamassassin too.
there processes are running on all cores (we once had a webserver
with two many pre-forked apache-childs, the scheduler (apache root-process)
was running on 100%, even when the workers where under 100% per core,
slowing done everything).

it really seems that our linux-kernel does not split all threads
to all available cores byt itself, meaning that we can have concurrent
mails having concurrent threads on several cores until the performance
of opendkim reaches 100% on the one core its running on.

threads really do not help a lot in this scenario.

> > Multi-threaded on a single CPU does not help a lot these days,
>
> It's up to your OS to properly support POSIX threads if
> multiple CPUs ("cores") are available.
>
> Maybe you can provide some information about your testing,
> i.e., what's your setup, what are the results?

we rebooted the server yesterday and turned of hyperthreading
in the bios cutting the available cores to the half.
And guess what ?

our throughput of signed emails nearly doubled (surely only
nearly, because other applications now dont perform that good
anymore) ..

Furthermore we stopped the virus scanning of outgoing mails
(with are the once to be signed), because the clamav-milter also
does not support multiple clamd-children.
This increased the throughput also a bit (but "top"
shows that opendkim now really always runs on 100%
on one core).

So: libmilter isnt really the problem, because libmilter
already runs several times on several cores, because sendmail
already forked.

The bottleneck is opendkim running on one core.

So I still think, that it would be helpfull to have
an additional opendkim-process (lets call it opendkim-milter),
that simply gets the threads and spreads them to preforked
opendkim-childs, that do the real work, so that the forked
children are running on several cores.


Kind regards, Frank
--
PHADE Software - PowerWeb                       http://www.powerweb.de
Inh. Dipl.-Inform. Frank Gadegast             mailto:frank_at_powerweb.de
Schinkelstrasse 17                                fon: +49 33200 52920
14558 Nuthetal OT Rehbruecke, Germany             fax: +49 33200 52921
======================================================================
Received on Fri Dec 02 2011 - 08:06:15 PST

This archive was generated by hypermail 2.3.0 : Mon Oct 29 2012 - 23:20:22 PST