Re: async resolver

From: Daniel Black <daniel.subs_at_internode.on.net>
Date: Tue, 20 Oct 2009 14:31:33 +1100

On Tuesday 20 October 2009 05:11:33 Murray S. Kucherawy wrote:

>
> libar does provide asynchronous resolution. It doesn't have a callback,
> but it has a wait timeout that allows libopendkim to implement the
> callback.

To me, and I'm an expert on threading by any stretch of the imagination, this
is still largely synchronous. The level of concurrency achieved seems to be
limited to 1/2 a page of code on the same function between the ar_request and
wait function.

To my understanding async processing would be:

1. found DKIM-Signature - send of async request for DNS key with a callback to
A)
2. see From header - send async ADSP request off with callback to B
3. continue buffering/hashing message.
4. eom milter request received - check we've got the answer that are pending
and if not we can wait here as the MTA needs an answer as to what to do.

A. DKIM call back received
 - if NX/tempfail/permail /timeout- set flag for main milter thread to stop
processing the DKIM-signature and set AR results for it
 - if found put the key in the data-structure for the signature and allow the
mail milter thread to hash the message buffer

B - ADSP call back
- set store ADSP response in call back

Hopefully this will be less latency, slightly more buffering or hashing
waiting for responses, but overall I predict it will have a greater
throughput.

From a libopendkim perspective the blocking occurs in the API functions that
retrieve requests based on the query. The blocking is only necessary if the
result isn't already there.

> libopendkim implements the callback mechanism. The function
> dkim_set_dns_callback()
from a package maintenance this is an anomaly. This function only exists if
enabled when libopendkim is compiled. The opendkim needs to be compiled with
the same option to actually use it. This actually creates to ABI's depending
on who it was compiled.

For libopendkim recommend - making dkim-set_dns_callback always exist and
return 'no supported' if not compiled right to the applications knows not to
expect callbacks.

For opendkim - the callback could be set and just ignore the result of
dkim_set_dns_callback as we can't do anything with it anyway.

> lets the caller specify a function to be called
> while DNS replies are pending, and a frequency. Once set, libopendkim
> will only call ar_waitreply() with that timeout set; when the timeout
> hits, the requested function is called and then it repeats the cycle until
> the query times out or an answer arrives. OpenDKIM uses this to arrange
> to call smfi_progress(), which tells the MTA "I'm still waiting for an
> answer, don't time out on me yet." In the case of sendmail at least (and
> probably postfix) this resets the milter read timeout.
>
> If you use the stock resolver (usually bind, of course) there's no callback
> mechanism, but recent versions are at least asynchronous.
didn't know about the recent versions. thanks.

> So we're already doing what c-ares offers, except for the "mature" part
> (depending on how you define that). :-)
fairly loosely - used in other applications of reasonable quality without too
many bug reports queueing from a reasonably reputable author.

> In fact looking at http://c-ares.haxx.se/why.html, libar meets all of their
> requirements except for 1b and 5a (1b is "synchronous interface" which we
> don't want or need here, and they don't really do it anyway; 5a is "must
> not require multi-threading" but this is definitely a multi-threaded
> operation).

yep - I didn't quite follow the no-multi-threading requirement here either. As
an aside Python I think does something tricky in faking an async in a single
thread. I'm not suggesting we do this but for a different API look at
threading its interesting.
Received on Tue Oct 20 2009 - 03:31:51 PST

This archive was generated by hypermail 2.3.0 : Mon Oct 29 2012 - 23:32:29 PST