Re: Design ideas for stats and LDAP caching in 2.1.0

From: SM <sm_at_resistor.net>
Date: Sat, 27 Mar 2010 08:25:41 -0700

Hi Murray,
At 10:41 26-03-10, Murray S. Kucherawy wrote:
>Some quick notes about how I'm planning to tackle these issues in
>2.1.0. I'm trying to force myself not to work on this until early
>April, and the CVS outage is helping. :)
>
>Comments welcome.
>
>Stats:
>- still use Sleepycat for persistent storage
>- have a "sentinel" record, key of just "." or "_at_" or something,
>that contains version information so that future iterations know how
>to interpret the records (e.g. "opendkim-stats" can always read
>older databases)

That's a good idea.

>- new version "2" starts to be used
>- new binaries won't write to old DBs (log an error and disable stats)

We'll need a way to move the data from the old DBs to the new format.

>- v2 stats keys are just domain names
>- v2 stats values are structs containing most or all of the big
>array of stats currently suggested in RFE #SF2964378

Please add number of messages where only part of the body is signed
to that. We could also do number of messages with more than one signature.

>- "opendkim-stats" would have a new flag that hashes domain names on
>output so that data submission is somewhat anonymized but can still be collated

Ok.

>LDAP caching, to alleviate repetitive LDAP server load:
>- enabled by a new --enable flag; forces --with-db
>- creates an in-core (NULL path) Sleepycat hash whose keys are LDAP
>queries and whose values are previously discovered results and TTLs
>(configurable)
>- on a query, check that DB for a cached result; use it if it's there
>- if not, check a linked list for pending queries (pending queries
>have pthread conditions associated with them)
>-- if there's a pending query for the same thing, go to sleep
>waiting on the associated condition
>-- if there's no pending query, create a new linked list node with
>associated pthread condition and start an LDAP query; go to sleep
>waiting for the result
>-- when the reply comes in, put it in the cache DB and signal
>corresponding pthread condition; all blocked query threads wake up
>and get the answer from the DB cache

We could also have a cache hits record for the DB cache but that
would be an additional write operation.

Regards,
-sm
Received on Sat Mar 27 2010 - 15:26:33 PST

This archive was generated by hypermail 2.3.0 : Mon Oct 29 2012 - 23:32:52 PST