Design ideas for stats and LDAP caching in 2.1.0

From: Murray S. Kucherawy <msk_at_blackops.org>
Date: Fri, 26 Mar 2010 10:41:12 -0700 (PDT)

Some quick notes about how I'm planning to tackle these issues in 2.1.0.
I'm trying to force myself not to work on this until early April, and the
CVS outage is helping. :)

Comments welcome.

Stats:
- still use Sleepycat for persistent storage
- have a "sentinel" record, key of just "." or "_at_" or something, that
contains version information so that future iterations know how to
interpret the records (e.g. "opendkim-stats" can always read older
databases)
- new version "2" starts to be used
- new binaries won't write to old DBs (log an error and disable stats)
- v2 stats keys are just domain names
- v2 stats values are structs containing most or all of the big array of
stats currently suggested in RFE #SF2964378
- "opendkim-stats" would have a new flag that hashes domain names on
output so that data submission is somewhat anonymized but can still be
collated

LDAP caching, to alleviate repetitive LDAP server load:
- enabled by a new --enable flag; forces --with-db
- creates an in-core (NULL path) Sleepycat hash whose keys are LDAP
queries and whose values are previously discovered results and TTLs
(configurable)
- on a query, check that DB for a cached result; use it if it's there
- if not, check a linked list for pending queries (pending queries have
pthread conditions associated with them)
-- if there's a pending query for the same thing, go to sleep waiting on
the associated condition
-- if there's no pending query, create a new linked list node with
associated pthread condition and start an LDAP query; go to sleep waiting
for the result
-- when the reply comes in, put it in the cache DB and signal
corresponding pthread condition; all blocked query threads wake up and get
the answer from the DB cache
Received on Fri Mar 26 2010 - 17:41:30 PST