Re: Revised stats gathering approach

From: Murray S. Kucherawy <msk_at_blackops.org>
Date: Sat, 14 Aug 2010 07:52:17 -0700 (PDT)

On Sat, 14 Aug 2010, Alessandro Vesely wrote:
> That is quite some structure for a flat file. For users of the library,
> I think any decent MTA has a Message-ID, possibly different from the
> 5322.Message-ID, that can work as a key for all lines originating from
> that message.

Alas, this is not the case. The current version of the stats stuff did
make that assumption because that is the case with sendmail job IDs (some
people call them envelope IDs too), but unfortunately postfix job IDs can
repeat. If that happens, then depending on the local database the message
re-using the job ID would overwrite the previous use and the related
signature data becomes confused, or the insertion fails because the record
already exists and data about the second message are lost entirely.

Also, unfortunately, the 5322.Message-ID cannot be trusted to be unique.

> Either a sequential number or header.b may be used to identify
> signatures. However, do you need a line for each signature/header?

Since a single message can have more than one signature, and since each
signature could sign a completely different set of header fields, I
believe this is necessary in order to report the kinds of statistics we'd
like to have. I don't plan to record the contents of signed header
fields, but only their names, to answer questions about which fields are
typically signed and which field typically gets changed to break
signatures.

> Why not dumping the complete signature, unwrapped into a single line and
> with truncated b= and bh= tags to save space?

Further compression of things like canonicalization and algorithm to
one-byte integers is possible as well, plus tabular ordered columns saves
the space of repeated tag names and equal signs. The file doesn't need to
be human-readable; we can have opendkim-stats do that translation.
Received on Sat Aug 14 2010 - 14:52:37 PST