Re: do we block ourself ?

On Tue, 13 Dec 2011, SM wrote:
>> Is there some documentation about the core design?
>
> I don't think so.

There is, but it's in a white paper I have yet to finish. It will be made
available with 2.5.0. Here it is in a few paragraphs:

Essentially, opendkim makes a reputation decision about a message based on
all of the signatures that passed. If none passed, it behaves as if the
NULL domain signed the message (which means all unsigned mail shares a
common reputation). The reputation of each of those domains is extracted.
The reptuation consists of two values: (a) a prediction of what percent of
mail bearing that domain's valid signauture is expected to be spam, and
(b) a prediction of how much mail signed by that domain should be
accepted. Both of these values are updated daily.

So when I get mail from "example.com", opendkim asks the ReputationLimits
data set for the limit (the (b) value above) and the ReputationRatios data
set for the ratio (the (a) value above). It then divides the (b) value by
the ReputationFactor setting; since opendkim is presently hard-coded to
flush its counts, ratios, and limits hourly, this should be set to 24,
which makes sense since the computed limit is a daily limit; this means if
your limit is 48, you get 2 per hour, for example. This prevents a domain
from consuming its whole daily total within the first few minutes of the
day.

The ratio is predicted based on all the feedback about the percent of mail
signed by each domain that's determined to be spam, aggregated over all
sites reporting.

The limit is predicted based on your own history of total flow of signed
mail from each domain, and is also affected by the ratio. For example,
where a domain might normally "earn" 100 email slots to your inboxes by
having a certain history, if it's predicted to have a high percentage of
spam, then only a very low amount of that 100 will actually be allowed in.

So, the ratio is the same for everyone, but you get your own tailored
limit.

If a domain hits either limit, the mail is temp-failed until the counts
reset.

The "is spam" determination is controlled by the ReputationSpamCheck
regular expression.

I think that's a very basic summary of how it all works.

> The temp-fail is not a major issue. However, we should consider an
> approach that does not impact timely delivery of messages if there are
> traffic bursts. For example, I may have sent one message to a recipient
> weekly. And then I get into a rapid exchange with the recipient.
> Would that burst be less visible when there is aggregation of domain
> traffic and the temp-fail be avoidable with the existing approach?

The math that computes the limits does account for some bursting,
especially if your history shows a generally clean stream of mail coming
from you. You can also do a limited whitelist using the ReputationMinimum
setting, which always allows some fixed number of messages per domain for
each time slice (hour) regardless of the reported values.

Your specific example of "one per week" would allow for a daily burst of
up to one message (note that that's a 7x burst), especially if it's
historically 100% clean. But that tiny rate is not much on which to base
an analysis.

When a message is temp-failed, it is still recorded by the statistics
system. Duplicates are suppressed, so re-tries are not counted against
the domain. However, that means if a domain bursts past its limit, the
burst is counted in today's data, allowing later trending upward to occur.

-MSK
Received on Tue Dec 13 2011 - 23:43:24 PST