From dwmw2 at infradead.org  Mon Feb 23 23:00:21 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Mon Feb 23 23:00:30 2004
Subject: Initial thoughts.
Message-ID: <1077577191.11584.785.camel@imladris.demon.co.uk>

The problem that SPF tries to solve is a real one -- _anyone_ can send
mail claiming to be from, for example, dwmw2@infradead.org, and there's
currently no way of doing anything more than verifying that this is at
least a real address.

It's not the _only_ problem, but it's an interesting one.

The real problem with SPF is that placing restrictions on the IP
addresses from which mail for a given domain may originate is
fundamentally incompatible with the way SMTP actually works and has done
for decades. 

SRS attempts to fix this, but has its own problems, and even so could
only _actually_ fix the problem with SPF if it were to be implemented
_everywhere_.

Basically, the SPF+SRS 'solution' is no more realistic than any of the
myriad other schemes which have cropped up over the years which required
a fundamental departure from the status quo; a complete obsolescence of
SMTP as-is. It's just slightly more cunningly disguised as a practicable
plan than some of the others.

The problem which it addresses, however, remains as I said an
interesting one.

A practicable solution will work end-to-end, without the need for
participation by conservative and uninterested third parties. I think
that has to be considered a fundamental requirement, along with
backward-compatibility with the status quo. It'll let those who _want_
to participate advertise information to each other and verify each
others mail, but won't break if third parties don't play. And hopefully
it'll eventually gain critical mass so that we can stop talking to those
who _don't_ do it.

One possibile solution which looks like it could answer the same problem
is some form of cryptographic verification of the sender. By signing the
outgoing mail with a private key, the public complement of which is
available in the DNS, we can just as easily verify that it really come
from the address it _claims_ to come from.

GPG signing is an obvious first stab at a solution. One might easily
declare to the world at large, if there were a standardised way in which
to declare it, that "All mail from dwmw2@infradead.org will be
GPG-signed. If you receive a mail claiming to be from that address which
is _not_ GPG-signed, reject it".

GPG serves to illustrate the point, but isn't great when you look at the
details. The plain-text signing method introduces cumbersome noise, and
the MIME method, well, requires MIME -- which is considered by many to
be cumbersome noise. :)

Ideally we'd be able to introduce a signing scheme where we put a hash
into the headers, cryptographically verifying some part of the mail
headers and contents. The questions are as follows:

	1. What part of the mail and its contents need verification?
	2. How shall it be done?

The problem we have is that forwarding MTAs may perform arbitrary
mutilation of our mail, disrupting our signature.

We don't have to worry _too_ much about mailing lists -- although they
often munge headers and add bits to the mail, they _also_ change the
reverse-path, so our original signature becomes irrelevant anyway. But
we do need to be concerned about virus checking crapware which adds its
own little adverts, and perhaps also MTAs which do address rewriting.

Actually I'm not entirely sure either of those _are_ good examples. If
we're doing the crypto-signing in the MTA, that can quite happily happen
_after_ the virus-crapware does its thing, or the address rewriting.
These things happen on _outgoing_ mail, and it's the responsibility of
the sender to ensure that the mail leaves their servers with correct
signatures, if they choose to publish their pubkeys for the purpose of
verification.

It's late -- I must be missing something here. What are the _valid_
instances of mails getting rewritten in transit, by intermediate
forwarding hosts between those which are doing the signing and those
border MX hosts on the reception side which are _verifying_ the
signatures?

--- 

Also interesting to note is that there's another really simple solution
to the original problem of faked senders. That is if we accept, as is
going to _have_ to be the case if we're honest with ourselves, that
_any_ scheme is only going to work between participating senders and
receivers. We just implement some form of SRS on our own outgoing mail,
then perform sender verification callouts on received mail. If someone
attempts to send mail claiming to be from dwmw2@infradead.org, my server
will refuse the sender verification callout with empty reverse-path,
because nobody _ever_ sends mail from that address. We only send mail
from SRS0+XXX+YY+infradead.org+dwmw2@infradead.org; addresses which are
extremely hard to fake.

-- 
dwmw2



From B.Candler at pobox.com  Tue Feb 24 09:39:13 2004
From: B.Candler at pobox.com (Brian Candler)
Date: Tue Feb 24 10:12:28 2004
Subject: Initial thoughts.
Message-ID: <20040224093913.GA4143@uk.tiscali.com>

> The problem that SPF tries to solve is a real one -- _anyone_ can send
> mail claiming to be from, for example, dwmw2 at infradead.org, and there's
> currently no way of doing anything more than verifying that this is at
> least a real address.

The problem that SPF tries to solve is actually smaller than that: it only
verifies the envelope-sender, i.e. the return address for bounces. By the
time the message is displayed to the user, this information has either been
lost, or has been put in a Return-Path: header which only experts would know
how to view and interpret. The MUA displays the From: header, and SPF makes
no attempt to validate this.

I believe there are two reasons why people want to validate the
envelope-sender:

(1) To prevent bounces being sent to forged addresses, otherwise known as
"joe jobs" (that is, if a spammer puts my address as the envelope sender on
a spam, and it bounces, then I get the bounce).
This is a worthy aim, but
- SPF won't help until the whole Internet adopts SPF (if ever)
- SPF won't help with spam sent directly to an end-user with a null
  envelope sender
- There are simpler and better solutions anyway

(2) To identify spam. This is IMO a pointless exercise, since all it will do
is change the spammers' behaviour so that their mails are SPF-compliant (and
note that it's still easy to forge mails under SPF using any domain which
your ISP's IP addresses permit)

> A practicable solution will work end-to-end, without the need for
> participation by conservative and uninterested third parties.

Well, I still think the first thing which needs to be done is to agree on
the problem, before we look at the solution. I put some thoughts up here:
http://archives.listbox.com/spf-discuss@v2.listbox.com/200402/0603.html

Personally, I exclude "verifying the envelope sender on deliverable mails"
from the problem set; if the message is deliverable, then by definition we
don't use the envelope sender anyway, because there's no bounce to be sent.
But I do include forged headers (at least the From: header) as a worthwhile
aim.

> Ideally we'd be able to introduce a signing scheme where we put a hash
> into the headers, cryptographically verifying some part of the mail
> headers and contents. The questions are as follows:
> 
>         1. What part of the mail and its contents need verification?
>         2. How shall it be done?

3. How is this done in a way which is at least partly useful in the interim
period, with unmodified MUAs?

> The problem we have is that forwarding MTAs may perform arbitrary  
> mutilation of our mail, disrupting our signature.

Indeed, although in many cases only the headers are munged.

I did wonder whether it was worthwhile adding a sort of cryptographic
"postmark", like a signed version of the Received: header. This could make
declarations such as:
- this mail was handled by the mail relay at foo.com
- this mail was received from IP address x.x.x.x
- this mail was sent by user X who authenticated using SMTP AUTH

Having this information in a proper structured format (unlike the ad-hoc
format of Received: headers), *and* signed for non-repudiation, would lead
to faster tools for reporting of spam and hence for dynamic black-lists.

Unfortunately, anyone who picks up one of these messages with signed
postmarks can then modify it and re-send it, unless the signature covers the
entire message. And thus we are back to the problem that the whole message
needs to be signed.

If we are building an infrastructure which converts plain mails into fully
OpenGPG or S/MIME signed mails: that is cool, in some ways. But the
automatic verification then has to be tied into whatever their existing key
infrastructure is, which is generally not suitable for our needs - we want
something DNS-based.

And if an end-user has signed a message using their own private key, we must
be able to add our own ISP-generated signature as well as it passes through
our mailserver.

(Actually, this is not really a "signature", it's more a "certificate". I,
ISP.COM, do humbly declare that this E-mail was sent by X@Y.COM.)

This kind of turns the normal usage of public-key on its head: normally, the
end-user would sign the mail with their own key, and then the ISP would
certify their *key* to say it belongs to the person with E-mail address
X@Y.COM. Instead, we're saying that the ISP should certify each *message* to
say that it came from X@Y.COM. This has the big plus, from our point of
view, that the end-user does not need to have their own key or sign the
message at all.

Then you have to consider how an end-point will verify the signatures. If
it's an MTA, then it can make a lookup in the DNS in real-time. However if
it's an off-line MUA, which has downloaded the mail using POP3 say, then
either it must trust the final MTA to stamp the message somehow, or it must
be able to verify the signatures off-line (which means a chain of trust up
to some root certificates)

> It's late -- I must be missing something here. What are the _valid_
> instances of mails getting rewritten in transit, by intermediate
> forwarding hosts between those which are doing the signing and those
> border MX hosts on the reception side which are _verifying_ the
> signatures?

- extra Received: headers are added (as required by RFC2821)
- other extra headers may be added (Sender:, X-wotsit:)
- some evil systems rewrite From: (From: foo@mail.mydomain.com to
  From: foo.bar@mydomain.com; typically only at the sending end though)
- often footers are added (adverts, company signature, pointless virus
  sweeper signatures)

Those are the major things. Given that a small but non-zero set of people
use GPG successfully, including sending to mailing lists, I think in
principle it's possible to sign the whole message.

Whether people will accept the extra signature noise, or the concept of
mails being signed by third-parties in transit, is another matter...

Cheers,

Brian.

From dwmw2 at infradead.org  Tue Feb 24 11:30:02 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Tue Feb 24 11:30:08 2004
Subject: Initial thoughts.
In-Reply-To: <20040224093913.GA4143@uk.tiscali.com>
References: <20040224093913.GA4143@uk.tiscali.com>
Message-ID: <1077622201.7826.219.camel@hades.cambridge.redhat.com>

On Tue, 2004-02-24 at 09:39 +0000, Brian Candler wrote:
> The problem that SPF tries to solve is actually smaller than that: it only
> verifies the envelope-sender, i.e. the return address for bounces. By the
> time the message is displayed to the user, this information has either been
> lost, or has been put in a Return-Path: header which only experts would know
> how to view and interpret. The MUA displays the From: header, and SPF makes
> no attempt to validate this.

This is true. It's a very limited problem.

> I believe there are two reasons why people want to validate the
> envelope-sender:
> 
> (1) To prevent bounces being sent to forged addresses, otherwise known as
> "joe jobs" (that is, if a spammer puts my address as the envelope sender on
> a spam, and it bounces, then I get the bounce).
> This is a worthy aim, but
> - SPF won't help until the whole Internet adopts SPF (if ever)
> - SPF won't help with spam sent directly to an end-user with a null
>   envelope sender
> - There are simpler and better solutions anyway

Like what I'm doing with SRS of my own address. I can now reject bounces
to dwmw2@infradead.org; the problem is solved. I receive no more bounces
to joe-jobs. Yes, there are people out there who don't bother to do any
form of checking on their incoming mail. There always _will_ be. But
they don't affect _me_.

> (2) To identify spam. This is IMO a pointless exercise, since all it will do
> is change the spammers' behaviour so that their mails are SPF-compliant (and
> note that it's still easy to forge mails under SPF using any domain which
> your ISP's IP addresses permit)

(3) To obsolete SPF, by proposing a _practicable_ solution which doesn't
throw the baby out with the bathwater by making changes which are 
fundamentally incompatible with the way the world actually works.

> > A practicable solution will work end-to-end, without the need for
> > participation by conservative and uninterested third parties.
> 
> Well, I still think the first thing which needs to be done is to agree on
> the problem, before we look at the solution. I put some thoughts up here:
> http://archives.listbox.com/spf-discuss@v2.listbox.com/200402/0603.html

Interesting reading; thanks for the pointer.

Your (1) I think my SRS solution already covers. This does of course
require that I'm willing to accept (1c). But as I already refuse to
accept mail from sites which don't like 'MAIL FROM:<>' and which don't
accept mail to postmaster, etc... that's a policy decision I'm happy
with. They're violating a MUST. You can't work round people like that.
 
(1a) is easily fixable since it's only locally-generated addresses. I
control all the inputs -- I can make it fit. (1b) I think isn't too
likely. Some lists _do_ filter so only subscribers can post, but I
believe they all do it on the From: header not the reverse-path.

Your (2) isn't solved by my SRS toy, but crypto-signing of mail _would_
solve it. You mandate that 'all mail From: dwmw2@infradead.org _shall_
be signed with key XXXX'. That's basically what we're here for, I think.
It's a far more interesting problem, and not one which SPF addresses.

> Personally, I exclude "verifying the envelope sender on deliverable mails"
> from the problem set; if the message is deliverable, then by definition we
> don't use the envelope sender anyway, because there's no bounce to be sent.

You don't know that at RCPT time. You know that you don't want to
generate a bounce _now_, but you (or one of your other machines) might
need to do so _later_. Consider the case of a backup MX. You accept the
mail because the primary is down (and hence can't tell you it's to an
invalid address). Later it comes back and you need to bounce. Or if
you're running virtual domains, you can get similar situations.

I agree that if you _can_ reject at SMTP time rather than accepting and
later bouncing, you should. But there _are_ situations in which you'll
still need to generate a bounce.

And, FWIW, I disagree with the philosophy that "the spammers will catch
up so it's all pointless". The spammers will work hard to catch up with
the _majority_. They won't necessarily catch up with the clueful. 

But these are policy decisions to be made locally. I'm sure you agree
that it's worthwhile to be able to verify the envelope sender, and
precisely _when_ we do it is a site-local choice. There's no need for us
to digress.

> But I do include forged headers (at least the From: header) as a worthwhile
> aim.

Yes. 

> > Ideally we'd be able to introduce a signing scheme where we put a hash
> > into the headers, cryptographically verifying some part of the mail
> > headers and contents. The questions are as follows:
> > 
> >         1. What part of the mail and its contents need verification?
> >         2. How shall it be done?
> 
> 3. How is this done in a way which is at least partly useful in the interim
> period, with unmodified MUAs?

Oh and the important one I forgot to mention, because it's so obvious:
	4. How do we prevent replay attacks?

> > The problem we have is that forwarding MTAs may perform arbitrary  
> > mutilation of our mail, disrupting our signature.
> 
> Indeed, although in many cases only the headers are munged.

A problem which is solved by signing only those headers which must _not_
be munged. I think it's reasonable, for example, to assume that a remote
MTA will not munge the From: header. The _local_ one may do so before
sending, but if we implement signing at the MTA level then that's fine.

I think we _should_ implement signing at the MTA level, precisely
because so much of the munging that goes on is done by the sending MTA.

> I did wonder whether it was worthwhile adding a sort of cryptographic
> "postmark", like a signed version of the Received: header. This could make
> declarations such as:
> - this mail was handled by the mail relay at foo.com
> - this mail was received from IP address x.x.x.x
> - this mail was sent by user X who authenticated using SMTP AUTH

Those are mostly uninteresting to me, I think. Well, except for 'this
mail was sent by user X@domain.com', where that is a statement signed
with the public key for domain.com. I don't care _how_ they
authenticated themselves; only that someone with the private key for
domain.com actually believed them.

> Having this information in a proper structured format (unlike the ad-hoc
> format of Received: headers), *and* signed for non-repudiation, would lead
> to faster tools for reporting of spam and hence for dynamic black-lists.

I think this is a digression. It's a cute idea, but it still requires
implementation by everyone. Only those who care would implement it, and
those who care _already_ put useful Received: headers in. It's hardly
difficult to see where real Received: headers stop and the faked ones
start; we don't need to address this here.

> Unfortunately, anyone who picks up one of these messages with signed
> postmarks can then modify it and re-send it, unless the signature covers the
> entire message. And thus we are back to the problem that the whole message
> needs to be signed.

Yes. We have to protect against replay attacks.

> If we are building an infrastructure which converts plain mails into fully
> OpenGPG or S/MIME signed mails: that is cool, in some ways. But the
> automatic verification then has to be tied into whatever their existing key
> infrastructure is, which is generally not suitable for our needs - we want
> something DNS-based.

And I think that the fact that OpenGPG and S/MIME are _visible_ to
uninterested third parties would be a showstopper to widespread
adoption. We really need something that can be hidden in the headers
where it doesn't offend anyone.

> And if an end-user has signed a message using their own private key, we must
> be able to add our own ISP-generated signature as well as it passes through
> our mailserver.

Why would a message need to be signed with more than one key? I was
envisaging a scheme where the end-user identifies themselves as usual
(and with unmodified software) with SMTP AUTH and the _ISP_ signs with
the appropriate key.

If we _do_ have a per-user key scheme where the end-user has signed a
message using their own key, why would it need signing by the ISP too?

> Then you have to consider how an end-point will verify the signatures. If
> it's an MTA, then it can make a lookup in the DNS in real-time. However if
> it's an off-line MUA, which has downloaded the mail using POP3 say, then
> either it must trust the final MTA to stamp the message somehow, or it must
> be able to verify the signatures off-line (which means a chain of trust up
> to some root certificates)

This should be left as flexible as possible, to allow sites to implement
their own policy. It should be possible to do _either_. 

Verifying the signatures in the MUA can still do the DNS lookups to get
the pubkey, though, surely? Just like SpamAssassin can do RBL and SPF
lookups. You're normally online at the time you _fetch_ the mail, or you
wouldn't be able to talk to the mail server.

> > It's late -- I must be missing something here. What are the _valid_
> > instances of mails getting rewritten in transit, by intermediate
> > forwarding hosts between those which are doing the signing and those
> > border MX hosts on the reception side which are _verifying_ the
> > signatures?
> 
> - extra Received: headers are added (as required by RFC2821)
> - other extra headers may be added (Sender:, X-wotsit:)
> - some evil systems rewrite From: (From: foo@mail.mydomain.com to
>   From: foo.bar@mydomain.com; typically only at the sending end though)
> - often footers are added (adverts, company signature, pointless virus
>   sweeper signatures)

I think we should avoid including any header in the signature which we
don't _really_ need; especially those which might get munged. We really
need to avoid false negatives, and err on the side of permissiveness.

In fact, I suspect we should include only the Date:, From:, and
Message-Id: headers. We have a very reasonable expectation that those
will not be changed in transit, except perhaps by the local MTA which as
discussed is fine.

And yes, we should also include some form of fuzzy hash of the mail. As
a primitive example, we could perhaps specify the number of lines which
were present when the mail was signed, and then the receiver knows how
much to trim from the bottom. 

We probably also need to canonicalise the character set and encoding;
transition between systems supporting 8BITMIME and not will cause
breakage otherwise. 

> Whether people will accept the extra signature noise, or the concept of
> mails being signed by third-parties in transit, is another matter...

Signature noise can be hidden in the headers so that people don't notice
it. The concept of mail being signed by third parties would be a
conceptual problem for some people if we used GPG or S/MIME, but if we
introduce a new encryption scheme which is _specifically_ for automatic
use by the sending party and not their agent, and which nobody ever
wants to use as _legal_ proof of sender's identity, that should be fine.


==============================

The proposal as it stands; as a straw man to be argued against...

Publish records in DNS stating somehow 'all mail From: *@infradead.org
shall be signed with key XXX' 

On sending, when the sender is authenticated (via SMTP AUTH or other
methods like just being a valid local user):

Generate an SHA1 hash of the From: Date: and Message-ID: headers. Call
it $HDR_HASH.

For each MIME part of the message (considering non-MIME messages to be a
single text/plain), generate a hash for _that_ part, including $HDR_HASH
in the hash input. Undo any encoding (i.e. use the _binary_ as hash
input, not base64 encoding of it). If a part is _text_ and a charset is
specified, also convert to UTF-8 for the purposes of generating the
hash, and count the number of lines in the part, calling it NLINES. 

Generate an SHA1 hash of $HDR_HASH, $NLINES, and the actual contents,
together. Sign it with the private key and add this in a Content-Hash:
header in the MIME part. If the part was text, also add a
Content-Hash-Lines: header with the number NLINES. Finally, add an
_insecure_ hash, a simple checksum, of the same textual content. Add
this in a Content-Hash-Locator: header.

Obviously for plain non-MIME mails these headers go in the main headers
where Content-* headers normally go.

---

On receipt: 

Look up the TXT records (or whatever) for domain of each mailbox listed
in the From: header. If one has a record which specifies a signature is
required, verify it as follows:

For each binary part...
  Look for the Content-Hash* headers, do the obvious thing.

For each text part...
  Look at NLINES and see how many extra lines have been added in
transit. If nothing's been added, do the obvious thing -- there's been
no munging.

If lines have been added, we use the trivial 'locator' checksum to find
the correct lines within what we've received. (Local policy may dictate
that you want to reject the mail at this state if too many lines have
been added in 'transit'.)

Use the rolling checksum to _find_ a suitable NLINES inside the text,
matching the contents of the Content-Hash-Locator: header. The reason we
use a rolling checksum is so that it's trivial to add a line _and_
'remove' a line from an _existing_ checksum. If you want to look for a
match at ten different places in the text, you don't have to recalculate
an SHA1 hash from scratch each time. 

Once you find a match for the cheap checksum, verify the strong hash. If
it fails, continue trying to find a match for the cheap checksum.

----

I think it's OK to assume that the messages won't be munged in the
_middle_. We accept messages with crap added at the start, and at the
end, and we should even handle entire MIME parts being dropped by list
software. 

We cryptographically verify the sender, date and contents of the
message, without much chance of false negatives. Seem reasonable at
first glance?

-- 
dwmw2


From B.Candler at pobox.com  Tue Feb 24 13:30:52 2004
From: B.Candler at pobox.com (Brian Candler)
Date: Tue Feb 24 13:33:44 2004
Subject: Initial thoughts.
In-Reply-To: <1077622201.7826.219.camel@hades.cambridge.redhat.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	<1077622201.7826.219.camel@hades.cambridge.redhat.com>
Message-ID: <20040224133052.GA4716@uk.tiscali.com>

On Tue, Feb 24, 2004 at 11:30:02AM +0000, David Woodhouse wrote:
> Your (2) isn't solved by my SRS toy, but crypto-signing of mail _would_
> solve it. You mandate that 'all mail From: dwmw2@infradead.org _shall_
> be signed with key XXXX'. That's basically what we're here for, I think.
> It's a far more interesting problem, and not one which SPF addresses.

Agreed.

> > Personally, I exclude "verifying the envelope sender on deliverable mails"
> > from the problem set; if the message is deliverable, then by definition we
> > don't use the envelope sender anyway, because there's no bounce to be sent.
> 
> You don't know that at RCPT time. You know that you don't want to
> generate a bounce _now_, but you (or one of your other machines) might
> need to do so _later_.

Absolutely true. But suppose you just accept the envelope-sender as given.
Then either:

1. it's correct, in which case the bounce goes to the right place; or

2. it's maliciously incorrect (a joe job pointing to someone else), in which
case it falls under the different problem case (1), i.e. I don't want to
accept bounces to mail I didn't send; or

3. it's an accidental misconfiguration by the end-user. Unfortunately, most
mail has already been through one relay (the sender's ISP's smarthost), so
detecting an invalid return address after this stage is futile.

It would be nice for the ISP's smarthost to validate the MAIL FROM address,
so it could reject mails (550) before accepting the content. This is
something which I have configured - at least, on smarthosts, I do a simple
DNS check that the MAIL FROM domain is OK.

To do a stronger check at this point would involve requiring all users to do
SMTP AUTH and binding their AUTH username to the MAIL FROM name(s) they are
permitted to use.

> And, FWIW, I disagree with the philosophy that "the spammers will catch
> up so it's all pointless". The spammers will work hard to catch up with
> the _majority_. They won't necessarily catch up with the clueful. 

I think the SPF people are counting on some sort of equilibrium: say only 5%
of the Internet adopts SPF, so little that spammers won't care. But they may
get some benefit if, say, spammers send mail from foo@aol.com but aol.com is
declaring information in SPF.

They will also take the pain of SPF breakage, but they may gain some net
benefit.

But this is an unstable equilibrium. If SPF does actually deliver some
benefit, then everyone will want it, at which point the spammers will bypass
it. Personally I think the SRS-signing of outgoing mail as you have
implemented gives much greater benefit to the implementor, immediately, at
very little cost.

> But these are policy decisions to be made locally. I'm sure you agree
> that it's worthwhile to be able to verify the envelope sender

Actually, no :-) See above.

> > > The problem we have is that forwarding MTAs may perform arbitrary  
> > > mutilation of our mail, disrupting our signature.
> > 
> > Indeed, although in many cases only the headers are munged.
> 
> A problem which is solved by signing only those headers which must _not_
> be munged. I think it's reasonable, for example, to assume that a remote
> MTA will not munge the From: header. The _local_ one may do so before
> sending, but if we implement signing at the MTA level then that's fine.

Or else you put in a brand new header, say "X-Signed-From:", which includes
the crypto signature.

Any MUA which understands it can use it. Any final end-point MTA can compare
the signed address with the From: address, and modify the From: address
appropriately:

    From: "Dubya Bush" <president@whitehouse.gov>
becomes:
    From:  "Dubya Bush [PROBABLY FORGED, claimed president@whitehouse.gov, real sender spam123@isp.net]" <spam123@isp.net>

The sending MTA can add a header saying "real sender was spam123@isp.net"
without touching the original header.

But as you say, you have to prevent replay (i.e. cut-and-paste of the signed
header), which means the header has to include, say, a hash of the entire
message body, and the message must be in a form which is not likely to be
corrupted in transit otherwise the hash is invalidated.

A review of existing work in OpenPGP, S/MIME etc would be useful here I
think, because they must have exactly the same problem.

> > I did wonder whether it was worthwhile adding a sort of cryptographic
> > "postmark", like a signed version of the Received: header. This could make
> > declarations such as:
> > - this mail was handled by the mail relay at foo.com
> > - this mail was received from IP address x.x.x.x
> > - this mail was sent by user X who authenticated using SMTP AUTH
> 
> Those are mostly uninteresting to me, I think. Well, except for 'this
> mail was sent by user X@domain.com', where that is a statement signed
> with the public key for domain.com. I don't care _how_ they
> authenticated themselves; only that someone with the private key for
> domain.com actually believed them.

There are different levels of trust which could be indicated:

- a mail from "b.candler@pobox.com", sent via myisp.net, could be signed via
myisp.net saying "this mail came through me, but I cannot vouch for the
sender; the real sender could be be anyone@isp.net"

- a mail from "random@myisp.net", sent via myisp.net, could be signed as
"this mail came through me, and it's one of my domains, the RHS may be right
but I can't vouch for the LHS of the address" (this is roughly what SPF
attempts)

- a mail from "random@myisp.net", where the user authenticated with SMTP
AUTH and with a username which matched that address, could be signed "this
mail came through me and I vouch for the authenticity of the sender"

- a mail from "b.candler@pobox.com", where I authenticated with SMTP AUTH as
brian@myisp.net" could be signed "this mail came through me from the user
brian@myisp.net, but I cannot vouch for the sender address given"

Such policies could allow existing practices to continue working, and for an
address which cannot be fully vouched for (which would be the vast majority
in the early days), you at least get _some_ information about the sender or
the domain.

Then it's a logical extension to include the IP address. This is the
information which the ISP needs to trace the original user if they did not
authenticate with SMTP AUTH. A spam complaint containing such information
could be processed in a highly automated fashion, AND provide strong
evidence (e.g. if needed in a court of law) for kicking off the spammer.

Hmm. Let me put it another way. Suppose in our smarthost MTA we see a mail
which says
   From: president@whitehouse.gov

but whitehouse.gov has not published any key information, so we know the
recipient won't be able to verify it one way or the other.

I think it would be better for us to sign it saying "This mail probably came
from *@myisp.net", than not sign it at all. The receiver then has something
to verify (using the myisp.net key in the DNS) and show to the end-user.

Since the certification disagrees with the originally-provided From: header,
the end-user has a better indication of a forgery than if no signature were
applied, even though whitehouse.gov has not published anything.

> > Having this information in a proper structured format (unlike the ad-hoc
> > format of Received: headers), *and* signed for non-repudiation, would lead
> > to faster tools for reporting of spam and hence for dynamic black-lists.
> 
> I think this is a digression. It's a cute idea, but it still requires
> implementation by everyone. Only those who care would implement it, and
> those who care _already_ put useful Received: headers in.

If we digress, I don't think it's far, if we are talking initially about
crypto-signing with an ultimate view towards reduction of spam, not
crypto-signing for E-commerce applications etc.

Of course, crypto-signing will never be able to identify a piece of spam,
and it will never be able to identify a spammer (because spammers can create
new electronic identities for themselves at will). However, if it can help
reliably trace the *source* of a spam, then that pressure can be put on that
source. And with dynamic IPs, having a fast response to spam is important
(spamcop etc)

> It's hardly
> difficult to see where real Received: headers stop and the faked ones
> start; we don't need to address this here.

Not for experts like you or me. But there are plenty of people with enough
knowledge to find full headers, but not enough to understand them, who send
spam complaints to the wrong places (based on forged HELO names, forged
Received: headers, and forged envelope senders). At least one person on
spf-discuss was arguing that as a case for validating HELO names.

> > And if an end-user has signed a message using their own private key, we must
> > be able to add our own ISP-generated signature as well as it passes through
> > our mailserver.
> 
> Why would a message need to be signed with more than one key? I was
> envisaging a scheme where the end-user identifies themselves as usual
> (and with unmodified software) with SMTP AUTH and the _ISP_ signs with
> the appropriate key.

Sure. I meant if we used an existing mechanism like OpenGPG. What if we come
across a mail which is already in OpenGPG format?

> If we _do_ have a per-user key scheme where the end-user has signed a
> message using their own key, why would it need signing by the ISP too?

We need to add some level of the ISP 'vouching for' the user - which, unless
the end-user's key has been signed by the ISP, means signing the message
somehow.

Otherwise, we are unable to add any trust for an existing GPG message.
Spammers can always start sending OpenGPG-signed mails using throwaway keys.

If we treat OpenGPG as just a lump of MIME, and add our own signature in the
top headers, this should be fine.

> > Then you have to consider how an end-point will verify the signatures. If
> > it's an MTA, then it can make a lookup in the DNS in real-time. However if
> > it's an off-line MUA, which has downloaded the mail using POP3 say, then
> > either it must trust the final MTA to stamp the message somehow, or it must
> > be able to verify the signatures off-line (which means a chain of trust up
> > to some root certificates)
> 
> This should be left as flexible as possible, to allow sites to implement
> their own policy. It should be possible to do _either_. 

A good goal.

> Verifying the signatures in the MUA can still do the DNS lookups to get
> the pubkey, though, surely? Just like SpamAssassin can do RBL and SPF
> lookups. You're normally online at the time you _fetch_ the mail, or you
> wouldn't be able to talk to the mail server.

Not if you're offline (i.e. disconnected from the Internet)

> In fact, I suspect we should include only the Date:, From:, and
> Message-Id: headers. We have a very reasonable expectation that those
> will not be changed in transit, except perhaps by the local MTA which as
> discussed is fine.

If done properly then probably the 'right' thing to do would be to
encapsulate the whole message, with authenticated headers, inside a
message/rfc822 container, and sign that. But that would be too intrusive for
day one.

> And yes, we should also include some form of fuzzy hash of the mail. As
> a primitive example, we could perhaps specify the number of lines which
> were present when the mail was signed, and then the receiver knows how
> much to trim from the bottom. 

A bit too easy to replay-attack. Just take the first N lines of a message to
a mailing list, say, and paste your spam at the bottom.

I actually find it hard to conceive of a value of N which would be useful.
Many messages are only 2 lines long; after that you get the extra footers
(virus-sweeper, mailing list info, etc)

The value of N could be included in the signature itself: i.e. "this message
was originally N lines long, and the hash covers those N lines; anything
after that is not signed". But the replay-and-add-spam attack is too easy.

> We probably also need to canonicalise the character set and encoding;
> transition between systems supporting 8BITMIME and not will cause
> breakage otherwise. 

I guess already addressed by OpenGPG etc. Or maybe they just base64 encode
the lot, which is not a friendly thing to do for us.

Regards,

Brian.

From dwmw2 at infradead.org  Tue Feb 24 14:31:00 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Tue Feb 24 14:31:06 2004
Subject: Initial thoughts.
In-Reply-To: <20040224133052.GA4716@uk.tiscali.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	 <1077622201.7826.219.camel@hades.cambridge.redhat.com>
	 <20040224133052.GA4716@uk.tiscali.com>
Message-ID: <1077633059.7826.425.camel@hades.cambridge.redhat.com>

On Tue, 2004-02-24 at 13:30 +0000, Brian Candler wrote:
> 2. it's maliciously incorrect (a joe job pointing to someone else), in which
> case it falls under the different problem case (1), i.e. I don't want to
> accept bounces to mail I didn't send; or

Indeed you don't. That's nice for you. Now _I_ am stuck with an
undeliverable bounce because you quite rightly don't want to accept it.

> I think the SPF people are counting on some sort of equilibrium: say only 5%
> of the Internet adopts SPF, so little that spammers won't care. But they may
> get some benefit if, say, spammers send mail from foo@aol.com but aol.com is
> declaring information in SPF.

They're on crack if they think this, since SPF can only be workable _if_
the whole of the Internet works around its brokenness by implementing
SRS.

In the case of SPF, I agree that it's pointless. Since by definition if
it ever becomes workable, it becomes pointless. Other schemes are
different, in that they don't _need_ critical mass in order to help, and
hence aren't obsolete by the time they're useful.

> But this is an unstable equilibrium. If SPF does actually deliver some
> benefit, then everyone will want it, at which point the spammers will bypass
> it. Personally I think the SRS-signing of outgoing mail as you have
> implemented gives much greater benefit to the implementor, immediately, at
> very little cost.

To the implementor yes, but not to the poor sods out there who don't
bother to _check_ with me whether that's a valid return-path. 

That's their choice -- and it's yours. I like to check, though.

> > But these are policy decisions to be made locally. I'm sure you agree
> > that it's worthwhile to be able to verify the envelope sender
> 
> Actually, no :-) See above.

Fine. But we are still digressing from the point we _do_ agree on...

> Or else you put in a brand new header, say "X-Signed-From:", which includes
> the crypto signature.

That's basically what I was suggesting, surely?

> Any MUA which understands it can use it. Any final end-point MTA can compare
> the signed address with the From: address, and modify the From: address
> appropriately:
> 
>     From: "Dubya Bush" <president@whitehouse.gov>
> becomes:
>     From:  "Dubya Bush [PROBABLY FORGED, claimed president@whitehouse.gov, real sender spam123@isp.net]" <spam123@isp.net>
> 
> The sending MTA can add a header saying "real sender was spam123@isp.net"
> without touching the original header.

I wouldn't recommend editing the From: address. I think RFC2821 says you
MUST NOT. It will suffice to add a warning header that a MUA can look
for if it wants, or which procmail can use to dump the message into a
spam box.

> But as you say, you have to prevent replay (i.e. cut-and-paste of the signed
> header), which means the header has to include, say, a hash of the entire
> message body, and the message must be in a form which is not likely to be
> corrupted in transit otherwise the hash is invalidated.

Yes.

> A review of existing work in OpenPGP, S/MIME etc would be useful here I
> think, because they must have exactly the same problem.

They make requirements about the MIME parts (like restricting to 7 bits)
which I don't think we can make in the general case. They don't handle
munging of text parts either.

You're right that we must avoid reinventing the wheel. As it is, I don't
think either of those wheels will fit. I'm not completely familiar with
S/MIME though -- I only glanced at the RFCs this morning for inspiration
and to assess reusability.

> There are different levels of trust which could be indicated:
> 
> - a mail from "b.candler@pobox.com", sent via myisp.net, could be signed via
> myisp.net saying "this mail came through me, but I cannot vouch for the
> sender; the real sender could be be anyone@isp.net"

I don't see what this achieves. If the host in question is clueful, its
Received: headers are already enough, surely?

> - a mail from "random@myisp.net", sent via myisp.net, could be signed as
> "this mail came through me, and it's one of my domains, the RHS may be right
> but I can't vouch for the LHS of the address" (this is roughly what SPF
> attempts)
> 
> - a mail from "random@myisp.net", where the user authenticated with SMTP
> AUTH and with a username which matched that address, could be signed "this
> mail came through me and I vouch for the authenticity of the sender"

I _can_ see the point in making a distinction between the above two
cases, but I wonder if it's really _necessary_. Any proposal has to be
lightweight and we must avoid featuritis.

> - a mail from "b.candler@pobox.com", where I authenticated with SMTP AUTH as
> brian@myisp.net" could be signed "this mail came through me from the user
> brian@myisp.net, but I cannot vouch for the sender address given"

According to RFC2822, in this case you should have brian@myisp.net in
the Sender: header, and b.candler@pobox.com in the From: header. 

I wonder if it's worth extending the scheme to allow verification of
Sender: addresses as well as From: addresses; I suspect that's another
example of featuritis though. 

> Such policies could allow existing practices to continue working, and for an
> address which cannot be fully vouched for (which would be the vast majority
> in the early days), you at least get _some_ information about the sender or
> the domain.

I think we could address this differently. The pubkey-in-DNS scheme
should allow you to specify different pubkeys on a per-user basis if you
want to. That way, the user himself can add the signatures using his
_own_ private key, and the ISP-de-jour doesn't have to be involved.

That'll be by _far_ the most common case, surely?

> Then it's a logical extension to include the IP address. This is the
> information which the ISP needs to trace the original user if they did not
> authenticate with SMTP AUTH. A spam complaint containing such information
> could be processed in a highly automated fashion, AND provide strong
> evidence (e.g. if needed in a court of law) for kicking off the spammer.

It's in the Received: headers already. Outside our scope for today. 

> Hmm. Let me put it another way. Suppose in our smarthost MTA we see a mail
> which says
>    From: president@whitehouse.gov
> 
> but whitehouse.gov has not published any key information, so we know the
> recipient won't be able to verify it one way or the other.
> 
> I think it would be better for us to sign it saying "This mail probably came
> from *@myisp.net", than not sign it at all. The receiver then has something
> to verify (using the myisp.net key in the DNS) and show to the end-user.

But it means nothing. It's like signing a memo you found on your desk
without reading it, just to say "I saw this and it really did exist".

I disagree that such a signature would be useful.

> Since the certification disagrees with the originally-provided From: header,
> the end-user has a better indication of a forgery than if no signature were
> applied, even though whitehouse.gov has not published anything.

Why a better indication of a forgery? No information is available as to
whether this could be valid or not. Do you think it _is_ or it _isn't_
if you find such a signature? I don't see how any information is gained.

> > It's hardly
> > difficult to see where real Received: headers stop and the faked ones
> > start; we don't need to address this here.
> 
> Not for experts like you or me. But there are plenty of people with enough
> knowledge to find full headers, but not enough to understand them, who send
> spam complaints to the wrong places (based on forged HELO names, forged
> Received: headers, and forged envelope senders). At least one person on
> spf-discuss was arguing that as a case for validating HELO names.

Nothing will fix these people apart from a cluebat. Smell the coffee :)
Seriously, the Received: headers will always have to exists; people will
always misinterpret them. If you think such a scheme would help in any
but the most trivial amount, then I respectfully submit that you're
crazy. And, given your cynicism about even sender-verification,
inconsistently optimistic. :)


> Sure. I meant if we used an existing mechanism like OpenGPG. What if we come
> across a mail which is already in OpenGPG format?

We won't use GPG, because it's too obvious to non-participating users.
It would never get implemented. 

But if we see a message which is already signed with _our_ scheme, it is
a local policy decision as to whether we re-sign it or leave the
original signature to stand. Why should the RFC make this decision?

I think it's OK to allow only _one_ signature to exist for any given
mail. It gets complicated otherwise.

> > If we _do_ have a per-user key scheme where the end-user has signed a
> > message using their own key, why would it need signing by the ISP too?
> 
> We need to add some level of the ISP 'vouching for' the user - which, unless
> the end-user's key has been signed by the ISP, means signing the message
> somehow.

The ISP vouches for the user by listing the appropriate pubkey in the
DNS for that user@isp-domain. Isn't that sufficient?

> Otherwise, we are unable to add any trust for an existing GPG message.
> Spammers can always start sending OpenGPG-signed mails using throwaway keys.

True. The DNS (or whatever) must list the keys which individual users
are using, if signing is done by individual users.

> If we treat OpenGPG as just a lump of MIME, and add our own signature in the
> top headers, this should be fine.

Yep.

> > > Then you have to consider how an end-point will verify the signatures. If
> > > it's an MTA, then it can make a lookup in the DNS in real-time. However if
> > > it's an off-line MUA, which has downloaded the mail using POP3 say, then
> > > either it must trust the final MTA to stamp the message somehow, or it must
> > > be able to verify the signatures off-line (which means a chain of trust up
> > > to some root certificates)
> > 
> > This should be left as flexible as possible, to allow sites to implement
> > their own policy. It should be possible to do _either_. 
> 
> A good goal.
> 
> > Verifying the signatures in the MUA can still do the DNS lookups to get
> > the pubkey, though, surely? Just like SpamAssassin can do RBL and SPF
> > lookups. You're normally online at the time you _fetch_ the mail, or you
> > wouldn't be able to talk to the mail server.
> 
> Not if you're offline (i.e. disconnected from the Internet)

Then you're unlikely to be fetching your mail, surely? I think we can
declare this an uninteresting sub-problem.

	"How do you look up keys in the public database while offline"
	"You don't."

> If done properly then probably the 'right' thing to do would be to
> encapsulate the whole message, with authenticated headers, inside a
> message/rfc822 container, and sign that. But that would be too intrusive for
> day one.

Yes. That's the _technical_ reason we're not using GPG or S/MIME.

> > And yes, we should also include some form of fuzzy hash of the mail. As
> > a primitive example, we could perhaps specify the number of lines which
> > were present when the mail was signed, and then the receiver knows how
> > much to trim from the bottom. 
> 
> A bit too easy to replay-attack. Just take the first N lines of a message to
> a mailing list, say, and paste your spam at the bottom.

Hence the suggested policy of rejecting a mail where there's more than a
very few lines added to the original NLINES. Also bear in mind that the
date was included in the signature, so you can place a time limit on
replay attacks. Again up to local policy.

> I actually find it hard to conceive of a value of N which would be useful.
> Many messages are only 2 lines long; after that you get the extra footers
> (virus-sweeper, mailing list info, etc)
> 
> The value of N could be included in the signature itself: i.e. "this message
> was originally N lines long, and the hash covers those N lines; anything
> after that is not signed". But the replay-and-add-spam attack is too easy.

Yes, that's what I suggested. You include the number of lines in the
headers, signed for authenticity. 

> > We probably also need to canonicalise the character set and encoding;
> > transition between systems supporting 8BITMIME and not will cause
> > breakage otherwise. 
> 
> I guess already addressed by OpenGPG etc. Or maybe they just base64 encode
> the lot, which is not a friendly thing to do for us.

Indeed.


-- 
dwmw2


From B.Candler at pobox.com  Tue Feb 24 14:59:14 2004
From: B.Candler at pobox.com (Brian Candler)
Date: Tue Feb 24 14:59:42 2004
Subject: Initial thoughts.
In-Reply-To: <1077633059.7826.425.camel@hades.cambridge.redhat.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	<1077622201.7826.219.camel@hades.cambridge.redhat.com>
	<20040224133052.GA4716@uk.tiscali.com>
	<1077633059.7826.425.camel@hades.cambridge.redhat.com>
Message-ID: <20040224145914.GA5037@uk.tiscali.com>

On Tue, Feb 24, 2004 at 02:31:00PM +0000, David Woodhouse wrote:
> On Tue, 2004-02-24 at 13:30 +0000, Brian Candler wrote:
> > 2. it's maliciously incorrect (a joe job pointing to someone else), in which
> > case it falls under the different problem case (1), i.e. I don't want to
> > accept bounces to mail I didn't send; or
> 
> Indeed you don't. That's nice for you. Now _I_ am stuck with an
> undeliverable bounce because you quite rightly don't want to accept it.

Then you just discard it. (Exim's default is to freeze double-bounces; first
thing you do on a new exim install is to get it to drop them silently. So
much mail on our smarthosts was bounces to non-existent addresses..)

Yep, I agree that if you could have validated the sender first, it wouldn't
have happened. The only place that can happen usefully is at the ISP's
smarthost though.

If you receive a (deliverable) mail, but refuse to accept it because you
have determined that the return address is invalid, which is an accidental
misconfiguration, then:
- no mail will be delivered, and
- no bounce will be delivered (at best you return 550 to the sending ISP's
  smarthost, which is then left holding the baby and cannot do anything
  with it)

I think think that's a worse situation than just delivering the mail in the
first place.

> > But this is an unstable equilibrium. If SPF does actually deliver some
> > benefit, then everyone will want it, at which point the spammers will bypass
> > it. Personally I think the SRS-signing of outgoing mail as you have
> > implemented gives much greater benefit to the implementor, immediately, at
> > very little cost.
> 
> To the implementor yes, but not to the poor sods out there who don't
> bother to _check_ with me whether that's a valid return-path. 
> 
> That's their choice -- and it's yours. I like to check, though.

Sure. But there are two halves to the equation:
- implement signing of outgoing mails, and bounce validation
- implement callback sender verification on incoming mails

If you implement either one or both of these, you gain immediate benefit. If
the people you are communicating with implement them too, you gain even more
benefit. And you don't have to wait for the rest of the Internet to
implement them before you benefit.

> I wouldn't recommend editing the From: address. I think RFC2821 says you
> MUST NOT. It will suffice to add a warning header that a MUA can look
> for if it wants, or which procmail can use to dump the message into a
> spam box.

OK. End-point policy decision, anyway.

Perhaps in extreme cases, where the sender is known to be very likely to be
forged, the receiving MTA can wrap the entire message inside a
message/rfc822, with a disclaimer on the outside saying that the From:
address is bad.

> > "this mail came through me, and it's one of my domains, the RHS may be right
> > but I can't vouch for the LHS of the address" (this is roughly what SPF
> > attempts)
> > 
> > - a mail from "random@myisp.net", where the user authenticated with SMTP
> > AUTH and with a username which matched that address, could be signed "this
> > mail came through me and I vouch for the authenticity of the sender"
> 
> I _can_ see the point in making a distinction between the above two
> cases, but I wonder if it's really _necessary_. Any proposal has to be
> lightweight and we must avoid featuritis.

The first is easy to implement now, but doesn't actually validate the sender
(much). The second is a much more desirable position of having a
fully-validated sender, which ought to be a long-term goal.

> > - a mail from "b.candler@pobox.com", where I authenticated with SMTP AUTH as
> > brian@myisp.net" could be signed "this mail came through me from the user
> > brian@myisp.net, but I cannot vouch for the sender address given"
> 
> According to RFC2822, in this case you should have brian@myisp.net in
> the Sender: header, and b.candler@pobox.com in the From: header. 

Sure. Sender: headers can be forged too, of course. Effectively this would
be a crypto equivalent of the Sender: header though.

> > Such policies could allow existing practices to continue working, and for an
> > address which cannot be fully vouched for (which would be the vast majority
> > in the early days), you at least get _some_ information about the sender or
> > the domain.
> 
> I think we could address this differently. The pubkey-in-DNS scheme
> should allow you to specify different pubkeys on a per-user basis if you
> want to. That way, the user himself can add the signatures using his
> _own_ private key, and the ISP-de-jour doesn't have to be involved.
> 
> That'll be by _far_ the most common case, surely?

The ISP would have to be involved by publishing the end-user's public key in
the DNS somehow, e.g. under  "b.candler._at.pobox.com."

(or else by signing the end-user's key, as mentioned before)

The ISP probably does not want to be responsible for holding the user's
*private* key, in which case the end-user's MUA must be responsible for
signing the message. Here we move away from an ISP-centric solution to an
end-user-centric solution, which has defied widespread acceptance to date.

> > Hmm. Let me put it another way. Suppose in our smarthost MTA we see a mail
> > which says
> >    From: president@whitehouse.gov
> > 
> > but whitehouse.gov has not published any key information, so we know the
> > recipient won't be able to verify it one way or the other.
> > 
> > I think it would be better for us to sign it saying "This mail probably came
> > from *@myisp.net", than not sign it at all. The receiver then has something
> > to verify (using the myisp.net key in the DNS) and show to the end-user.
> 
> But it means nothing. It's like signing a memo you found on your desk
> without reading it, just to say "I saw this and it really did exist".
> 
> I disagree that such a signature would be useful.

I will think about it some more, but it seems useful to me, particularly in
the early stages where otherwise early adopters gain nothing.

> > Since the certification disagrees with the originally-provided From: header,
> > the end-user has a better indication of a forgery than if no signature were
> > applied, even though whitehouse.gov has not published anything.
> 
> Why a better indication of a forgery? No information is available as to
> whether this could be valid or not. Do you think it _is_ or it _isn't_
> if you find such a signature? I don't see how any information is gained.

If whitehouse.gov does not participate in this scheme, then mails you
receive which say 'From: president@whitehouse.gov' may:

(1) be genuine, coming through the whitehouse.gov systems
(2) be invalid, coming through some other system which supports crypto
signing
(3) be invalid, coming through some other system which does not yet support
our nice crypto signing scheme

If we sign such mails, we can identify case (2), even if otherwise we cannot
distinguish between cases (1) and (3). Some forgeries may be detected. If
our system became popular, case (3) would be squeezed out.

It's essentially an ISP making a negative assertion: I do *not* handle mail
for whitehouse.gov.

> > > If we _do_ have a per-user key scheme where the end-user has signed a
> > > message using their own key, why would it need signing by the ISP too?
> > 
> > We need to add some level of the ISP 'vouching for' the user - which, unless
> > the end-user's key has been signed by the ISP, means signing the message
> > somehow.
> 
> The ISP vouches for the user by listing the appropriate pubkey in the
> DNS for that user@isp-domain. Isn't that sufficient?

Yes it would be. This wasn't discussed until now.

The problem is that either we support keys and signing for all the various
existing mechanisms, or we just implement a new mechanism which can sit on
top of all of them anyway. If we do that, we don't care that it's already
been GPG-signed, say.

> > > Verifying the signatures in the MUA can still do the DNS lookups to get
> > > the pubkey, though, surely? Just like SpamAssassin can do RBL and SPF
> > > lookups. You're normally online at the time you _fetch_ the mail, or you
> > > wouldn't be able to talk to the mail server.
> > 
> > Not if you're offline (i.e. disconnected from the Internet)
> 
> Then you're unlikely to be fetching your mail, surely? I think we can
> declare this an uninteresting sub-problem.
> 
> 	"How do you look up keys in the public database while offline"
> 	"You don't."

So I guess you mean, remove the goal that signatures can be checked offline.

Cheers,

Brian.

From dwmw2 at infradead.org  Tue Feb 24 15:13:20 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Tue Feb 24 15:13:24 2004
Subject: sender verification
In-Reply-To: <20040224145914.GA5037@uk.tiscali.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	 <1077622201.7826.219.camel@hades.cambridge.redhat.com>
	 <20040224133052.GA4716@uk.tiscali.com>
	 <1077633059.7826.425.camel@hades.cambridge.redhat.com>
	 <20040224145914.GA5037@uk.tiscali.com>
Message-ID: <1077635599.7826.464.camel@hades.cambridge.redhat.com>

On Tue, 2004-02-24 at 14:59 +0000, Brian Candler wrote:
> Then you just discard it. (Exim's default is to freeze double-bounces; first
> thing you do on a new exim install is to get it to drop them silently. So
> much mail on our smarthosts was bounces to non-existent addresses..)

Site policy here is different. I like to _see_ frozen messages, and I
have freeze_tell set accordingly. Frozen messages occasionally alert me
to a _real_ problem.

Your site policy is fine by me. I have no objections; I prefer to
configure my system differently though.

> If you receive a (deliverable) mail, but refuse to accept it because you
> have determined that the return address is invalid, which is an accidental
> misconfiguration, then:
> - no mail will be delivered, and
> - no bounce will be delivered (at best you return 550 to the sending ISP's
>   smarthost, which is then left holding the baby and cannot do anything
>   with it)
> 
> I think think that's a worse situation than just delivering the mail in the
> first place.

Site policy here is different. I think it's better to reject early,
because if it stays on the _sender's_ queue, that's one hop closer to
the actual misconfiguration, and much more likely to get noticed by
someone who can actually fix it.

And from the converse POV, if I break something I'd _much_ rather have
my outgoing mail rejected at delivery time than accepted and
subsequently dropped.

But we really have digressed.

You won't convince me to stop, and I won't convince you to start -- and
I don't think either of us is actually _trying_ to achieve that :)

> Sure. But there are two halves to the equation:
> - implement signing of outgoing mails, and bounce validation
> - implement callback sender verification on incoming mails
> 
> If you implement either one or both of these, you gain immediate benefit. If
> the people you are communicating with implement them too, you gain even more
> benefit. And you don't have to wait for the rest of the Internet to
> implement them before you benefit.

Absolutely. 


-- 
dwmw2


From dwmw2 at infradead.org  Tue Feb 24 15:41:40 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Tue Feb 24 15:41:44 2004
Subject: From: (and maybe Sender:) header authentication
In-Reply-To: <20040224145914.GA5037@uk.tiscali.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	 <1077622201.7826.219.camel@hades.cambridge.redhat.com>
	 <20040224133052.GA4716@uk.tiscali.com>
	 <1077633059.7826.425.camel@hades.cambridge.redhat.com>
	 <20040224145914.GA5037@uk.tiscali.com>
Message-ID: <1077637299.7826.496.camel@hades.cambridge.redhat.com>

On Tue, 2004-02-24 at 14:59 +0000, Brian Candler wrote:
> > I wouldn't recommend editing the From: address. I think RFC2821 says you
> > MUST NOT. It will suffice to add a warning header that a MUA can look
> > for if it wants, or which procmail can use to dump the message into a
> > spam box.
> 
> OK. End-point policy decision, anyway.
> 
> Perhaps in extreme cases, where the sender is known to be very likely to be
> forged, the receiving MTA can wrap the entire message inside a
> message/rfc822, with a disclaimer on the outside saying that the From:
> address is bad.

Or just drop/reject it. Policy again. We can leave a _lot_ to local
policy, although we should perhaps offer guidelines. What I'm really
interested in providing is the _mechanism_ whereby we can sign such
email to authenticate the address(es) in the From: (and/or Sender:)
header.


> > > "this mail came through me, and it's one of my domains, the RHS may be right
> > > but I can't vouch for the LHS of the address" (this is roughly what SPF
> > > attempts)
> > > 
> > > - a mail from "random@myisp.net", where the user authenticated with SMTP
> > > AUTH and with a username which matched that address, could be signed "this
> > > mail came through me and I vouch for the authenticity of the sender"
> > 
> > I _can_ see the point in making a distinction between the above two
> > cases, but I wonder if it's really _necessary_. Any proposal has to be
> > lightweight and we must avoid featuritis.
> 
> The first is easy to implement now, but doesn't actually validate the sender
> (much). The second is a much more desirable position of having a
> fully-validated sender, which ought to be a long-term goal.

Yeah, that's true. Perhaps, then, the pubkeys listed in the DNS should
be accompanied by a 'trust level'. We might define 'domain' and 'user'
as the only two trust levels, for now. 

It's up to the MUA how this is represented to the user -- you might want
to display the From: addresses in a different colour, for example:
	Red: Should have been signed but wasn't.
	Green: Signed with 'user' key.
	Blue: Signed with 'domain' key.
	Black: Unsigned, shouldn't have been.

> > > - a mail from "b.candler@pobox.com", where I authenticated with SMTP AUTH as
> > > brian@myisp.net" could be signed "this mail came through me from the user
> > > brian@myisp.net, but I cannot vouch for the sender address given"
> > 
> > According to RFC2822, in this case you should have brian@myisp.net in
> > the Sender: header, and b.candler@pobox.com in the From: header. 
> 
> Sure. Sender: headers can be forged too, of course. Effectively this would
> be a crypto equivalent of the Sender: header though.

Yeah. As the risk of suffering from featuritis, we could also allow for
signing according to the addresses in the Sender: header, as well as the
addresses in the From: header. You'd end up with a parallel set of
headers, allowing two or more signatures to coexists.

Come to think of it, you have to allow for that anyway since a mail can
have more than one From: address. Let's put the signing address (or just
the domain if appropriate) into the Content-Hash- headers too, and that
way multiple such headers can coexist.

> The ISP would have to be involved by publishing the end-user's public key in
> the DNS somehow, e.g. under  "b.candler._at.pobox.com."
> 
> (or else by signing the end-user's key, as mentioned before)

The former is simpler -- verifying key signatures can get expensive,
especially if there's a long chain of signatures to be verified due to
multiple delegations. I think I'd prefer that unless there's a real need
to handle such delegation.

> The ISP probably does not want to be responsible for holding the user's
> *private* key, in which case the end-user's MUA must be responsible for
> signing the message. Here we move away from an ISP-centric solution to an
> end-user-centric solution, which has defied widespread acceptance to date.

If the ISP is doing SMTP AUTH and actually checking the _user_, or is
verifying usernames against actual live dialup records, the ISP _can_
use a key which has 'user' trust level. In fact the ISP can have two
keys for the same domain -- one with each trust level -- and use
whichever is appropriate for each mail.

We need to allow _either_ ISP-centric or user-centric signing. 

> It's essentially an ISP making a negative assertion: I do *not* handle mail
> for whitehouse.gov.

The ISP has no reason to state such a thing. That's the assumption that
SPF makes, and it's _wrong_. The ISP in question _does_ validly handle
mail for whitehouse.gov, if <monica@whitehouse.gov> happens to dial up
today and try sending mail, or if she sends mail to a user at $ISP who
happens to have a .forward file.

> > The ISP vouches for the user by listing the appropriate pubkey in the
> > DNS for that user@isp-domain. Isn't that sufficient?
> 
> Yes it would be. This wasn't discussed until now.

Sorry, I certainly _thought_ it quite hard. I appreciate that method of
communication leaves a little to be desired though :)

> The problem is that either we support keys and signing for all the various
> existing mechanisms, or we just implement a new mechanism which can sit on
> top of all of them anyway. If we do that, we don't care that it's already
> been GPG-signed, say.

Yes. I think we need to implement a new mechanism. Something basically
like the one I documented at the bottom (after the ================) of
http://lists.infradead.org/pipermail/sender-auth/2004-February/000002.html


> So I guess you mean, remove the goal that signatures can be checked offline.

I think that's reasonable, yes. After all, you don't really want to do
the verification more than once at the time you receive the mail, do
you? And you're likely to be online when you _receive_ the mail.


-- 
dwmw2


From B.Candler at pobox.com  Tue Feb 24 15:48:12 2004
From: B.Candler at pobox.com (Brian Candler)
Date: Tue Feb 24 15:52:29 2004
Subject: Initial thoughts.
In-Reply-To: <1077633059.7826.425.camel@hades.cambridge.redhat.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	<1077622201.7826.219.camel@hades.cambridge.redhat.com>
	<20040224133052.GA4716@uk.tiscali.com>
	<1077633059.7826.425.camel@hades.cambridge.redhat.com>
Message-ID: <20040224154812.GA5116@uk.tiscali.com>

On Tue, Feb 24, 2004 at 02:31:00PM +0000, David Woodhouse wrote:
> > - a mail from "b.candler@pobox.com", sent via myisp.net, could be signed via
> > myisp.net saying "this mail came through me, but I cannot vouch for the
> > sender; the real sender could be be anyone@isp.net"
> 
> I don't see what this achieves. If the host in question is clueful, its
> Received: headers are already enough, surely?

In theory yes.

I categorise this discussion under the heading "what do people think they
will gain from SPF?". If we understand what people think they will gain from
SPF, we can perhaps offer them a better solution.

The main reason for SPF I've seen on the list is: "I don't want my domain to
be abused". I translate this as: "I don't want my domain to appear in the
Return-Path: header or Received: headers in spam." They see SPF as a way to
stop their domain name being used in MAIL FROM and HELO respectively. Why do
they want this? Presumably because they receive misdirected complaints based
on their domain appearing in those places.

But: these are *obvious* forgeries. Anyone who knew what they were talking
about would look at Received: headers only, and only the parts which can be
trusted (i.e. ignoring HELO names provided by remote hosts).

So the only benefit which SPF gives is to give better quality information in
these headers, so you don't have to sort the wheat from the chaff manually.

IF that's what they want, THEN structuring and/or crypto-signing that
information is a better solution than attempting to validate MAIL FROM and
HELO names based on the IP address of the person connecting to you, a la
SPF.

> > Hmm. Let me put it another way. Suppose in our smarthost MTA we see a mail
> > which says
> >    From: president@whitehouse.gov
> > 
> > but whitehouse.gov has not published any key information, so we know the
> > recipient won't be able to verify it one way or the other.
> > 
> > I think it would be better for us to sign it saying "This mail probably came
> > from *@myisp.net", than not sign it at all. The receiver then has something
> > to verify (using the myisp.net key in the DNS) and show to the end-user.
> 
> But it means nothing. It's like signing a memo you found on your desk
> without reading it, just to say "I saw this and it really did exist".

Well, at least you said "this memo has been on my desk". Someone *could*
attach some importance to that fact, depending on whose desk it is, and how
hard it is to get something onto it.

Let's face it, if you are just checking the RHS of a domain (like SPF or the
system we're discussing, without SMTP AUTH), you are saying no more than
this:
1. this mail has been through myisp.net's mailserver
2. its RHS domain happens to be one of the domains which myisp.net handles

A large ISP can easily have 10,000 domains handled for its customers. In
that case, "this mail has been through myisp.net's mailserver" is actually
the main piece of information.

I had another thought. Given the information "this mail has been through
myisp.net's mailserver", a hybrid SPF-like system might become feasible,
because you've eliminated the forwarding problem.

i.e. (1) outgoing mail is signed "I've been through myisp.net"
     (2) domain foo.com declares in SPF "mail can originate from myisp.net"
         (rather than "mail can originate from IP addresses X,Y,Z")

Doesn't really gain you much, except that the outgoing mail relay needs no
intelligence (it doesn't need a database of which domains to sign). That
information goes back into the DNS. It does hand some control back to the
domain owner though; if I own a domain, and decide I am going to send mail
via isp1.net at work and isp2.net at home, I can declare in the DNS that
fact.

As for the problem of mails being modified and resent: the alternative
solution is to make the signatures time-expire, like SRS, and check them at
receipt time only. Much less strong because they could be captured and
re-used for a short period of course, but at least not subject to the SPF
forwarding problem. But then, SRS and callbacks achieve the same without any
public key infrastructure at all.

Regards,

Brian.

From B.Candler at pobox.com  Tue Feb 24 15:57:55 2004
From: B.Candler at pobox.com (Brian Candler)
Date: Tue Feb 24 16:26:36 2004
Subject: From: (and maybe Sender:) header authentication
In-Reply-To: <1077637299.7826.496.camel@hades.cambridge.redhat.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	<1077622201.7826.219.camel@hades.cambridge.redhat.com>
	<20040224133052.GA4716@uk.tiscali.com>
	<1077633059.7826.425.camel@hades.cambridge.redhat.com>
	<20040224145914.GA5037@uk.tiscali.com>
	<1077637299.7826.496.camel@hades.cambridge.redhat.com>
Message-ID: <20040224155755.GA5201@uk.tiscali.com>

On Tue, Feb 24, 2004 at 03:41:40PM +0000, David Woodhouse wrote:
> > The ISP probably does not want to be responsible for holding the user's
> > *private* key, in which case the end-user's MUA must be responsible for
> > signing the message. Here we move away from an ISP-centric solution to an
> > end-user-centric solution, which has defied widespread acceptance to date.
> 
> If the ISP is doing SMTP AUTH and actually checking the _user_, or is
> verifying usernames against actual live dialup records, the ISP _can_
> use a key which has 'user' trust level. In fact the ISP can have two
> keys for the same domain -- one with each trust level -- and use
> whichever is appropriate for each mail.

[Aside: I don't see the need for two keys. Just a one-bit flag within the
signed message is all that's needed]

> We need to allow _either_ ISP-centric or user-centric signing. 

Sure. But then we are definitely talking about two separate mechanisms:

- the user signs with their own key
- the ISP signs with their user-signing key

because those two keys would be found in two different places in the DNS.

> I think that's reasonable, yes. After all, you don't really want to do
> the verification more than once at the time you receive the mail, do
> you? And you're likely to be online when you _receive_ the mail.

Some people still live in a uucp or fido world, but thankfully not very many
:-)

Regards,

Brian.

From dwmw2 at infradead.org  Tue Feb 24 16:40:02 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Tue Feb 24 16:40:05 2004
Subject: HELO verification
In-Reply-To: <20040224154812.GA5116@uk.tiscali.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	 <1077622201.7826.219.camel@hades.cambridge.redhat.com>
	 <20040224133052.GA4716@uk.tiscali.com>
	 <1077633059.7826.425.camel@hades.cambridge.redhat.com>
	 <20040224154812.GA5116@uk.tiscali.com>
Message-ID: <1077640801.7826.566.camel@hades.cambridge.redhat.com>

On Tue, 2004-02-24 at 15:48 +0000, Brian Candler wrote:
> I categorise this discussion under the heading "what do people think they
> will gain from SPF?". If we understand what people think they will gain from
> SPF, we can perhaps offer them a better solution.

OK.... I haven't been aware of much discussion about SPF verifying
_HELO_ addresses, although I was aware of the possibility of using it
when the sender is empty. Actually SPF as a way of limiting _HELO_,
rather than MAIL FROM:<> is a relatively _sane_ plan. Except of course
it'd be called HPF :)

> But: these are *obvious* forgeries. Anyone who knew what they were talking
> about would look at Received: headers only, and only the parts which can be
> trusted (i.e. ignoring HELO names provided by remote hosts).
> 
> So the only benefit which SPF gives is to give better quality information in
> these headers, so you don't have to sort the wheat from the chaff manually.
> 
> IF that's what they want, THEN structuring and/or crypto-signing that
> information is a better solution than attempting to validate MAIL FROM and
> HELO names based on the IP address of the person connecting to you, a la
> SPF.

I really think you'll never manage to make it obvious enough that it
actually improves the behaviour of random dimwits out there. Show a
concrete proposal which disproves my scepticism and I may change my
mind.

> Well, at least you said "this memo has been on my desk". Someone *could*
> attach some importance to that fact, depending on whose desk it is, and how
> hard it is to get something onto it.

OK, OK. It's just another 'trust level' I suppose.

> Let's face it, if you are just checking the RHS of a domain (like SPF or the
> system we're discussing, without SMTP AUTH), you are saying no more than
> this:
> 1. this mail has been through myisp.net's mailserver
> 2. its RHS domain happens to be one of the domains which myisp.net handles
> 
> A large ISP can easily have 10,000 domains handled for its customers. In
> that case, "this mail has been through myisp.net's mailserver" is actually
> the main piece of information.

Assuming that ISP does nothing to prevent the users from forging each
others' addresses, yes.

> I had another thought. Given the information "this mail has been through
> myisp.net's mailserver", a hybrid SPF-like system might become feasible,
> because you've eliminated the forwarding problem.
> 
> i.e. (1) outgoing mail is signed "I've been through myisp.net"
>      (2) domain foo.com declares in SPF "mail can originate from myisp.net"
>          (rather than "mail can originate from IP addresses X,Y,Z")
> 
> Doesn't really gain you much, except that the outgoing mail relay needs no
> intelligence (it doesn't need a database of which domains to sign). That
> information goes back into the DNS. It does hand some control back to the
> domain owner though; if I own a domain, and decide I am going to send mail
> via isp1.net at work and isp2.net at home, I can declare in the DNS that
> fact.

You're speaking of something akin to the SPF 'include' mechanism. Yes,
we want a fairly flexible way of specifying, for any given
domain/address, what keys should be considered acceptable. 

I don't really consider that to be an interesting problem. Feel free to
fill in the blanks :)

> As for the problem of mails being modified and resent: the alternative
> solution is to make the signatures time-expire, like SRS, and check them at
> receipt time only. 

You can limit the amount of modification you permit -- as a local
policy. You can also limit the amount of time after the date in the
signed Date: header which you'll allow. 

I think that has to be enough -- I don't think we can do better.


-- 
dwmw2


From sethg at GoodmanAssociates.com  Tue Feb 24 14:45:23 2004
From: sethg at GoodmanAssociates.com (Seth Goodman)
Date: Tue Feb 24 20:45:35 2004
Subject: Initial thoughts.
In-Reply-To: <20040224154812.GA5116@uk.tiscali.com>
Message-ID: <MHEGIFHMACFNNIMMBACAEEAOHIAA.sethg@GoodmanAssociates.com>

> {Brian Candler]
> I had another thought. Given the information "this mail has
> been through
> myisp.net's mailserver", a hybrid SPF-like system might
> become feasible,
> because you've eliminated the forwarding problem.
>
> i.e. (1) outgoing mail is signed "I've been through myisp.net"
>      (2) domain foo.com declares in SPF "mail can originate
> from myisp.net"
>          (rather than "mail can originate from IP addresses X,Y,Z")
>
> Doesn't really gain you much, except that the outgoing mail
> relay needs no
> intelligence (it doesn't need a database of which domains to
> sign). That
> information goes back into the DNS. It does hand some control
> back to the
> domain owner though; if I own a domain, and decide I am going
> to send mail
> via isp1.net at work and isp2.net at home, I can declare in
> the DNS that
> fact.
>
> As for the problem of mails being modified and resent: the alternative
> solution is to make the signatures time-expire, like SRS, and
> check them at
> receipt time only. Much less strong because they could be captured and
> re-used for a short period of course, but at least not
> subject to the SPF
> forwarding problem. But then, SRS and callbacks achieve the
> same without any
> public key infrastructure at all.

I tend to like this proposal for a couple of reasons, but I want to
express a couple of thoughts before I say why.

Here is my own personal list of possible goals for something better than
the current SPF+SRS, some of which were already listed by others.  Some
of these are easier than others, some are already done by other means
and some we can live without.

1) Permit a gateway SMTP-recipient to detect and reject as many sender
forgeries as possible before DATA (authenticate original sender at the
receiving end).

2) Facilitate a gateway SMTP-sender to detect and reject as many sender
forgeries as possible.

3) Permit a gateway SMTP-recipient to detect and reject as many DSN
forgeries as possible before DATA (reject joe-jobs at the receiving
end).

4) Permit a gateway SMTP-recipient (or perhaps the user MUA?) to detect
and reject certain header forgeries after DATA (authenticate DATE:,
FROM:, TO:, etc. headers at the receiving end).  Though spammers'
SMTP-senders often hang up after dumping their DATA and don't wait for
rejects, this is the best we can do under the current SMTP framework.

5) Permit a gateway SMTP-recipient (or perhaps the user MUA?) to detect
and optionally reject certain body tampering after DATA (authenticate
message integrity).

6) Use DNS rather than public-key infrastructure.

7) Have significant benefits even in the absence of widespread adoption.

8) Interoperates with legacy systems.

9) Don't create open relays.

10) Don't create replay exploits, if possible.


I put a high value on being able to reject mail before DATA, or at least
immediately after the DATA command but before receiving the data, as
some have argued for.  This saves large amounts of bandwidth and may
possibly get you off some spam lists.  With current SMTP, about the only
things we can add to the existing RFC reasons for rejecting non-DSN mail
early in the transaction is a bogus return-path or an SMTP-sender that
we think shouldn't be sending mail for a given domain.  Everything else
comes after data.

I like Brian's combination of single-level SRS with SPF to make
per-domain configuration at ISP's automatic.  This completely avoids
sender rewriting and we only do checks at the originating and target
MTA's.  Here's another straw-man proposal along the lines of David's
combining Brian's ideas with some of my own.

----------------------------------------------------

1) At the originating gateway MTA, validate the user.  If the MTA is
within the organization that owns the domain(s) that appears in the
From: and Reply-To: fields, and the user passes local policy tests,
accept the message.  Otherwise, look up the SPF record for the domain in
question, verify that the user is allowed to use the domain name from
this MTA and if so, accept the message.

2) At the originating gateway MTA, create a return path with two private
key signatures:

MAIL FROM:<sender_sig1+sender_sig2+TT+local_part+domain@originating_MTA>

where

   sender_sig1 protects the remainder of the address

   sender_sig2 protects the SA (Sender_Auth) header, the
      headers listed in the SA header and the number of
      bytes of the message body specified in the SA
      header; this signature is optional but is
      recommended to prevent replay attacks


3) At the originating gateway MTA, create an optional SA header
containing information as to which header lines protect and the size, in
bytes, of the message body to protect.  This header is as follows:

X-Sender-Auth:<headername1+headername2+...+headernameN+body_bytes>

where

   headernameK is the name of the K-th header field
      to protect

   body_bytes is the number of bytes in the message body


4) At the final gateway MTA, perform any desired tests on the
SMTP-sender:  rDNS, forward lookup, IP blacklist, etc.  If it fails any
test, terminate the session.


5) Upon receiving the MAIL FROM: line, do a DNS lookup to get the public
key for domain.  Verify sender_sig1 with this public key.  If it fails,
terminate the session.  Upon receiving RCPT TO:, check for valid
addresses, if possible.  If invalid, terminate the session.  Otherwise,
accept the address and enter the DATA phase.


5) Upon receiving the message data, verify sender_sig2 using the public
key for the domain.  If it passes, accept the message for delivery,
otherwise, reject it, if the SMTP-sender is still there.  If the
SMTP-sender has hung up, send a DSN back to the SMTP-sender.


Notes:

a) The DNS for a domain must contain an SPF record and a public key.

b) No callbacks are required in this implementation, but an alternate
implementation using them is possible.  Specifically, the verification
of sender_sig1 in step 5 could be replaced by a sender callback.  In
that case, the sender_sig1 field would be replaced by a hash.

--

Seth Goodman


From sethg at GoodmanAssociates.com  Tue Feb 24 14:58:22 2004
From: sethg at GoodmanAssociates.com (Seth Goodman)
Date: Tue Feb 24 20:58:32 2004
Subject: Initial thoughts.
In-Reply-To: <MHEGIFHMACFNNIMMBACAEEAOHIAA.sethg@GoodmanAssociates.com>
Message-ID: <MHEGIFHMACFNNIMMBACAEEAPHIAA.sethg@GoodmanAssociates.com>

As an improvement to what I've just proposed, sender_sig2, which
protected the SA header, the headers mentioned in the SA header and the
message body might as well be a CRC32.  You already have confirmation
that the sender is who he said he was, by either public key verification
or callback and a strong hash.  It's we're not trying to encrypt
anything with sender_sig2, just make sure nothing has been tampered with
so that a replay attack is not possible.  CRC32 is very good at that and
computationally very cheap.  Signing with private keys, decrypting with
public keys and computing SHA-1 hashes are computationally expensive,
despite what some on the SPF list said.  We can lighten the load by
replacing the second private key signature with a CRC32 and save a lot
of CPU cycles.  In hardware, a CRC is almost free.

--

Seth Goodman


From sethg at GoodmanAssociates.com  Tue Feb 24 15:32:02 2004
From: sethg at GoodmanAssociates.com (Seth Goodman)
Date: Tue Feb 24 21:32:13 2004
Subject: Initial thoughts.
In-Reply-To: <MHEGIFHMACFNNIMMBACAEEAPHIAA.sethg@GoodmanAssociates.com>
Message-ID: <MHEGIFHMACFNNIMMBACAKEBAHIAA.sethg@GoodmanAssociates.com>

A few more thoughts here.  If we take the proposal that I just made,
used the option for an SRS hash with a sender callback from the far end,
used a CRC32 instead of sender_sig2, we have just removed any
requirement for public key crypto at the receiving end, which is a huge
savings in CPU cycles (unless you've got dedicated hardware).  The
sender callback from the far end, which I suppose would be to the MX for
the sending domain, has one distinct advantage that other anti-spam
folks have been suggesting for a while:  add a cost to _sending_ email,
not to receiving it.  In any case, this is very lightweight at the
receiving end:  DNS tests on the SMTP-sender, one sender callback to the
return address domain and a CRC32 calculation.

The only downside here is that nothing stops a spammy sender from just
saying ACK to any sender callback from his domain without doing the hash
verification.  He opens _himself_ up to forgery and joe-jobs, but he
probably doesn't care, so I suppose we shouldn't either.  If he says the
mail came from him and it didn't, then it's _his_ problem.  Perhaps not
as airtight as the public key method, but pretty good.

--

Seth Goodman


From dwmw2 at infradead.org  Wed Feb 25 13:26:23 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Wed Feb 25 13:26:27 2004
Subject: Initial thoughts.
In-Reply-To: <MHEGIFHMACFNNIMMBACAEEAOHIAA.sethg@GoodmanAssociates.com>
References: <MHEGIFHMACFNNIMMBACAEEAOHIAA.sethg@GoodmanAssociates.com>
Message-ID: <1077715583.7826.678.camel@hades.cambridge.redhat.com>

On Tue, 2004-02-24 at 14:45 -0600, Seth Goodman wrote:
> Here is my own personal list of possible goals for something better than
> the current SPF+SRS, some of which were already listed by others.

Some of these are redundant. #1 and #2 are basically the same, for
example -- 'permit an interested party to detect sender forgeries.'

That aside, they're all basically sane, although...

> 4) Permit a gateway SMTP-recipient (or perhaps the user MUA?) to detect
> and reject certain header forgeries after DATA (authenticate DATE:,
> FROM:, TO:, etc. headers at the receiving end). 

Date: and From: I'll grant you. But To: gets rewritten and I'm not sure
it needs to be verified. You don't learn a lot from the To: header. 

> 10) Don't create replay exploits, if possible.

It should be noted that being able to replay the same mail _if_ it also
contains the same body, and isn't a new spam piggybacking on a
previously signed message header, is acceptable.

I think it's a _good_ thing, in fact. I'll elaborate in a moment.

> I put a high value on being able to reject mail before DATA, or at least
> immediately after the DATA command but before receiving the data, as
> some have argued for. 

Yes, absolutely. As much as possible should be done before accepting
DATA.

But checking _after_ receiving the data is the only way you can check
certain things, and this also prevents you from the moral duty to
generate bounces, if you reject before _accepting_ the data.

> 5) Upon receiving the message data, verify sender_sig2 using the public
> key for the domain.  

> b) No callbacks are required in this implementation, but an alternate
> implementation using them is possible.  Specifically, the verification
> of sender_sig1 in step 5 could be replaced by a sender callback.

This is a very good idea. Using public/private keys for signing your SRS
addresses allows interested third parties to _know_ that it's a real SRS
address generated by you.... and hence this could actually help to
alleviate the multiple-hop SRS problem too.

That is, you rewrite an SRS1 address back to an SRS0 address at someone
else's domain _only_ if they're publishing the pubkey records and the
hash checks out.

-- 
dwmw2


From dwmw2 at infradead.org  Wed Feb 25 13:54:12 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Wed Feb 25 13:54:16 2004
Subject: Address->Mail signing, multiple signatures
In-Reply-To: <1077622201.7826.219.camel@hades.cambridge.redhat.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	 <1077622201.7826.219.camel@hades.cambridge.redhat.com>
Message-ID: <1077717251.7826.712.camel@hades.cambridge.redhat.com>

On Tue, 2004-02-24 at 11:30 +0000, David Woodhouse wrote:
> The proposal as it stands; as a straw man to be argued against...

I'll expand on it a bit.

First, we can note that we should allow _multiple_ signatures. If a mail
has From:, Sender:, and Resent-From: headers, then any or all of those
may have signed the message. Any or all may be publishing records in the
DNS saying that you should disregard mail claiming to be from them which
is not signed.

So if we add a new header to contain some form of signed hash of the
contents, we can include the pubkey which was used to generate it. 

Given a mail, and no access to the outside world, we can know which keys
signed which parts of the mail.

The person listed in the From: header may have signed the body.
The person listed in the Sender: header may have signed the body, and
 also the Message-Id: and Date: headers too.
The person listed in the Resent-From: header may have signed body,
 and the headers Message-Id:, Date:, Resent-Date: and Resent-Message-Id:

All we need to provide is a way for people to sign those things (and
possibly some other things) and include the signatures transparently in
the headers of the mail.

And then a way for them to publish in DNS a promise that "All mail I
send will have _these_ parts signed by _this_ key".

In fact, I think we should standardise the '_these_' parts list,
possibly to those I listed above, to avoid unnecessary complexity in the
DNS records.

So you start off with a mail which looks like this:

--------------

Resent-From: Joe <engineer@company.com>
Resent-To: Sleuth <reporter@news.org>
Resent-Message-Id: <102@server.company.com>
Resent-Date: Tue, 24 Feb 2004 11:30:02 +0000
From: Fred <boss@company.com>
Sender: Jane <secretary@company.com>
Message-ID: <101@server.company.com>
To: Staff <all@company.com>
Date: Mon, 23 Feb 2004 14:20:34 +0000

You're all going to be fired new week. Sucks to be you.

--
Fred.

This mail was checked for viruses by some company's free advert.

--------------

If it were signed by Fred, as the author, he'd not have known the
Message-Id or Date. Only the content can he authorised. But he'd have
added these headers (don't argue about names):

X-Auth-1-Pubkey: aaaabbbbccccdddd
X-Auth-2-hash-content: 3,3d58b23a,hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

That content-hash says that you're looking for three lines in the body,
with a cheap-to-calculate rolling checksum of 3d58b23a. Find three lines
matching that checksum and then take a strong hash of those, along with
the numbers 3 and 3d58b23a. It'll come out to hhhhhhhhhhhhhhhhhhhhhh.

Then Jane might have signed it too. She will have added:

X-Auth-2-Pubkey: bbbbccccddddeeee
X-Auth-2-hash-content: 3,3d58b23a,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

As the actual sender, she will _also_ have added:

X-Auth-2-hash-Date: jjjjjjjjjjjj
X-Auth-2-hash-Message-Id: kkkkkkkkkkkkkkk

which are just trivial signatures of the contents of those headers.

Now when Joe resends it, he adds his own headers...

X-Auth-3-Pubkey: ccccddddeeeeffff
X-Auth-3-hash-content: 5,5c2d7823,mmmmmmmmmmmmmmmmmmmmmmmm

Note that the content hash now signs _five_ lines, because it's grown a
stupid virus checker advert. The original hashes _also_ work, because
you can find the original three lines and check them, and it's
reasonable to accept the mail with only _two_ extra lines added.

Joe will also add headers with hashes of the Date:, Message-Id:,
Resent-Date: Resent-Message-Id: headers.

That's about it.

--

The only vaguely interesting bit is how we make the hash resilient --
and I think the rolling checksum to find N lines within the body does
that OK. 

You take each MIME part separately if it's a MIME mail. Do the fuzzy
hash thing only on text parts -- after un-base64-encoding if appropriate
and converting to UTF-8. Other parts you just do a straight signature
without screwing about.

S/MIME doesn't look sufficiently flexible to do this, but maybe we could
use it for the signatures of binary parts? 

-- 
dwmw2


From dwmw2 at infradead.org  Wed Feb 25 14:11:23 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Wed Feb 25 14:11:27 2004
Subject: Address->Mail signing, multiple signatures
In-Reply-To: <1077717251.7826.712.camel@hades.cambridge.redhat.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	 <1077622201.7826.219.camel@hades.cambridge.redhat.com>
	 <1077717251.7826.712.camel@hades.cambridge.redhat.com>
Message-ID: <1077718282.7826.718.camel@hades.cambridge.redhat.com>

On Wed, 2004-02-25 at 13:54 +0000, David Woodhouse wrote:
> On Tue, 2004-02-24 at 11:30 +0000, David Woodhouse wrote:
> > The proposal as it stands; as a straw man to be argued against...
> 
> I'll expand on it a bit.

We could also add an 'Original-To:' header and sign that. We can't sign
the 'To:' header itself though because that _will_ get rewritten. 

And we may want to include the Message-Id in _all_ the hashes we make,
to stop a potential attack making some kind of replay attack by mixing 
up parts of signed headers with parts of signed body from _other_ mails.

-- 
dwmw2


From B.Candler at pobox.com  Fri Feb 27 12:40:53 2004
From: B.Candler at pobox.com (Brian Candler)
Date: Fri Feb 27 23:41:25 2004
Subject: Address->Mail signing, multiple signatures
In-Reply-To: <1077717251.7826.712.camel@hades.cambridge.redhat.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	<1077622201.7826.219.camel@hades.cambridge.redhat.com>
	<1077717251.7826.712.camel@hades.cambridge.redhat.com>
Message-ID: <20040227174053.GA10660@uk.tiscali.com>

On Wed, Feb 25, 2004 at 01:54:12PM +0000, David Woodhouse wrote:
> So you start off with a mail which looks like this:
> 
> --------------
> 
> Resent-From: Joe <engineer@company.com>
> Resent-To: Sleuth <reporter@news.org>
> Resent-Message-Id: <102@server.company.com>
> Resent-Date: Tue, 24 Feb 2004 11:30:02 +0000
> From: Fred <boss@company.com>
> Sender: Jane <secretary@company.com>
> Message-ID: <101@server.company.com>
> To: Staff <all@company.com>
> Date: Mon, 23 Feb 2004 14:20:34 +0000
> 
> You're all going to be fired new week. Sucks to be you.
> 
> --
> Fred.
> 
> This mail was checked for viruses by some company's free advert.
> 
> --------------
> 
> If it were signed by Fred, as the author, he'd not have known the
> Message-Id or Date.

Why not? Those are headers added by the MUA.

> Only the content can he authorised. But he'd have
> added these headers (don't argue about names):
> 
> X-Auth-1-Pubkey: aaaabbbbccccdddd
> X-Auth-2-hash-content: 3,3d58b23a,hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh

Why would you put a pubkey identifier in? I thought the point was that you
would have to refer back to the DNS to find the public key.

> The only vaguely interesting bit is how we make the hash resilient --
> and I think the rolling checksum to find N lines within the body does
> that OK. 

I remain to be convinced; I think unless an MUA is aware of this signing
protocol, and is prepared to highlight the signed part in green and the
unsigned part in flashing red, then it's a weakness which is far too easy to
exploit.

> You take each MIME part separately if it's a MIME mail. Do the fuzzy
> hash thing only on text parts -- after un-base64-encoding if appropriate
> and converting to UTF-8. Other parts you just do a straight signature
> without screwing about.

S/MIME probably has something to say about how to handle MIME parts. The
only transformations you can expect to have occured is likely
Content-Transfer-Encoding (some MTAs do this, e.g. courier); I wonder if any
MTA would attempt a character set transformation? Sounds like a dubious
thing to do.

Regards,

Brian.

From dwmw2 at infradead.org  Sat Feb 28 00:12:37 2004
From: dwmw2 at infradead.org (David Woodhouse)
Date: Sat Feb 28 00:12:41 2004
Subject: Address->Mail signing, multiple signatures
In-Reply-To: <20040227174053.GA10660@uk.tiscali.com>
References: <20040224093913.GA4143@uk.tiscali.com>
	 <1077622201.7826.219.camel@hades.cambridge.redhat.com>
	 <1077717251.7826.712.camel@hades.cambridge.redhat.com>
	 <20040227174053.GA10660@uk.tiscali.com>
Message-ID: <1077927156.6336.802.camel@hades.cambridge.redhat.com>

On Fri, 2004-02-27 at 12:40 -0500, Brian Candler wrote:
> Why not? Those are headers added by the MUA.

Generally by the _sender_ not the author. See the example in RFC2822.

> Why would you put a pubkey identifier in? I thought the point was that you
> would have to refer back to the DNS to find the public key.

Which DNS? There can be multiple signatures by multiple pubkeys.

> I remain to be convinced; I think unless an MUA is aware of this signing
> protocol, and is prepared to highlight the signed part in green and the
> unsigned part in flashing red, then it's a weakness which is far too easy to
> exploit.

The MTA can reject if there's too much, and many MTAs are quite capable
of the flashing red bit.

> S/MIME probably has something to say about how to handle MIME parts. The
> only transformations you can expect to have occured is likely
> Content-Transfer-Encoding (some MTAs do this, e.g. courier); I wonder if any
> MTA would attempt a character set transformation? Sounds like a dubious
> thing to do.

True, but that doesn't meant it would _surprise_ me :)

-- 
dwmw2


