The Simple Mail Transfer Protocol permits any computer to send email claiming to be from any source address.
It’s been conceived for a very trusting environment, not exactly what’s the internet today…
Over time the amount of spam has grown substantially and with that the effort to filter that out.
The net result is that one can never expect 100% deliverability of emails.
Automatically generated emails are the most affected as:
- they are sent on behalf of the sender by 3rd party servers (exactly like spam)
- ISPs have learned to filter aggressively against them
- they don’t originate from trusted domains/servers (gmail, hotmail etc)
Unless you really know what you are doing (like the guys at basecamp), never try and send emails yourself, trust the people that do that for a living and use a 3rd party service.
What to expect
Email deliverability for automated emails can never be consistently 100%. and it’s not only because of spam.
Using Amazon SES you can get around 80/85% as you don’t have the option of using dedicated IPs. Moreover spammers love SES. Using SendGrid, Mailgun or even Mandrill with dedicated IPs you can get to 90/95%.
I’m personally a big fan of Mailgun because of their high free threshold (15k emails/month)
The deliverability of your automated email is linked (among others) to the reputation of the public IP address of the server sending the email(and sometimes of the whole subnet). This reputation is built over time and if you are to setup an EC2 instance and start sending emails you can easily see that most of your emails will end up in spam.
A key element is the speed at which emails are sent from each unique IP address. Technically you can deliver 1 million emails/minute from one IP. Spammers do exactly that so please don’t emulate them. So if you want your emails to enjoy higher deliverability rates they need to be sent slowly. If you need more speed distribute your traffic among multiple dedicated IP addresses. It’s non trivial to do it, so just use a 3rd party provider.
SPF and DKIM
Authentication is a way to prove an email isn’t forged. A way to do that is to have emails digitally
signed. There are a variety of authentication methods out of which SPF and DKIM (both relying on DNS records) are the most common. They only authenticate the domain you are claiming to send from
When you add authentication information to your DNS, an added benefit is that many ISPs will use that to track sending reputation (and not only the IPs)
SPF allows the owner of an Internet domain to specify which computers are authorized to send mail with sender addresses in that domain. In other words SPF allows to whitelist sender IPs for a specific domain.
Specifically SPF works by setting up TXT entries as follows (example configuration taken from mailgun.com)
mydomain.com TXT v=spf1 include:mailgun.org ~all
where the include resolves to
mailgun.org. TXT "v=spf1 include:spf1.mailgun.org include:spf2.mailgun.org ~all"
which finally resolves to
spf1.mailgun.org. TXT "v=spf1 ip4:220.127.116.11/27 ip4:18.104.22.168/27 ip4:22.214.171.124/27 ip4:126.96.36.199/30 ip4:188.8.131.52/24 ip4:184.108.40.206/24 ~all" spf2.mailgun.org. TXT "v=spf1 ip4:220.127.116.11/24 ip4:18.104.22.168/22 ip4:22.214.171.124/23 ip4:126.96.36.199/23 ip4:188.8.131.52/24 ip4:184.108.40.206/24 ip4:220.127.116.11/28 ~all"
- if the email comes from these IPs, mark as PASS
- otherwise mark as SOFT-FAIL (~all)
DomainKeys Identified Mail (DKIM) allows receiving mail exchangers to check that incoming mail from a domain is authorized by that domain’s administrators and that the email (including attachments) has not been modified during transport.
A digital signature included with the message can be validated by the recipient using the signer’s public key published in the DNS as a TXT record.
In most cases, the email gateway acts on behalf of the author organization or the originating service provider by inserting a
DKIM-Signature: header field for you. It roughly looks as follows:
DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mydomain.com; q=dns/txt; s=mx; t=1395781559; h=From: To: Subject: Date: Content-Type: Content-Transfer-Encoding: Mime-Version: Message-Id: Sender; bh=JJseo+CTGDOUQ4MTFhmjOE3RXQRxvepZvRq+6TuxJaU=; b=d2YWDhnqfvgJYSPKk12RManc184UMi59bDMlCibgvdptkbIr1ruDKt+PXPSPG7sbpsguMdgSDVUpMjY3lYVlcUgSMiK0GA5nrM3uTQCBzX0AcJk8F6Yd6q3coDHpX+o+upW8bNTSPjF22F0qPA8oDn/r63tv/hVdOZGDRR5Qf7o=
DKIM barely forces senders to show a correct source domain. It does not identify spam in itself. The information around the true source domain is usually fed into a reputation system allowing to identify good and trustworthy source domains from untrusted ones
Emails claiming to be from heavily phished domains is not even accepted without a valid DKIM signature by Gmail etc
Domain or subdomain
As your reputation as a sender is linked to your domain it’s a good idea to have a dedicated domain (or a subdomain) to send emails from, like in the following example:
`example.com` = naked domain, use for www `m.example.com` = subdomain, use to send emails **only**
This helps you starting fresh without inheriting the reputation of your master domain which may have been sending email using a misconfigured Exchange server in the past and have now low reputation.
It lets you receive emails programmatically (by specifying different MX servers for the subdomain) and costs close to nothing. A no brainer.
This seems to be source of confusion…
Generally speaking transactional emails don’t need unsubscribe links, but newsletter emails always do.
Also login to unsubscribe is a bad practice, not even legal in the US