Qoba.lt

27th dimension of the Internet

Improving email delivery

by , on

Recently I saw a post on Hackers News linking to an article entitled Why does Gmail hate my domain?, complaining about Gmail redirecting the mails sent by the writer to the junk box of his recipients. While the tone and conclusion (“Google does this on purpose so that we all switch to Gmail to have working emails”) are debatable, I couldn’t help to relate to my own case: I recently deployed my own mail server, and have the same issue.

At the time of this writing I am still trying to figure out ways to improve the ratio of mails successfully delivered (that is to say, not into the spam box). This article is a list of resources and ideas taken from Hacker News or from my own web searches in this way. For now I am unable to determine precisely what method is efficient and to what extent, even though I can make some suppositions. I shall update this post if I get any more useful information.

I will not provide details here on how to implement or to set up each of the mechanism listed below, because there are plenty of tutorials available already, just ask to your favorite search engine. This article is rather a list of links and of elements to consider once you’ve basically installed your mail server. There will be five parts:

1 Mechanisms to fight against spam

Some software are specifically designed to run on the recipient host to analyze incoming email, to check blacklists and to run bayesian filters and to decide if the email is to be considered as legit or as junk. This is not the software we focus on, because we can’t really interact with it when we’re the mail senders. But prior that this software is run on received mail, other checks can occur.

The first three mechanisms presented here are must-haves for your server if you do face important delivery issues.

1.1 Reverse DNS lookup

This one seems to be very important to keep a correct delivery ratio. On email reception, many recipients perform a reverse DNS lookup on the sender’s IP address. It means that they query a DNS server on the IP to see what domain name it resolves to, and compare this with the sender’s domain name: it is best if the values are consistent. So make sure that your DNS knows what domain you want your server to resolves to. This might be done from the web manager of your dedicated server, for example. In practice, it’s up to the ISP that “owns” the IP block your address comes from to set it.

1.2 SPF

Sender Policy Framework is a mechanism implying a dedicated DNS field (for the sender), by which the recipient of an email can check whether the host that sent the email is actually authorized by the (sender) mail server administrator to do so.

It only involves setting up your policy in a DNS field which is pretty easy to do.

1.3 DKIM

DomainKeys Identified Mail is a mechanism used by the sender to sign a message with a private key. The public key associated to this private key is stored in yet another DNS field, where the mail recipient can retrieve it to verify that the email was actually sent by the host whose address appears in the headers.

Along with SPF, it seems to be a widely deployed mechanism in the fight against spammers. It is a little more complicated to set up, since in addition to modifying a DNS field (to add the public key), you have to install a DKIM software agent on your host to sign the outgoing emails. But it remains easy to find all the instructions you need on the Internet.

1.4 DMARC

Domain-based Message Authentication, Reporting and Conformance is an extension to DKIM used to indicate what policy to follow when emails do not pass the SPF and DKIM verification on reception. It also provides a “return address” to which the recipient can send daily report, so that the mail server administrator can receive some feedback and see what is wrong (or who is trying to impersonate their domain).

As for SPF, there is only a DNS field to edit, but you must have DKIM implemented to have DMARC work.

DMARC is supported by the major email providers, but it does not seem to be as widespread as SPF or DKIM.

2 Training recipient mail providers

I read this hint on Hacker News, and many people seemed to confirm its efficiency on big email providers: companies such as Microsoft or Google seem to use learning methods to determine whether emails are junk or not. The principle is as follows: if you send “blank” mail, or mail with “weird” content (not looking like a standard conversation), and that the person receiving the mail never answers to you, this does not looks like a “real” conversation between humans. On the opposite, if you regularly write “conversation-like” emails and if your recipient regularly answers to it, creating a thread of messages symbolizing a real conversation between all parties, Gmail would allegedly learn to recognize emails coming from your domain as rightful.

Also, having your (human) recipient systematically moving your mails from their junk box to their normal inbox should help the software to learn that your domain is not the one of an evil spammer. On the day I started to write this post, Google released its Gmail Postmaster Tools, stating on Gmail blog:

Since the beginning, machine learning has helped make the Gmail spam filter more awesome. When you click the “Report spam” and “Not spam” buttons, you’re not only improving your Gmail experience right then and there, you’re also training Gmail’s filters to identify spam vs. wanted mail in the future.

Some people also reported that adding a “Click to unsubscribe” link at the end of their emails helped passing through spam detection, but of course that’s not really suitable for personal emails.

3 Checking for web reputation

Your mail server could appear on one or more blacklist dedicated to spam detection for several reasons.

3.1 Current misuse of the server

If you did not set up your server correctly (if it is an open relay for emails), spammers could use it to redistribute junk mail. Do not believe your server is protected because nearly no one is aware of its existence: check your logs. Every day, many attempts are made by anonymous people to relay spam through your SMTP software.

Also, if you did not implement SPF or DKIM, some spammers could “impersonate” your domain name by filling the sender address of junk mail with it, which would quickly get you blacklisted.

3.2 Past misuses

You may also be blacklisted if you get a domain name or an IP address that has been used by a spammer before you received it. This sometimes happen if you rent a server, for instance. Your IP could even be blacklisted if you share it with someone else and if this someone is using it for rogue purposes, for example if you use a VPS (the host runs several virtual private servers, sometimes sharing the same IP).

It is somewhat possible to ask for un-blacklisting. Luckily, I could not find my domain and IP on any list, so I did not have to try this, but you would probably have to search for dedicated forms on websites of blacklist maintainers.

4 Miscellaneous parameters

4.1 Denying mail relaying

Surely you do not want your mail server to accept to relay anyone’s (that is to say, all spammers’) email. If you do not want to get instantly blacklisted, do not enable this feature.

4.2 Setting up TLS

Setting up TLS is more secure, maybe it also makes your server more trusted. It’s at least 2015: please don’t use SSL. I don’t know whether verification is performed on the authenticity of the TLS certificate in order to trust the email or not. In my case I only have a self-signed certificate for the mail server, because I won’t pay a CA for setting up my own personal mail, but it might be a drawback for mail delivery.

4.3 Setting up postmaster and abuse mail boxes

Set up valid addresses postmaster@yourdomain.tld and abuse@yourdomain.tld. This is considered as good practice, and unless you change its configuration it is possible to interrogate your web server to check whether these addresses exist. When they do exist, it’s a sign that probably someone can be reached in case of email abuse coming from that domain, whereas they would be no one to contact otherwise. Of course, you can redirect them to your main inbox.

4.4 Other

4.4.1 Attaching a Google Apps account to the domain

I’ve never tried this, but in the comments of the original article cited at the top someone reported that registering (then immediately canceling, but anyway) a Google Apps account for the domain did the trick and helped their mail pass through Gmail filters.

4.4.2 Top-level domain

It’s only a supposition, but I fear that selecting a .lt domain was not the best choice to write to (mostly) French people. Mail we receive here from Lithuania is mostly spam. Or is it? I’m being unfair, this is only a cliché and I might be totally wrong; but mail providers could do the same after all. I would not be that surprised if some of them were to put my messages to junk box because I chose a top-level domain from which people I write to are not accustomed to receive mails from.

4.4.3 Filling in dedicated forms

Microsoft mail service (formerly Hotmail, now it seems to be Outlook) as well as Gmail seem to have dedicated form to complain about emails not passing the filters, but most of the comments I read about it stated that this is useless, and that people usually get no answer. I did my own experiment: Microsoft never replied to my answer indeed.

5 Web resources

Conclusion

If you are facing delivery issues, try to use one of the above link to access debug tools for your setting. Make sure that your reverse PTR record is valid; and set up SPF and DKIM correctly. Having one’s emails passing the filters seems to be a difficult (and maybe not totally deterministic) problem, I’m trying to figure out how to improve my ratio myself. If you know of additional efficient ways to get it, don’t hesitate to mail me so that I can add it here − I won’t discard your mail. If you’re trying to set up your mail server, I hope some of it could help you; and good luck!