In early 2008, new changes were added to these email providers in the strict rules used to accept email from these providers. The exact details are secret, but what we do know is that SPF records are now a more important factor in these rules.
You can use google to inquire about other people with these issues (you're not alone). A few more descriptive results: Delivering emails to GMail, Yahoo and Hotmail
Disclaimer: This is still a work in progress, I might update this soon.
Introduction: in the mid-2002, many e-mail providers like Yahoo and Hotmail (no GMail at that time) started the war against spam and they went on improving spam fighting since then. Now domain owners are fighting with anti-spam solutions which are blocking their innocent e-mails. This blog entry is a result of one of the fights happening everyday :) GMail:
Google was the easiest mail vendor to deal with, it relies solely on an open standard called SPF which is simply a TXT record a domain owner can put on his domain's DNS zone to state that the domain is fully responsible for e-mails sent from it under specific circumstances.
In the OpenSPF website you will find a wizard that will help you generate your SPF signature. After adding the SPF TXT record, e-mails started to appear in the Inbox instead of junk. I must confess that I strongly believe that GMail does *NOT* discard any e-mail coming to your account even if it recognized it's even a phishing e-mail! Yahoo!:
Yahoo is also fond of open source solutions and the folks there started relying on DomainKeys Identified Mail, aka DKIM, which is a collaborative work between many IT big names like: Alt-N Technologies, AOL, Brandenburg Internetworking, Cisco, EarthLink, IBM, Microsoft, PGP Corporation, Sendmail, StrongMail Systems, Tumbleweed, VeriSign plus Yahoo! for sure. PS: Microsoft was participating in DKIM and then it used its own technology instead of DKIM! These guys keep surprising me. Anyways DKIM is a bit tough solution since it requires patching your mail server to attach your DKIM Public Key with every outgoing e-mail which is then being matched with what the mail receiver requests from the sender domain and in case they match then Yahoo, for example, will allow this e-mail to its inbox if the sender domain isn't blacklisted by Yahoo or by any public blacklist.
The interesting thing was: when Yahoo doesn't find a DKIM in the message, it falls back to SPF and will behave like GMail will do with a message. Hotmail:
Microsoft, as I mentioned above, was collaborating in the DKIM. But they use their own technology, called SenderID. Sender ID is basically SPF with Microsoft's spices. SenderID is a caching system for SPF records. Thus you will find things a bit quicker if you e-mailed senderid@microsoft.com to inform them you changed your SPF record or created one. Then they will reload your domain's cache.
Once again this means that Microsoft relies on SPF. After creating SPF record for the domain under investigation, e-mails started to appear in Hotmail's junk mail which isn't expected and is an indicator that there's something else to do.
Lets start off with a quick overview of what an SPF record is. An SPF record is simply a TXT type dns record which is used to store information about what IPs are allowed to send email for a specific domain. A receiving mailserver can do a lookup on this TXT record get get this data, and use the information to decide if an IP sending email from a specific domain really is allowed to be sending email for that domain.
There are a few options you can use with SPF records to govern the severity of encforcement on these rules.
By default (as of May 12, 2008), DirectAdmin uses the ?all value for it's SPF records. This means that all receiving mailservers should be neutral about who is allowed to be sending mail for a domain. For example, mail would be allowed to come from your DA box, and allowed to come from your clients ISP mailserver (since many ISPs block all smtp traffic out execpt from their own servers). This is why we use a default of ?all, so that it's more allowing of mail from different locations, since many users simply don't have the ability to send through their DA server. (Note: using port 587 instead of 25 to the DA box is the current workaround for that, which is recommended for this fix).
Now these big email providers don't like this email sending neutrality. They want to know for sure that a sender is correct, so they assume that any domain sending email with a neutral SPF record must be spam, thus your emails end up in their spam folders.
The solution is to setup your SPF record to use the -all option insetad of the ?all option so that email can *only* be sent from your server. This means that you *must* get all of your clients to send their smtp through your server, or they will be tagged as spam for spoofing. If they're blocked on port 25 to your server, tell them to send on port 587 to your server (same thing, just not block by the ISP)
To setup a value of -all, you can either change your DNS zone manually from: Admin Level → DNS Administration → domain.com or User Level → DNS Management change:
"v=spf1 a mx ip4:1.2.3.4 ?all"
to be
"v=spf1 a mx ip4:1.2.3.4 -all"
of course, where 1.2.3.4 should be replaced by your server IP.
If you'd rather do this to all of your domain, you can use perl
cd /var/named perl -pi -e 's/\?all/\-all/' *.db
change /var/named to the appropriate directory for your db files for your OS (/etc/bind, /etc/namdb)
If you want this to be your default for new domains, type:
cd /usr/local/directadmin/data/templates/custom cp ../dns_txt.conf . perl -pi -e 's/\?all/\-all/' dns_txt.conf
which will change the default TXT template to use the -all value.
Restart named after making the changes. Allow at least 4 hours for the new records to propogate (possibly more if they ignore the TTL values)
There are other basic things on top of the SPF records, like making sure your hostname resolves, and that your server IP has a reverse IP lookup (see How to setup the Mail System)
More information about this will be added as it becomes known.