The email system is similar to real mail, but the first one is more complicated and confused. I'll try to cover all important points of the theory and fundamental concepts.
- Local mail storage
- Email address
- Mail format
- Simple Mail Transport Protocol (SMTP)
- Open relay
- How email operation works
- DNS lookup
- POP vs IMAP
Mail User Agent (MUA): a client program which provides a user interface for writing, reading and sending mail. This application doesn't send an email to a recipient, just transmits it to an MSA. Another name for MUA is email client. Examples: Thunderbird, mutt, sylpheed, web-based gmail.
Mail Submission Agent (MSA): a server program which receives email messages from an MUA, check for any errors, and tranfers it to the MTA on the same server. Historically, MSA is the component of an MTA. I don't know separate programs, that implement functions of MSA.
Mail Transport Agent (MTA): a server program which transfers email messages from one computer to another using SMTP. Examples: Postfix, Sendmail, Exim
Mail Delivery Agent (MDA): a server program which is responsible for the final delivery of a message to a mailbox. This can be a separate program, sometimes it's built into the MTA. Examples: Dovecot (also includes POP3 and IMAP server), procmail, maildrop
Simple Mail Transer Protocol (SMTP): the protocol used between MTAs for sending email messages from one host to another. This protocol is also used between an MUA and an MSA. The standard SMTP port is 25.
Post Office Protocol (POP) and Internet Message Access Protocol (IMAP): protocols used by MUAs to retrieve mail from a user's mailbox on a remote server. You can find comparison of these protocols below. The standard ports: POP3 is 110, IMAP is 143.
Local mail storage
In ancient times, when dinosaurs were alive, UNIX users sent mail to each other on the local system. Most of the formats with some improvements have been used since those times. Now, in most cases, mail is stored in the following formats:
- The mbox is traditional UNIX mailbox format. It stores all messages in one large file.
- The maildir stores messages in seperate files. Therefore, you can delete a message from the mailbox without having a copy the entire mailbox. Mailboxes of this format are usually kept in the user's home directory and contains the
Of course, other formats are existed, for example: MMDF, MH, dbox. But they aren't very popular.
An email address consists of three parts: a local part, the symbol @ (read as "at"), and a domain, which may be a domain name or an IP address enclosed in brackets. The general format of an email address is local-part@domain, e.g. firstname.lastname@example.org, email@example.com.
Real internet email (that is transferred via the SMTP protocol) consists of the following parts:
SMTP data (Information that is transmitted with
Message (All data that is transmitted after the
- Headers (It's simply metadata. Provides service information of mail servers. An MUA parses it and shows it to an user in a pretty way)
- Blank line (Headers is separated from the body by a blank line)
- Body (Actually the message itself. It includes text and various attachments)
Common header fields
From:- sender's name and email. Might not match the SMTP
To:- recipients names and email addresses.
Cc:- carbon copy recipients.
Reply-To:- address should be used to reply to the message.
Message-ID:- unique ID of a message. Uses for reference.
Subject:- topic of a mail.
Date:- the date when an email is sent.
Content-type:- info about how the message is to be displayed, usually a MIME type.
Mail headers are described in RFC 2076. If you want know more about message format, see RFC 5322.
Simple Mail Transport Protocol (SMTP)
SMTP is an internet protocol for email transmission, it allows MTAs to exchange email witch each other over a network. In theory, this single protocol would have been enough for us if our computers had been always turned on and online (it was true for mainframe). But nowadays we receive mail from the network (because a personal computer isn't a mail server) and need mailbox access protocols (POP, IMAP).
When I talk about the SMTP protocol, I and many other people actually mean ESMTP (Extended SMTP). The original SMTP doesn't support authentication (see POP-before-SMTP) and encryption.
Protocol communication looks like this: a client connects to the server and sends commands, the server responds to them with special codes. Let's talk about some important commands:
HELO / EHLO
Starts the conversation and identify itself. The EHLO (Extended Hello) means that the client can support the ESMTP protocol. The domain or IP address of the SMTP client is sent as an argument together with the command (
EHLO smtp.joursoir.net). Of course, the client is free to lie. But the server already knows the client's IP address. I can't understand the idea of
HELO / EHLO.
Indicates who is sending the mail (
MAIL FROM: <firstname.lastname@example.org>). If the receiving MTA cannot deliver the message, it sends a failure ("bounce") message to this address. This command actually starts the email transfer.
Specifies the recipient of the mail (
RCPT TO: <email@example.com>). This command can be repeated muptiple times if a message must be delivered to multiple recipients.
Starts the transfer of the message contents. The message text must end with the following five letter sequence: "\r\n.\r\n".
Authenticates itself to the server, giving its username and password. This command must be combined with name of mechanism as PLAIN, GSSAPI, DIGEST-MD5 or others (
AUTH PLAIN). After that, we send username and password in the format required by the mechanism. This authorization technology is called Simple Authentication and Security Layer (SASL) and is used by some other protocols too.
Initiates the use of TLS. SMTP servers communicate with each other using plain text over the Internet. To improve security, an TLS connection should be used (a large number of servers require it).
Example of SMTP session
You can start communication with the MTA server using telnet(1):
$ telnet <domain> <port>
Or use openssl(1), if encryption is required:
$ openssl s_client -starttls smtp -crlf -connect <domain>:<port>
= 220 smtp.joursoir.net ESMTP Postfix
> EHLO gmail-smtp-in.l.google.com
= 250-SIZE 10240000
= 250-AUTH PLAIN
= 250 CHUNKING
> MAIL FROM: <firstname.lastname@example.org>
= 250 2.1.0 Ok
> RCPT TO: <email@example.com>
= 550 5.1.1 <firstname.lastname@example.org>: Recipient address rejected: User unknown in virtual mailbox table
> RCPT TO: <email@example.com>
= 454 4.7.1 <firstname.lastname@example.org>: Relay access denied
> RCPT TO: <email@example.com>
= 250 2.1.5 Ok
= 354 End data with <CR><LF>.<CR><LF>
> From: Name Surname <firstname.lastname@example.org>
> To: Joursoir <email@example.com>
> Subject: test message
> This is a test
= 250 2.0.0 Ok: queued as C15F841DD5
= 221 2.0.0 Bye
In the previous conversation, the '>' lines are the SMTP commands sent by the client. The '=' lines are the server's replies.
If you want know more, see RFC 5321
Nowadays mail servers are configured to accept mail from any user to their user, or from their user to anyone. But it wasn't true always. Until about 1994, almost all Internet mail server were ready to accept mail from anyone to anyone (such servers are called open relays). Unfortunately, this was actively used by spammers. The community had to change its attitude towards mail settings. In 1996, there were almost no servers on the Internet ready to receive and transmit mail to anyone. In our time, open relays appear but they're immediately included in various blocking lists, as a result of which most of the Internet refuses to accept mail from them. I should say that getting into such a "black list" is much easier than getting out of it later.
When you will set up your mail server, pay special attention to disable the open relay function.
How email operation works
I hope the next scheme will help you understand principles of sending mail (the picture from the Internet):
Let's say Alice (her email: firstname.lastname@example.org) wants to send a mail to Bob (email@example.com), then:
1. Alice composes an email using any MUA. The recipient is firstname.lastname@example.org. The MUA transfers the email to Alice's MTA (smtp.aliceserver.com). Somewhere here is authorization and spam checking.
2. Alice's MTA determines the domain of the recipient's email address. Next, the MTA get information out of the DNS. The last one has special MX (Mail eXchanger) record associated with the domain. Imagine that in our case, the DNS server return mx.bobserver.net. Read more in DNS lookup (below).
3. Alice's MTA (smtp.aliceserver.com) sends the message to Bob's MTA (mx.bobserver.net) using SMTP.
4. The mail transfer is finished after the Bob's server has responded "OK" and delivered it to the mailbox of user bob. Hooray, mail is delivered! Bob can access his mail on the server, but he wants to do it using his PC.
5. Bob's MUA contacts his POP/IMAP server and recieves mail.
I consider that you know what the Domain Name System (DNS) is. Suppose we want to send a mail to email@example.com. First, we request for MX records that the gmail.com domain has. We receive a list of mail servers with priority. After that, we try to connect to the mail server with the lowest value of priority. If server doesn't work, try to connect to another.
We can perform DNS lookup with special tools. For example, dig(1) or drill(1):
[joursoir@arch ~]$ drill -Q mx gmail.com
[joursoir@arch ~]$ drill -Q gmail-smtp-in.l.google.com
[joursoir@arch ~]$ drill -Q google.com
The IP addresses of gmail-smtp-in.l.google.com (mail server) and gmail.com are different. So, they are not the same machines.
POP vs IMAP
If users will read their mail without shell access, then mail server needs a POP or IMAP server. Both protocols support SSL/TLS.
POP is designed for offline work with mail. You connect to a remote server, take mail to your MUA and that's it. Many people think that POP server deletes mail after it has been transferred to the user. This isn't entirely true, because it also has a "keed mode" in which nothing happens to messages after they are sent. Described in RFC 1939
IMAP is conceptually different. It assumes that user will be online as long as the its MUA is still running. And you won't download the messages to your PC (although you can), but leave them on a remote server. It's quite convenient if you want to synchronize mail between mulptiple clients. Described in RFC 2060
It's difficult to say which is the best. They're designed for different purposes.
Everyone knows about spam and how terrible it is. Next technologies is designed to protect against it and spoofing.
Sender Policy Framework (SPF)
SPF allows to verify that incoming mail comes from a server responsible for delivering mail in the sender's domain. How does it work? The DNS server has a TXT record with a list of such servers. Often these are the same addresses that are listed in the MX records. An MTA can easily query SPF information with a simple DNS query, allowing the sender's server to be verified. If a message is sent from an IP that is not in the SPF-record, then the message is probably fake. Simple example (read about syntax here):
example.org. IN TXT "v=spf1 +a +mx -all"
DomainKeys Identified Mail (DKIM)
DKIM adds a digital signature to the outgoing message, that allows the recipient to verify that the message was created by a responsible domain. Using the private key (it's stored on the server that sends the mail), DKIM-Signature is added to the headers of all outgoing mail. The public key is placed in the DNS server in the TXT record. The receiving MTA verifies the signature with this public key. Example of record:
Domain-based Message Authentication, Reporting and Conformance (DMARC)
DMARC extends SPF and DKIM. It tells a receiver what to do with a mail message depending on the results of checking authentication methods. Like all others, this technology has a TXT record in the DNS server (read about syntax here):