Serverless Fun: Using Amazon SES and Lambda to send and receive email

Background

I’m using Lightsail as a VPN, and don’t really want to use an elastic IP because my VPN instances tend to be ephemeral. And because I’m not using an Elastic IP, security event emails sent from the host to my Gmail account fail to send. I also want to be able to send and receive an email or two every once in awhile on my new domain. And I really, really don’t want to run my own mail server right now, nor can I justify $25/month for Google hosting for a vanity blog only accessed by bots, spiders, and Internet censii. I just want emails sent to anyone@mydomain.com to magically show up in my Gmail inbox, and I want to be able to reply as me@mydomain.com.

So, I decided to try out Amazon Simple Email Service (SES). It’s really created for the use case of sending bulk emails, not for providing a serverless email setup for small fries like me — but can we make this work anyway?

I initially configured it, went to create a forwarding rule and…guess what? You can send emails using your new domain in your email client of choice, but no forwarding rules are available out-of-the-box for received mail. All of the emails your domain receives are unceremoniously dumped into a bucket on S3.

But fear not! Amazon’s got a service called Lambda that allows you to upload scripts that trigger when new items hit the bucket and an awesome developer has provided a script you can use to parse and forward the email you receive in your S3 bucket to your actual email account.

Here’s a quick overview of the flow:

From the Amazon Management Console, set up your domain, configure the SES service, and configure a Lambda rule to forward mails you receive to your non-Amazon email account.
Use the SMTP credentials created in step one within your email client of choice so that you can “send as” your domain user.
Send a message to someone using your send-as identity in your non-Amazon account.
When your recipient replies, it gets forwarded to the S3 bucket, your Lambda function picks it up and forwards it to your non-Amazon account.
When you receive the message, it will be forwarded by the sender you designated in the Lambda function. But because the Lambda function copies the original sender to the “Reply-To” field, replying should respond to the actual message sender.

Limitations

SES itself has sandbox limitations; initially, you can receive from anyone, but only send to verified addresses and you’re limited to sending 200 emails per day. After you’re set up, you’re going to want to visit this link to request a sandbox escape.

Your mileage may vary, but despite a warning message from Amazon about high case loads, it took 9 hours from my request to approval, and they set my quota to 50,000 emails a day (more than I will likely ever need). Note also that requests to escape the sandbox apply on a region basis only; if you’re approved for US-EAST and want to send from S3 in US-WEST, you’ll need to send a separate sandbox request.
You can configure rules for specific mailboxes and a catchall rule in the Lambda function, but you’ll need to add each mailbox separately if you have multiple users and destinations. Not a showstopper or a real limitation, just needs a few more minutes to configure (in my case, everything goes to me, so it’s pretty easy). However, this is not a viable solution for a non-micro-sized organization.
Received mail is forwarded from sender@yourdomain.com and the actual sender can only be seen in the headers of the email in the Reply-To field. There can be some weird behavior here while testing, if you receive an email from another account that you’ve configured as a “Send as” account, for example, you’ll find that you reply to the recipient, not the sender — and that can get kind of confusing when testing.
You can’t encrypt messages at rest in S3 right now if you use the Lambda function (you can, and should configure TLS encryption for email in-transit).
SES is configured per-AWS region (currently US-East-1, US-West-2, and EU-West-1) and not necessarily per-domain. For instance, you can set up more than one domain for SES, but the SMTP credentials you use to send mail will be the same for all domains you set up in that region…and you might as well use the same Lambda forwarder for all domains. This actually makes setup a lot easier, but wasn’t initially very intuitive to me.
It’s not free (but it won’t break the bank): Email messages are $0.10 per 1000 sent messages, attachments you send are $.12 per GB, $0.09 per 1000 “mail chunks” received. Each chunk is 256 KB, so if you received an email of 768KB, that would count as three chunks, but an email of 255 KB would not count at all towards your 1000 “chunks.” You also have charges for S3, $0.005 per 1000 PUT/COPY/POST/LIST requests, $0.004 for every 1000 GET requests, and $0.023 per GB for the first Terabyte of storage used. Your mileage may vary, especially if you send/receive a lot of email. I’ve been using it for 10 days now, and have spent four cents (one on SES, three on S3).

S3 Bucket considerations

DO NOT configure access to your S3 bucket to “All authenticated AWS users” – that actually means…ALL authenticated AWS users, not just your account/IAM users with access to your account.
Consider enabling logging so that you can catch errors. You can log to a different bucket or a different folder in the same S3 bucket.
Consider enabling Lifecycle Rules that allow you to purge all contents after a certain number of days; clean up incomplete uploads, etc. This will save you money on S3 storage, too.

If you’ve read all of these caveats and the method still fits your use case (as it did mine), read on for a step-by-step walkthrough…it looks more involved and complicated than it really is, I promise.

Add and Verify Your Email Domain(s)

The first thing you’ll want to do is add the email domain(s) that you want to configure for SES:

Open the AWS Management Console and, under Messaging, click SES (or search for it in the search field).
From the left-side Identity Management menu, select Domains, then click Verify a New Domain.
Type the domain name in the Domain field and and enable Generate DKIM Settings.

DKIM stands for Domain Keys-identified Mail. This encrypts email headers that can be decrypted using a public key stored in your DNS records, allowing recipient email servers to verify that your email is authenticated and hasn’t been spoofed. (Without DKIM, Gmail and other mail servers may be more likely to dump your messages into the Spam folder.)
Click Verify This Domain.

A list of entries to be added to your domain settings appears.
Click Download Record Set as CSV to save the settings for your records.
Depending on where you host DNS records for your domain:
- If you’re using Amazon for DNS, the next step is super-easy, just click Use Route 53 and proceed to step 7.
- If you’re hosting DNS somewhere else, take the entries from your CSV file and add them to your DNS settings at your registrar or DNS provider. Amazon will periodically check your host record for the changes and, once it sees them, will verify your domain (typically within a few minutes but can take up to 72 hours). Don’t forget to add the MX record; that allows you to receive as well as send. You can jump to step 8.
Enable the Email Receiving Record checkbox, then click Create Record Sets. Amazon will auto-populate your domain record for you.
You’ll see Domain Status appear as pending verification. Wait a few minutes for Amazon to verify the entries and you’ll see pending verification change to verified when you reload the page. The domain I verified was hosted with Amazon Route 53 and it took about 5 minutes. It can take up to 72 hours if you host DNS somewhere else (but likely will take just a few minutes).

Verify Your Sender Recipient Email Address(es)

Amazon needs to know which email addresses it should allow to send email using SES. Also, while you’re still in testing mode in the sandbox, these will be the only email addresses you can reply to. For my testing, I added two personal accounts and one account on the domain I’m using.

Under Identity Management in SES, select Email Addresses.
Click Verify a New Email Address.
When prompted, enter the email address you want to verify and click Verify This Email Address.
Check the inbox of the email you set up, and click the verification link Amazon sent.
Reload the Email Addresses page and you should see pending verification update to verified.

Create SMTP Credentials for Email Clients

These are the credentials you’ll use to send email from your email client. Only one set of credentials is used per AWS region and you’ll use them for any domain you set up.

Select SMTP Settings from the sidebar menu and click Create My SMTP Credentials.
Set an IAM User Name (it doesn’t really matter what you use, but you can’t use the name of another IAM user) and click Create.
Credentials are generated. Click Show User SMTP Security Credentials to view your credentials, and click Download Credentials to save them. You should save them to a safe place and back them up as this is the only time Amazon will present them to you. Also, definitely keep them safe, don’t email them around, etc as they can be used by anyone who has access to them to send/receive email…and that could turn your cheap SES solution into a more expensive one!
You will find that you’re now in the IAM UI. You can close this window or return to SES > SMTP Credentials to view the SMTP server name you’ll use (I like to copy the SMTP server name into my credentials file for ease of use later; mail clients will auto-detect which SMTP server to use with domains hosted on Amazon, but they are never correct, so you’ll need the correct SMTP server name later).

Create a Rule Set

Under Email Receiving in the menu, click Rule Sets, then click Create a Receipt Rule (note, if you’ve already set up a domain and are adding another, click View Active Rule Set and Create Rule).
Enter a recipient email address and click Add Recipient — for my use, I just set a rule to capture all incoming mail, enter mydomain.com. If there are multiple domains or recipients, you can add them here.

Verification status should show Verified as we verified our domain by updating DNS records.
Click Next Step.
From the Add Action menu, select S3.
From the S3 Bucket drop-down, select Create S3 Bucket.
In the Object Key Prefix field, provide a folder name within the S3 bucket to store incoming mail.
Leave Encrypt Message unchecked (in the future, there should hopefully be some method of allowing the Lambda function to decrypt messages so that they can be stored encrypted, but for now…)
From SNS Topic, select Create SNS Topic and choose a name to describe this action and click Create Topic. Then, ensure the topic you created is selected in the SNS Topic drop-down.
Click Next Step.
Provide a Rule Name and enable the Require TLS checkbox to ensure that TLS is used to encrypt messages in transit. Leave all other options at their default setting and click Next Step.
Review your changes and click Create Rule.

Create the Lambda Function

From the AWS Management Console, select Compute > Lambda.
If this is your first time using Lambda, click Get Started Now. If this is not your first time using Lambda, click Create a Lambda Function.
Select Blank Function.
Click Next.
In the Name and Description fields, enter a name and description for your Lambda function.
From the Runtime drop-down, select Node.js 6.10 (typically the default).
Inside the Lambda Function Code text field, paste the contents of Arithmetric’s AWS Lambda SES Forwarder.
Scroll down to var defaultConfig = and replace the example values in this section with your own values:
- fromEmail: The email address that will forward messages; must be one of the domains you configured to use with SES. noreply@mydomain is a good option.
- subjectPrefix: Optional, if you want the forwarder to prepend the forwarded email subject with a string, add it here.
- emailBucket: Your S3 bucket name you created when you configured your email rule.
- emailKeyPrefix: The S3 folder you’re using (followed by a /) that you created when you configured your email rule.
- forwardMapping: Instructions for mapping. To forward all email for a domain, use @domain.com.
The settings in the file itself serve as a pretty good example, and here’s what I use to forward mail from my two domains 7minuteserver.com and sevenminuteserver.com:
```
var defaultConfig = {
  fromEmail: "noreply@sevenminuteserver.com",
  subjectPrefix: "",
  emailBucket: "sevenminutebucket",
  emailKeyPrefix: "incomingemail/",
  forwardMapping: {
    "@sevenminuteserver.com": [
      "myemailname@gmail.com"
    ],
    "@7minuteserver.com": [
        "myemailname@gmail.com"
        ]
  }
};
```
In the Handler field, keep the default as index.handler.
In the Role field, select Create Custom Role. A new tab for IAM will open up.
In that tab, select Create a New IAM Role from IAM Role and provide a name for your new role.

Keeping the defaults, expand View Policy Document and click Edit next to it. Click OK to dismiss the Edit Policy warning, and then paste the following into the View Policy Document field:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
         ],
         "Resource": "arn:aws:logs:*:*:*"
      },
      {
         "Effect": "Allow",
         "Action": "ses:SendRawEmail",
         "Resource": "*"
      },
      {
         "Effect": "Allow",
         "Action": [
            "s3:GetObject",
            "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::S3-BUCKET-NAME/*"
      }
   ]
}

Change S3-BUCKET-NAME to your S3 bucket name (this is the bucket itself, not the folder you’re using).
Click Allow and you’ll be returned to your Lambda window.
Expand Advanced and set your timeout to 10 seconds or so. You can change this to a shorter time period once you’ve verified everything’s working as you want it to.
Click Next, review your function, then click Create Function.

Now we’re ready to add your function to your default mail rule set.

Add the Lambda Function to Your Rule Set

Access SES again, and, under Email Receiving, select Rule Sets.
Click View Active Rule Set and click the active rule to expand it.
You’ll see your base S3 rule — we’ll be adding our Lambda rule right after the S3 rule. Scroll down to the Add Action section and select Lambda.
From the Lambda function drop-down, select the Lambda function you just created. Leave Invocation Type set to Event and change the SNS Topic to the SNS topic you created earlier in the procedure, then click Save Rule.
If you get a permissions error, click Add Permissions.

Configure Your Webmail Client to Send As Your Domain User

I’ll provide instructions for Gmail, if you’re using another provider or email client, you can extrapolate. Note that email addresses you configure should have been verified in SES as senders via Identity Management > Email Addresses. If you haven’t done that yet, do that before proceeding.

Tap the Gear icon in your Gmail interface and select Settings.
Select the Accounts and Import tab.
In the Send Mail As section, click Add another email address.
In the Name field, enter the display name you want to use for the account.
In the Email Address field, enter the sender you want to use. You can use pretty much any valid name here, even if you haven’t “created” the address on Amazon.
If you want to use the account as an alias, enable the Treat as an Alias checkbox. If you select Treat as Alias, searches for “from:me” or “to:me”, for example, will return messages sent from or to the Send As address you configure.
Click Next Step.
On the next page, you may see that Gmail has automatically added an Amazon SMTP server. However, this isn’t what you’ll be using. Open up the SMTP settings you generated via Amazon SES, and enter the address you find there (if you’re using US-East, this value is likely to be email-smtp.us-east-1.amazonaws.com or email-smtp.us-west-2.amazonaws.com in US-East). Keep the default port value of 587.
In the Username and Password fields, enter the credentials that Amazon gave you. They look similar to your AWS credentials, but are specific to Amazon SMTP, a 20-character username and 44-digit password.
Ensure that Secured connection using TLS is enabled, and then click Add Account.
Google will send a confirmation email — it may take a few minutes. Click the link in your email or paste the code you receive into the Verify box.

When you send an email, you should now be able to click the arrow inside the From field of an email and select the email account you just added.

Try sending an email using your new domain to any address on your new domain. It should show up in your email box, sent from “noreply@yourdomain” (or whichever sender you configured). If you Reply, it should reply to the original sender, which is included in the email as the “Reply-To” address.

An idiosyncrasy I noticed: Gmail will ignore the “Reply-To” configured if it believes the Reply-To is also “you.”” For example, on my personal Gmail, I have “Send As” configured for one of my company email accounts. If I send from that Company account to hi@mynewamazondomain.com, and it is then forwarded to my personal account, when I reply with my personal account, it replies to hi@mynewamazondomain.com, not to me@corporateaccount.com. Confusing! Very confusing for testing, but shouldn’t be an issue when you start receiving email from people who are absolutely not you. :)

Troubleshooting

If you don’t receive the message, from the Services menu, search for Cloudwatch, open it, then click Logs, and select the log group that corresponds to your Lambda action. That should provide errors that will provide a clue as to what’s wrong.

Escape the Sandbox

To send more than 200 emails a day, and more importantly, to be able to respond to incoming email from non-verified accounts, you must ask Amazon to escape the sandbox. This is no big deal — they’ll actually send you an email after you start using SES asking you to do this and will include the link to request sandbox escape.

Or, just go to Sending Statistics under Email Sending in SES and click the big blue button to Request a Sending Limit Increase. When they approve you, you’ll also be able to accept email from non-verified accounts, as well as send.

Credits

Many thanks to https://anil.io/blog/aws/use-ses-lambda-mail-server-with-custom-domain-to-receive-emails/ and https://github.com/arithmetric/aws-lambda-ses-forwarder both of which helped me get this working.