send-email-aws.html(Python)
Import Notebook

Sending Email Overview

Sending an email from with Databricks is quite easy using Amazon SES. In order to do this, you'll have to make sure that you have access and proper permissioning with the AWS SES service. The permission you need is ses:SendRawEmail. You can set that up on the cluster IAM Role or you can set it up with an access key. See here for more information on access control on Amazon SES.

Now Let's follow the steps that you'll need to complete in order to actually send an email.

Helper Functions

We're going to define several helper function in order to make it easier to actually send and construct an email.

Send Email Function

This function just provides a simple interface for actually creating and sending an email. It allows us to send an email to multiple people, with attachments as well as inlined images. You'll notice that this leverages the boto library to send them email.

import boto3

import matplotlib.pyplot as plt
from email import encoders
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart
from email.mime.application import MIMEApplication


def send_email(from_email, to_emails, subject, body_html, attachments=[], cc=[], bcc=[]):
  attachment_ready_html = []
  img_id = 0
  mime_images = []
  # iterate over raw HTML
  for l in body_html:
    # For each image in the body_html, convert each to a base64 encoded inline image
    if l.startswith("<img"):
      image_data = l[len("<img src='data:image/png;base64,"):-2]
      mime_img = MIMEImage(base64.standard_b64decode(image_data))
      mime_img.add_header('Content-ID', '<img-%d>' % img_id)
      attachment_ready_html.append("<center><img src='cid:img-%d'></center>" % img_id)
      img_id += 1
      mime_images.append(mime_img)
    else:
      attachment_ready_html.append(l)
  print("Added {} images".format(img_id))

  msg = MIMEMultipart()
  msg['Subject'] = subject
  msg['From'] = from_email
  msg['To'] = ", ".join(to_emails)
  body = MIMEText('\n'.join(attachment_ready_html), 'html')
  
  for i in mime_images:
    msg.attach(i)
    
  msg.attach(body)
  
  for raw_attachment in attachments:
    attachment = MIMEApplication(open(raw_attachment, 'rb').read())
    attachment.add_header('Content-Disposition', 'attachment', filename=raw_attachment)
    msg.attach(attachment)
  
  ses = boto3.client('ses', region_name='us-west-2')
  ses.send_raw_email(
    Source=msg['FROM'],
    Destinations=to_emails,
    RawMessage={'Data': msg.as_string()})    
  print "Sending Email."

Image Construction

The following function helps us create an image in a way that is actually compatible with email HTML. In short, HTML emails do not support the same image semantics and inline imaging that regular HTML does. Therefore, we can use this function to convert a matplotlib image that we create in a format that we can both display in a notebook and inline in an email. The details are largely irrelevant, just pass in a matplotlib image (either from pandas or manually created) and you're good to go.

from uuid import uuid4
import datetime
import base64

def makeCompatibleImage(image, withLabel=False):
  if withLabel:
    image.figure.suptitle("Generated {}".format(datetime.datetime.today().strftime("%Y-%m-%d")))
  imageName = "/tmp/{}.png".format(uuid4())
  image.figure.savefig(imageName)
  plt.close(image.figure)
  val = None
  with open(imageName) as png:
    val = "<img src='data:image/png;base64,%s'>" % base64.standard_b64encode(png.read())
  displayHTML(val)
  return val

Previewing the Email

Our last function is a simple helper function to help us preview our email. This takes a list of string objects as input (and filters out erroneously added objects in the process).

def previewHTMLEmail(html):
  displayHTML("\n".join([x for x in html if type(x) != type(type)]))

Creating an email

Now that we defined our helper functions, it's time to actually create our email. To do this, we're going to create a list object that we will append HTML strings to. In this example, we'll call it html but this list could have any name.

html = [
  "<center><h1>Example Email Title</h1></center>",
  """
  <p><b>A standard paragraph.</b></p>
  <ol>
      <li>This demonstrates a list.</li>
  </ol>
  """
]

At this point, we've filled in some basic content. Now we can create a plot and append it to the list. In this example, we'll create a nice radar plot using matplotlib. Notice how we use the makeCompatibleImage function to both display the image that we create, as well as convert it into the proper form for our html page.

import numpy as np
import matplotlib.pyplot as plt


# Compute pie slices
N = 20
theta = np.linspace(0.0, 2 * np.pi, N, endpoint=False)
radii = 10 * np.random.rand(N)
width = np.pi / 4 * np.random.rand(N)

ax = plt.subplot(111, projection='polar')
bars = ax.bar(theta, radii, width=width, bottom=0.0)

# Use custom colors and opacity
for r, bar in zip(radii, bars):
    bar.set_facecolor(plt.cm.viridis(r / 10.))
    bar.set_alpha(0.5)
# Convert image add append to html array
html.append(makeCompatibleImage(ax))

We can of course continue adding more content by simply appending to the list.

html.extend([
    "<br>"
" Some more",
    "<p>items go here</p>"
])

Email attachments

Now we've created an email that will inline text and images. However, sometimes you'll want to include a physical file (like a csv) as well. In order to do this, we simply specify a file as an attachment. In our case, we'll create a list called "attachments" that specifies filenames on the driver node. Therefore this can be a result of a DataFrame that you collect to the driver after your analysis and write out using Pandas.

with open('/tmp/some-file.txt', 'wt') as f:
  f.write("Hello World")

attachments = ['/tmp/some-file.txt']

Now that we've done all of this, we can preview our email content.

previewHTMLEmail(html)

Email Metadata

Now that we specified all of the content and attachments, we can specify who we would like to send the email to. We can also specify our subject line.

from_addr = "johndoe@gmail.com"
to_addrs = [
  "person1@gmail.com",
  "person2@gmail.com"
]
subject = "A Fantastic Databricks Report"

Send the Email!

Now we're all set, we can simply send the email specify each of the above details. html in this context is the content of our email while attachments are all the things we'd like to attach.

send_email(from_addr, to_addrs, subject, html, attachments=attachments)
Added 2 images Sending Email.

Conclusion

You can now specify this as a repeated job or augment it with more information!