How do I email?

posted tuesday, 2015-05-05 at 22:24 in category tech

Here are two things about me that some people don't know:

  1. I like e-mail. I mean, nobody looks forward to going through 700 e-mails every morning (which is about how many I get that I have to at least glance at), but it's far better than 700 meetings, 700 HipChats, 700 Slack messages, or anything else that requires synchronous attention. I'm all about being able to asynchronously "serially multitask", and being able to route everything through the dumb but asynchronous pipe of email makes that a lot easier. People who try to sell you on an e-mail-less office in favor of instant messaging tools are people who hate your productivity.
  2. I despise Gmail. I hate that most of the features only work in the awful web interface. I hate that the offline features of the mobile app only sort of work, and the web app hasn't been usable offline since Google Gears shut down. I hate that the IMAP server will sometimes turn off for 10 or 15 minutes and doesn't properly support the SEARCH command. Unfortunately, tech companies seem to exclusively use either Gmail or Outlook/Exchange, and Exchange is even worse than Gmail.

As you might expect given the intersection of those two facts, I have a pretty unusual mail setup. So I thought I'd share it on the Internet!

Overview

Here's a diagram of all of the pieces in my system:

mail architecture
  • gmail: the (in)famous web mail system (with IMAP and SMTP support)
  • offlineimap: a dæmon which synchronizes an IMAP server to a local mail repository
  • maildir: A very UNIX-y way to store mail on a local filesystem
  • mutt: A command-line e-mail client that I've been using for almost 10 years. Originally written by a fellow Mudd graduate!
  • notmuch: A fast e-mail indexer and full-text-search utility. Interacted with through mutt-notmuch.
  • muttdown: A script to compile outbound markdown text into multipart/alternative HTML e-mail

Configuration

This is going to be a long post. Pay attention to all of the users in these configs, because you'll need to sub them out.

I'm not going to cover everything here; in particular, I like to sign my outgoing mail with GPG, and that's well beyond the scope of this post.

Gmail

Make sure that IMAP is enabled on your Gmail account.

Create an application-specific password and put it in the OS X Keychain by creating a new "application password" with the name "mutt" and the Account of "foo@bar.com" (or whatever your email address is).

offlineimap

I've experimented with a bunch of different IMAP configurations — native mutt IMAP support (tends to hang), offlineimap running as a dæmon with experimental IDLE support (dies horribly if the network goes down), etc. I've settled on running offlineimap under launchd every couple of minutes. Here's my offlineimap config:

[general]
ui = basic
accounts = Work
pythonfile = ~/.offlineimap.py
socktimeout = 30

[Account Work]
localrepository = Work-Local
remoterepository = Work-Remote
status_backend = sqlite
postsynchook = notmuch new

[Repository Work-Local]
type = Maildir
localfolders = ~/Mail/Work

[Repository Work-Remote]
type = Gmail
remoteuser = foo@bar.com
remotepasseval = get_password_from_keychain("foo@bar.com")
realdelete = no
maxconnections = 5
cert_fingerprint = 77279a9a8bb4a9b3707e045a9ec0ab952f574dd7
folderfilter = lambda foldername: foldername not in ('[Gmail]/All Mail', '[Gmail]/Important',) and (foldername in ('devops/debmirror', 'devops/sentry') or not foldername.startswith('devops/'))

The cert_fingerprint setting is kind of fragile-feeling, and I should probably just install a proper CA bundle and point offlineimap at that. shrug

This depends on the following script being installed at ~/.offlineimap.py (to extract IMAP credentials from the OS X keychain):

#!/usr/bin/env python

import subprocess


def get_password_from_keychain(search):
    stdout = subprocess.check_output(
        ['/usr/bin/security', 'find-generic-password', '-w', '-s', 'mutt', '-a', search]
    )
    return stdout.strip()

To get launchd working, drop the following in ~/Library/LaunchAgents/org.offlineimap.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
   "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
        <string>org.offlineimap</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/offlineimap</string>
    </array>
    <key>RunAtLoad</key>
        <true />
    <key>StartInterval</key>
        <integer>60</integer>
    <key>TimeOut</key>
        <integer>180</integer>
    <key>StandardErrorPath</key>
        <string>/Users/foo/log/offlineimap_error.log</string>
    <key>StandardOutPath</key>
        <string>/Users/foo/log/offlineimap.log</string>
    <key>KeepAlive</key>
        <dict>
            <key>NetworkState</key>
            <true />
        </dict>
</dict>
</plist>

You'll probably want to run your first sync by running offlineimap -o and watching it; it took me a few days to download my 250,000 messages. Then run launchctl load ~/Library/LaunchAgents/org.offlineimap.plist to get syncing running regularly. Woo!

notmuch

Once you have some mail, you can index it by running notmuch new.

Since we're going to want to integrate notmuch with mutt, you should download mutt-notmuch from https://github.com/Roguelazer/mutt-notmuch-py, put it in ~/bin/, and make it executable.

muttdown

muttdown is a program I wrote that allows me to compose my mail in Markdown and have it transparently compiled into HTML mail for my co-workers who prefer rich emails. You can download it from https://github.com/Roguelazer/muttdown and install it like any Python program; after that, make a config file in ~/.muttdown.yaml that looks like the following:

smtp_host: smtp.gmail.com
smtp_port: 587
smtp_ssl: false
smtp_username: foo@bar.com
smtp_password_command: security find-generic-password -w -s mutt -a foo@bar.com

(no, don't worry, that doesn't actually disable TLS; it just switches from SMTPS to SMTP + STARTTLS)

mutt

Mutt is the best mail client, hands-down. Explaining how to use it would be a post in and of itself. I'm not going to go too far into it, but here's my ~/.muttrc:

set from = "foo@bar.com"
set realname = "James Brown"
set sendmail = "muttdown"
set envelope_from = yes

set mbox_type=Maildir
set folder="~/Mail/Work"
set spoolfile = "=INBOX"
set postponed = "=[Gmail].Drafts"
set mask=".*"

mailboxes =INBOX/ =security/ =engineering/ =engineering.outages/ =infra/

folder-hook "+[Gmail].Sent Mail" set sort=date-sent
folder-hook . "exec collapse-all"
set header_cache=~/.mutt-local/cache/headers
set message_cachedir=~/.mutt-local/cache/bodies
set certificate_file=~/.mutt-local/certificates

set move=no

ignore headers *
unignore headers from to subject date cc bcc
hdr_order From Date: To: From: Cc: Bcc: Subject:
set mime_forward=ask-yes
set index_format="%4C %?M?(+%3M)&      ?%Z %-20.20F (%4c) %-10.200s %> %[%b %d %R]"
set folder_format="%2C %t %N %f %> %d"
alternative_order text/plain text/enriched multipart/signed text/html *
auto_view text/html

set duplicate_threads
set forward_format="Fwd: %s"

set sort=threads
unset strict_threads
set sort_aux=reverse-last-date-sent
set editor="vim -c 'set syntax=mail ft=mail enc=utf-8'"

set followup_to
set honor_followup_to=yes
unset record

set delete_untag
set mark_old=yes
set reply_to=yes
unset reply_self

set pgp_autosign


source ~/.mutt/macros

set pgp_mime_signature_filename="signature.asc"
set pgp_mime_signature_description="gpg signature"

set alias_file=~/.mutt/aliases
source $alias_file
set reverse_alias=no

unset collapse_unread

This depends on a couple of other files. First, ~/mutt/macros:

macro index S ":toggle pgp_verify_sig<enter>"
macro index A ":toggle pgp_autosign<enter>"
bind index,pager c mail
bind index,pager a group-reply
bind index,pager s flag-message
bind index,pager F noop
macro index,pager m "<save-message>?" "move a message to a mailbox"
bind index,pager y delete-message
macro index I "<tag-prefix><clear-flag>N" "mark tagged messages as read"
macro index \CU "<untag-pattern>~A<enter>"
macro index,pager gi "<change-folder>\Cu=INBOX<enter>" "go to inbox"
macro index,pager gs "<change-folder>\Cu=[Gmail].Starred<enter>" "go to starred"
macro index,pager gl "<change-folder>\Cu=" "open folder"
macro index,pager G "<change-folder>?" "browse folder"

bind index - collapse-thread
bind index _ collapse-all

macro index g/ \
    "<enter-command>unset wait_key<enter><shell-escape>~/bin/mutt-notmuch --prompt search<enter><change-folder-readonly>~/.cache/mutt_results<enter>" \
    "search mail (using notmuch)"

# vim: set syntax=muttrc:

Many of these macros are based off of the Gmail interface, so if you master them, you'll be able to operate quasi-proficiently in those times when you're forced to use Gmail. The most unique one to be aware of is g/, which will launch a search of all mailboxes.

Finally, ~/.mailcap:

text/html; links %s ; nametemplate=%s.html
text/html; links -dump %s ; nametemplate=%s.html ; copiousoutput
image/png ; qlmanage -p %s 2>/dev/null ; nametmplate=%s.png
image/jpg ; qlmanage -p %s 2>/dev/null ; nametmplate=%s.jpg
image/gif ; qlmanage -p %s 2>/dev/null ; nametmplate=%s.gif

This will enable you to view HTML mail inline, and to view images in a QuickLook popup.

Conclusions

Well, that's what it looks like today. Probably by this time next year, I'll have something totally new!

The biggest thing I'm missing is a good way to manage Gmail filters locally. On my own mail server, I have a non-trivial collection of procmail rules, and I think that my next project might be something that compiles a procmailrc into a Gmail XML filter description and then somehow uploads it.

Let me know if you have any questions, concerns, or favorite fish.

Want to comment on this? How about we talk on Mastodon instead? mastodon logo Share on Mastodon