How do I email?
Here are two things about me that some people don't know:
- I like e-mail. I mean, nobody looks forward to going through 700 e-mails every morning (which is about how many I get that I have to at least glance at), but it's far better than 700 meetings, 700 HipChats, 700 Slack messages, or anything else that requires synchronous attention. I'm all about being able to asynchronously "serially multitask", and being able to route everything through the dumb but asynchronous pipe of email makes that a lot easier. People who try to sell you on an e-mail-less office in favor of instant messaging tools are people who hate your productivity.
- I despise Gmail. I hate that most of the features only work in the awful web interface. I hate that the offline features of the mobile app only sort of work, and the web app hasn't been usable offline since Google Gears shut down. I hate that the IMAP server will sometimes turn off for 10 or 15 minutes and doesn't properly support the
SEARCHcommand. Unfortunately, tech companies seem to exclusively use either Gmail or Outlook/Exchange, and Exchange is even worse than Gmail.
As you might expect given the intersection of those two facts, I have a pretty unusual mail setup. So I thought I'd share it on the Internet!
Here's a diagram of all of the pieces in my system:
- gmail: the (in)famous web mail system (with IMAP and SMTP support)
- offlineimap: a dæmon which synchronizes an IMAP server to a local mail repository
- maildir: A very UNIX-y way to store mail on a local filesystem
- mutt: A command-line e-mail client that I've been using for almost 10 years. Originally written by a fellow Mudd graduate!
- notmuch: A fast e-mail indexer and full-text-search utility. Interacted with through mutt-notmuch.
- muttdown: A script to compile outbound markdown text into
This is going to be a long post. Pay attention to all of the users in these configs, because you'll need to sub them out.
I'm not going to cover everything here; in particular, I like to sign my outgoing mail with GPG, and that's well beyond the scope of this post.
Make sure that IMAP is enabled on your Gmail account.
Create an application-specific password and put it in the OS X Keychain by creating a new "application password" with the name "mutt" and the Account of "firstname.lastname@example.org" (or whatever your email address is).
I've experimented with a bunch of different IMAP configurations -- native mutt IMAP support (tends to hang), offlineimap running as a dæmon with experimental IDLE support (dies horribly if the network goes down), etc. I've settled on running offlineimap under launchd every couple of minutes. Here's my offlineimap config:
[general] ui = basic accounts = Work pythonfile = ~/.offlineimap.py socktimeout = 30 [Account Work] localrepository = Work-Local remoterepository = Work-Remote status_backend = sqlite postsynchook = notmuch new [Repository Work-Local] type = Maildir localfolders = ~/Mail/Work [Repository Work-Remote] type = Gmail remoteuser = email@example.com remotepasseval = get_password_from_keychain("firstname.lastname@example.org") realdelete = no maxconnections = 5 cert_fingerprint = 77279a9a8bb4a9b3707e045a9ec0ab952f574dd7 folderfilter = lambda foldername: foldername not in ('[Gmail]/All Mail', '[Gmail]/Important',) and (foldername in ('devops/debmirror', 'devops/sentry') or not foldername.startswith('devops/'))
cert_fingerprint setting is kind of fragile-feeling, and I should probably just install a proper CA bundle and point offlineimap at that. shrug
This depends on the following script being installed at
~/.offlineimap.py (to extract IMAP credentials from the OS X keychain):
#!/usr/bin/env python import subprocess def get_password_from_keychain(search): stdout = subprocess.check_output( ['/usr/bin/security', 'find-generic-password', '-w', '-s', 'mutt', '-a', search] ) return stdout.strip()
To get launchd working, drop the following in
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>org.offlineimap</string> <key>ProgramArguments</key> <array> <string>/usr/local/bin/offlineimap</string> </array> <key>RunAtLoad</key> <true /> <key>StartInterval</key> <integer>60</integer> <key>TimeOut</key> <integer>180</integer> <key>StandardErrorPath</key> <string>/Users/foo/log/offlineimap_error.log</string> <key>StandardOutPath</key> <string>/Users/foo/log/offlineimap.log</string> <key>KeepAlive</key> <dict> <key>NetworkState</key> <true /> </dict> </dict> </plist>
You'll probably want to run your first sync by running
offlineimap -o and watching it; it took me a few days to download my 250,000 messages. Then run
launchctl load ~/Library/LaunchAgents/org.offlineimap.plist to get syncing running regularly. Woo!
Once you have some mail, you can index it by running
Since we're going to want to integrate notmuch with mutt, you should download
mutt-notmuch from https://github.com/Roguelazer/mutt-notmuch-py, put it in
~/bin/, and make it executable.
muttdown is a program I wrote that allows me to compose my mail in Markdown and have it transparently compiled into HTML mail for my co-workers who prefer rich emails. You can download it from https://github.com/Roguelazer/muttdown and install it like any Python program; after that, make a config file in
~/.muttdown.yaml that looks like the following:
smtp_host: smtp.gmail.com smtp_port: 587 smtp_ssl: false smtp_username: email@example.com smtp_password_command: security find-generic-password -w -s mutt -a firstname.lastname@example.org
(no, don't worry, that doesn't actually disable TLS; it just switches from SMTPS to SMTP + STARTTLS)
Mutt is the best mail client, hands-down. Explaining how to use it would be a post in and of itself. I'm not going to go too far into it, but here's my
set from = "email@example.com" set realname = "James Brown" set sendmail = "muttdown" set envelope_from = yes set mbox_type=Maildir set folder="~/Mail/Work" set spoolfile = "=INBOX" set postponed = "=[Gmail].Drafts" set mask=".*" mailboxes =INBOX/ =security/ =engineering/ =engineering.outages/ =infra/ folder-hook "+[Gmail].Sent Mail" set sort=date-sent folder-hook . "exec collapse-all" set header_cache=~/.mutt-local/cache/headers set message_cachedir=~/.mutt-local/cache/bodies set certificate_file=~/.mutt-local/certificates set move=no ignore headers * unignore headers from to subject date cc bcc hdr_order From Date: To: From: Cc: Bcc: Subject: set mime_forward=ask-yes set index_format="%4C %?M?(+%3M)& ?%Z %-20.20F (%4c) %-10.200s %> %[%b %d %R]" set folder_format="%2C %t %N %f %> %d" alternative_order text/plain text/enriched multipart/signed text/html * auto_view text/html set duplicate_threads set forward_format="Fwd: %s" set sort=threads unset strict_threads set sort_aux=reverse-last-date-sent set editor="vim -c 'set syntax=mail ft=mail enc=utf-8'" set followup_to set honor_followup_to=yes unset record set delete_untag set mark_old=yes set reply_to=yes unset reply_self set pgp_autosign source ~/.mutt/macros set pgp_mime_signature_filename="signature.asc" set pgp_mime_signature_description="gpg signature" set alias_file=~/.mutt/aliases source $alias_file set reverse_alias=no unset collapse_unread
This depends on a couple of other files. First,
macro index S ":toggle pgp_verify_sig<enter>" macro index A ":toggle pgp_autosign<enter>" bind index,pager c mail bind index,pager a group-reply bind index,pager s flag-message bind index,pager F noop macro index,pager m "<save-message>?" "move a message to a mailbox" bind index,pager y delete-message macro index I "<tag-prefix><clear-flag>N" "mark tagged messages as read" macro index \CU "<untag-pattern>~A<enter>" macro index,pager gi "<change-folder>\Cu=INBOX<enter>" "go to inbox" macro index,pager gs "<change-folder>\Cu=[Gmail].Starred<enter>" "go to starred" macro index,pager gl "<change-folder>\Cu=" "open folder" macro index,pager G "<change-folder>?" "browse folder" bind index - collapse-thread bind index _ collapse-all macro index g/ \ "<enter-command>unset wait_key<enter><shell-escape>~/bin/mutt-notmuch --prompt search<enter><change-folder-readonly>~/.cache/mutt_results<enter>" \ "search mail (using notmuch)" # vim: set syntax=muttrc:
Many of these macros are based off of the Gmail interface, so if you master them, you'll be able to operate quasi-proficiently in those times when you're forced to use Gmail. The most unique one to be aware of is
g/, which will launch a search of all mailboxes.
text/html; links %s ; nametemplate=%s.html text/html; links -dump %s ; nametemplate=%s.html ; copiousoutput image/png ; qlmanage -p %s 2>/dev/null ; nametmplate=%s.png image/jpg ; qlmanage -p %s 2>/dev/null ; nametmplate=%s.jpg image/gif ; qlmanage -p %s 2>/dev/null ; nametmplate=%s.gif
This will enable you to view HTML mail inline, and to view images in a QuickLook popup.
Well, that's what it looks like today. Probably by this time next year, I'll have something totally new!
The biggest thing I'm missing is a good way to manage Gmail filters locally. On my own mail server, I have a non-trivial collection of procmail rules, and I think that my next project might be something that compiles a procmailrc into a Gmail XML filter description and then somehow uploads it.
Let me know if you have any questions, concerns, or favorite fish.