Ode to a Pager

pager

I've been on-call for most of the last 11 years. I was on-call for the CS Department at Mudd1. I was on-call at Yelp, in a rotation that at times contained as few as three people. I was on-call at Uber in rotations ranging from one to twenty people. And I've been on-call at EasyPost — initially in a rotation with one other person2, and currently with two other people. I have responded to tens of thousands of pages. I have been woken up in the middle of the night hundreds3 of times. For the last seven or so years, I've worked at firms where on-call was a BYOD kind of a deal — you bring your own cell phone, register it in PagerDuty, and that's how you handle being on-call. This is my ode to the unfairly-hated pager, to the practices of yore.

Let's look at the phone you have in your pocket right now4:

  • It runs iOS or Android5
  • It gets at most two days of battery life
  • It receives phone calls, of which at least 90% are robots saying things like Hey buddy, this call is from the Department of Social Security
  • When it's not getting phone calls, it's constantly begging for your attention with notifications, most of which are some degree of spam

Is this the device you want to have to have on and audible 24 hours a day, 365 days a year? Do you love the idea of Apple's Do Not Disturb feature? Well, screw you because PagerDuty might need to reach you at any instant6. Do you miss going out into the woods for a hike? Too bad, Apple had to shave 0.7mm off the latest iPhone so now the antenna only works if it has direct line of sight to the AT&T worldwide headquarters in Dallas, TX. Want to quickly see what you're getting paged about? I hope you like watching this brief animation as all your icons swoosh in from whatever armpit of the universe they spend the off-time drinking in before you can actually do anything. Oh, you're using the native PagerDuty app? Well, then, you've got to give it 10 seconds to load (despite the fact that the Apple A12 CPU in your phone is faster than any computer CPU that existed anywhere on the planet 10 years ago) so that it can render some emojis and prompt you to take an "On-Call Selfie"7.

My ideal on-call device would look something like the following:

  • Small and lightweight
  • Extremely long battery life (imagine... weeks without recharging)
  • Only capable of receiving emergency notifications from PagerDuty so I can leave it on, unmuted at all times
  • On a network with great distance and building penetration8
  • Maybe a one or two line black-and-white display just long enough to print out messages like CRITICAL: web1sf - 4 packets transmitted, 0 received, 100% packet loss
  • Maybe two buttons so you could acknowledge or escalate incidents — but maybe not; I'm probably going to grab a laptop or bigger device to actually do the investigation9

Do you know what I've just described, you bunch of ingrates? You damned dirty apes? A bona fide two-way pager. We had the technology! We had built the perfect system! And we destroyed it! In our frivolous pursuit of only carrying one device, in our employers' endless pursuit of simpler procurement, we got rid of a system where your employer provides a simple-to-use single-function device to you, the employee, and replaced it with a system where you bring your own massively over-complicated device, pay your own connectivity bills, and then miss pages at 3 in the morning because you got too many goddamn Farmville notifications and your battery died.

You blew it up! You maniacs!
1

If I recall correctly, we had a physical pager and it went off exactly twice in the two years I was on the rotation.

2

Every on-call since Mudd has followed the best practice of having two people on-call at all times (Primary and Secondary) to minimize missed pages, so a two-person rotation means you are literally on-call 24 hours a day, 7 days a week, 366 days a year on leap years.

3

Hundreds might be conservative; I've never kept track but I'd say no less than once a month and no more than 20 times a month for the last 10 years, which gives us between 120 and 2000 wake-ups.

4

I don't really want to get into this side of things, but the other big issue is that this thing in your pocket is your device and it absolutely blows that the company who's already making you wake up at 3 in the morning is also making you use your personal property to do it. And if you've made the personal choice not to have a cell phone? Good luck getting a job in this industry, buddy.

5

From a security perspective, I dearly hope it either runs iOS or stock Android on a Pixel. Friends don't let friends use OEM-crapified Android.

6

Yes, I know, you can bypass Do Not Disturb for voice phone calls from specific numbers on iOS, and you can bypass it for more kinds of alerts on Android. I have never found the bypasses to be reliable.

7

Seriously: the on-call selfie thing was the most tone deaf crap I've ever seen from PagerDuty. When I get woken up at 3 in the morning because some other team broke something, I most definitely do not want to take my picture and tweet it.

8

How many industry outages do you think 5G/mmWave will cause from people losing their cell service because it got foggy?

9

That's right, I already have to have a second device with me all the time anyway because there are very few issues severe enough to page me but simple enough to fix from a 5" cell phone screen.

Ditching Gmail After 15 Years

Gmail Logo

Everyone who even casually follows the tech industry knows, intellectually, that Google builds an enormous dystopian profile of everything you do in order to sell ads1. But I think there's a difference between knowing that Google Analytics is a shameless back door to do cross-site tracking and actually coming face-to-face with your own profile. Yesterday, CNBC featured a story about a new Google UI which shows you a list of every purchase that you've made in the last few years: https://myaccount.google.com/purchases. For me, wide variety of commercial activity — every purchase I've ever made at a store that uses Square, every Amazon purchase, every Apple purchase, every movie ticket. I see gifts for my wife, work purchases, and even food for my rabbits. And on almost every item is a note: This purchase was found in your Gmail.

It's been clear for a long time that Google as a company no longer considers their users to be much more than piñatas full of delicious data; for me, seeing that list of the last 7+ years of every purchase I've made electronically is the last straw. As of today, I'm taking my personal domain (roguelazer.com) off of the grandfathered free-tier Gsuite2 mail hosting that it's had for the last 10+ years and moving it to a host that seems less inclined to aggressively mine it for data, and moving as many accounts as possible to no longer depend on my gmail.com account. I opened my Gmail account 15 years ago3, so I guess it's time to move on.

I went through the current crop of mail hosts, and evaluated them against the following conditions:

  • I don't want to self-host; I have no interest in actually maintaining (and monitoring and backing up) an SMTP and IMAP server. I get enough of my day job at my day job.
  • I prefer to use regular IMAP and SMTP protocols and mail clients (e.g., mutt); anything that requires that you use the provider's custom application is right out.
  • I'd like a hosting provider who's been around for a few years and seems unlikely to vanish off the face of the Earth tomorrow
  • Decent performance from the US

After doing some research, I decided that Fastmail was the least-worst option4. So I've migrated all my mail out of Gsuite onto Fastmail, re-pointed my MX records, and configured Gsuite to route any mails that still trickle into it over to Fastmail. Some time tomorrow when all the TTLs have expired, I will be shutting down my Gsuite account5.

If you want to have a chance in hell that our grandkids won't live in a surveillance dystopia, search with DuckDuckGo, close your Gmail account, switch to Firefox, and remind your elected officials that the world would be a better place without Big Data.

2

"Google Apps for Your Domain" when I signed up for it in 2007...

3

August 12, 2004. What a different time.

4

Yes, I've already had Protonmail recommended to me by a half-dozen people. I've tried it, but it misses half my criteria and at this point I think anyone pushing a commercial VPN as a product for any reason is probably a snake-oil salesman.

5

Don't worry, I used the helpful Google Takeout tool to dump out an .mbox of my mail for historical purposes.

GPG Key Transition

GnuPG Logo

My current PGP/GnuPG key is expiring, so I've rolled a new one. The ID of the new key is 0x3C7775DD37811E62 (full fingerprint: 1ED5 E5A3 01C3 D109 9040 2289 3C77 75DD 3781 1E62) and it should be in your favorite keyservers, cross-signed by my old key. You can also find it at https://files.roguelazer.com/roguelazer.gpg. It has also been attached to my keybase.io account and my Github profile. My previous key (0xAEE8F2454A41B87D) has not been revoked and has not been compromised, but you should still stop using it if possible. The new key is a 4096-bit RSA key with SHA-2 digest signatures — I'm not quite bold enough to switch to ECC for a long-lived key yet.

My signed transition document is below, and can also be found at 2019-04-27-key-transition-statement.txt.asc if you prefer to download it directly.

Additionally, I have generated a separately-signed key with ID 0x233E5EAF0EC3ABA9 (full fingerprint: 14E8 9660 188D BC9B 2C17 67AA 233E 5EAF 0EC3 ABA9). This key should not be used for communication, but will only be used to sign VCS commits/tags/&c (in Git and perhaps in Pijul). It's going to be on my [managed] work computer, so treat it with a grain of salt.

read more

Wildflowers

I went up yesterday with my wife's family to hike around on some Sonoma Land Trust property up in the North Bay and brought my camera. It's amazing to see the hills green and growing after so many years of drought! You can see a smattering of photos in this flickr album, but here are some of my favorites:

If you're in Northern California and are physically-able, you should try to get out and enjoy this; it's gorgeous!

Vim Setup: 2019

Neovim logo Vim logo

It's been a little while since I posted about my editor configuration, and I thought I might post what I'm using now. I guess the most notable change is that (after much prodding from my coworker Drew Ditthardt) I've switched from Vim to Neovim. Neovim is a vim-compatible editor written in C and Lua (as opposed to Vim, which is written in C, Vimscript, and prayers). I upgraded to Vim 8 last year and have had a few too many segmentation faults in the editor, so I decided to switch to something where more functionality was implemented in a memory-safe language. So far, Neovim has been pretty good to me, although the new process model means that it's pretty hard to write functions which invoke an external process which takes interactive input from a user.

As is probably expected for this sort of thing, here's a couple of screenshots; the first is of VimR, and the second is from NeoVim in Terminal.app, both editing files from rust-mysql-binlog:

read more

Site Updates

Some minor site updates over the last few days:

  • Comments are now powered by Commento instead of Disqus. This significantly reduces page load size on pages with comments.
  • Icons (social media, etc) are all now 2x resolution for modern retina screens.
  • Site-specific search is now powered by DuckDuckGo instead of Google. No longer loads an iframe, significantly faster site-loads, and better privacy to boot.
  • Archives page works again. Did this ever work?
  • Page is back to being fluid instead of fixed-width.
  • Improved CSS across the board, particularly on mobile devices.
  • Code page is updated.

I also managed to backport in some blog posts from the 2004-2007 era that were on the Internet Archive. For posterity!

Feel free to leave a comment or e-mail me if you notice any bugs.

Firefox Setup: 2019

Here's a quick post on how I use and configure Firefox on my Macs. The last time I posted about any of this was in 2011, and things have changed a fair bit since then. First, a screenshot (from my new iMac):

screenshot of firefox

Key extensions for this setup:

Vim Vixen : Vim keybindings for Firefox. Not quite as good as Vimperator, but works with modern Firefox, and getting better all the time. The only thing I really miss is macro recording, which I used all the time in Vimperator.

Tree-Style Tabs : Our monitors are widescreen; why would you try to stack tabs at the top of the window, taking up vertical space? Tree-style tabs lets you have a tab-hierarchy in a space-friendly manner.

uBlock Origin : I tried to go without an ad-blocker for a few years, to support independent businesses, but it's just not possible. Loading even relatively trustworthy sites (especially, for some reason, webcomics lately) is just a flood of malvertising now. uBlock Origin is the most popular ad-blocker these days. ¯\_(ツ)_/¯

Multi-Account Containers : This little extension makes it possible for me to have work tabs alongside personal tabs without corrupting all of my history all of the time. In particular, it lets me map sites (at the domain level) to automatically open in one tab container or another. I wish this supported wildcard domains so I could point every work domain at a container, but it's a start

New Container Tab : Adds a keybinding to open a new tab in the same container as the current tab (as opposed to the Firefox default, which opens the "Default", untagged container when you press Cmd-t

Mac OSX Light : This just makes the toolbar/titlebar color match the rest of macOS instead of being slightly off. It only works in "Light Mode".

1Password : Don't leave home without it.

One thing you might've noticed from the screenshot is that I've hidden tabs from the titlebar (since they're in the sidebar), but I don't have a big ugly blank space up there. That trick is actually the driving impetus for this blog post, since every other post on how to do that is broken as of Firefox 64. If you want the exact same appearance I have, just create a chrome/userChrome.css file inside your Firefox profile directory1 with the following contents:

Note: This was updated on December 1, 2024.

@namespace url("http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul");
/* hide the native tabs */
#TabsToolbar .toolbar-items {
	visibility: collapse;
}

/* fix the titlebar color and padding */
.browser-titlebar {
	background-color: var(--toolbar-bgcolor) !important;
	justify-content: space-between !important;
	padding: 8px 0 !important;
	--inactive-titlebar-opacity: 1.0;
}

/* hide the sidebar header so Tree-Style Tabs looks native */
#sidebar-header {
	visibility: collapse;
}

/* Hide the border under where native tabs would be, to get the "unified toolbar" appearance of modern macOS */
#navigator-toolbox {
	--tabs-border-color: transparent !important;
}

/* the coloration of the titlebar to look like a toolbar */
#titlebar {
	background: var(--toolbar-bgcolor);
}

/* hide a single stray vertical line that creeps in if you have tabs hidden */
#titlebar .titlebar-spacer[type="pre-tabs"] {
	border-inline-end: 0 !important;
}

You'll need to restart firefox for it to take effect.

NOTE: You may need to set the toolkit.legacyUserProfileCustomizations.stylesheets to true in about:config to get Firefox to load userChrome.css.

1

~/Library/Application Support/Firefox/Profiles/* on macOS; probably somewhere similar on other platforms.

Life Update: Moved

Hello infrequent readers: here's a piece of good news for once: as of October 27th, I've moved! $SPOUSE and I now live in the Elmwood neighborhood of Berkeley, California in a nice little two-bedroom one-and-a-half-bedroom single-family residence. It's a little surreal; even though where we live has a pretty nice mixed-use vibe (there's an apartment building around the corner, a bunch of restaurants nearby, and a Whole Foods only a few blocks away), it's still practically rural compared to the apartment in the Tendernob that I've lived in for the last eight years.

read more

Mastodon

I don't know if you're aware of this, but Twitter hasn't been a very good company in the last few years. Between the aiding and abetting of white supremacists, the continued hosting of our obnoxious orange tweeter-in-chief, and the ongoing user-hostile platform changes, it's just not as fun of a place as it was when I joined on November 14, 2007. So, uh, I'm not there any more.

No, I didn't delete my Twitter account. However, I have started trying to use a new microblogging system in its stead: Mastodon. Mastodon is an interesting idea: it's a federated social network1, which means it's made of a bunch of different copies run by different people but set up so that you can follow people and read their toots2 no matter which instance they happen to be on. I'm on mastodon.technology, which is generally themed around mainstream tech stuff, but I follow people at fosstodon and cybre.space and mastodon.social3. There are a couple of outstanding things about mastodon that I'd like to point out:

  • The local timeline is a timeline that shows every toot marked as "public" from people on your instance If you are on an instance that is relevant to your interests and which isn't too big, this can be a great way to get exposed to new people and new ideas. Mine has a few too many bitcoin enthusiasts on it, but is generally fun!4
  • Instances have rules and a code of conduct. Mastodon.technology (like many other instances) actively blocks hate speech and white supremacists. How novel!
  • You aren't the product! Most instances are either run by volunteers or funded by donations (mastodon.technology is funded by a Patreon), there are no ads, there's no garbage algorithmic timeline. It's just a chronological list of short mixed-media posts from people you choose to follow. Again... novel...

Of course, mastodon isn't the first technology product to try to move into Twitter's space. I also used the bizarrely- named App.net during its brief existence between mid-2013 and mid-2014; it was okay. There was also identi.ca in 2008 and the ever-popular Google Buzz, which I think I used when I was a Google employee 🤷‍♂️. The real advantage to Mastodon's design is that since it's open-source software and is federated, it's moderately harder for a single company to tank the platform. Of course, social networks are only useful as long as people you want to interact with use them, so it remains to be seen how well-populated Mastodon will be. If you're reading this, maybe you're someone who would be fun to interact with on Mastodon and you should make an account and follow me? Hm? If you still want more information, this Motherboard article is pretty good.5

The biggest weakness (for me) for Mastodon so far is a dearth of good native clients. I'm currently using Amaroq on my iPhone, which is feature complete but doesn't feel like a very good native iOS app, and beta testing Tusk, which feels like a native iOS app but is missing lots of features. There's nothing worth using yet on iPads or Macs. However! There is light at the end of the tunnel — Sean Huber of The Iconfactory (makers of Twitterrific, the best iOS/macOS Twitter client) appears to be working on a top-secret Mastodon client named Fantastodon. I'm excited.

So, yeah. If you @-me on Twitter, I probably won't respond6, but you should totally join me on Mastodon and follow me as @roguelazer@mastodon.technology. Awoo!

1

Mastodon folks call each Mastodon server an instance and the collective of all of them the fediverse. Technically, since the protocol is open, the fediverse also contains some other servers running totally unrelated packages such as GNU Social and Diaspora.

2

No, I don't really love toot as an un-trademarked substitute for tweet (especially since the bird association and the word tweet both came from third-party Twitter devs). Life goes on.

3

Mastodon.social is kind of the default instance for people who don't want to invest in picking a better one. You should pick a better one.

4

There's also something called the federated timeline which is basically the set-union of the timelines of everybody followed by anybody on your instance. Unless you are in a tiny and extremely focused instance, it's far too noisy to follow.

5

I didn't realize until I was writing this post that the Motherboard article was by Sarah Jeong, whose reporting in general is excellent and whose courtroom reporting for the ridiculous Oracle v Google Java trial was amazing. Another point for that article!

6

at least in part because I no longer receive notifications for Twitter because of their boneheaded API changes