I've been on-call for most of the last 11 years. I was on-call for the CS Department at Mudd1. I was on-call at Yelp, in a rotation that at times contained as few as three people. I was on-call at Uber in rotations ranging from one to twenty people. And I've been on-call at EasyPost — initially in a rotation with one other person2, and currently with two other people. I have responded to tens of thousands of pages. I have been woken up in the middle of the night hundreds3 of times. For the last seven or so years, I've worked at firms where on-call was a BYOD kind of a deal — you bring your own cell phone, register it in PagerDuty, and that's how you handle being on-call. This is my ode to the unfairly-hated pager, to the practices of yore.
Let's look at the phone you have in your pocket right now4:
- It runs iOS or Android5
- It gets at most two days of battery life
- It receives phone calls, of which at least 90% are robots saying things like
Hey buddy, this call is from the Department of Social Security
- When it's not getting phone calls, it's constantly begging for your attention with notifications, most of which are some degree of spam
Is this the device you want to have to have on and audible 24 hours a day, 365 days a year? Do you love the idea of Apple's Do Not Disturb feature? Well, screw you because PagerDuty might need to reach you at any instant6. Do you miss going out into the woods for a hike? Too bad, Apple had to shave 0.7mm off the latest iPhone so now the antenna only works if it has direct line of sight to the AT&T worldwide headquarters in Dallas, TX. Want to quickly see what you're getting paged about? I hope you like watching this brief animation as all your icons swoosh in from whatever armpit of the universe they spend the off-time drinking in before you can actually do anything. Oh, you're using the native PagerDuty app? Well, then, you've got to give it 10 seconds to load (despite the fact that the Apple A12 CPU in your phone is faster than any computer CPU that existed anywhere on the planet 10 years ago) so that it can render some emojis and prompt you to take an "On-Call Selfie"7.
My ideal on-call device would look something like the following:
- Small and lightweight
- Extremely long battery life (imagine... weeks without recharging)
- Only capable of receiving emergency notifications from PagerDuty so I can leave it on, unmuted at all times
- On a network with great distance and building penetration8
- Maybe a one or two line black-and-white display just long enough to print out messages like
CRITICAL: web1sf - 4 packets transmitted, 0 received, 100% packet loss - Maybe two buttons so you could acknowledge or escalate incidents — but maybe not; I'm probably going to grab a laptop or bigger device to actually do the investigation9
Do you know what I've just described, you bunch of ingrates? You damned dirty apes? A bona fide two-way pager. We had the technology! We had built the perfect system! And we destroyed it! In our frivolous pursuit of only carrying one device, in our employers' endless pursuit of simpler procurement, we got rid of a system where your employer provides a simple-to-use single-function device to you, the employee, and replaced it with a system where you bring your own massively over-complicated device, pay your own connectivity bills, and then miss pages at 3 in the morning because you got too many goddamn Farmville notifications and your battery died.
If I recall correctly, we had a physical pager and it went off exactly twice in the two years I was on the rotation.
Every on-call since Mudd has followed the best practice of having two people on-call at all times (Primary and Secondary) to minimize missed pages, so a two-person rotation means you are literally on-call 24 hours a day, 7 days a week, 366 days a year on leap years.
Hundreds might be conservative; I've never kept track but I'd say no less than once a month and no more than 20 times a month for the last 10 years, which gives us between 120 and 2000 wake-ups.
I don't really want to get into this side of things, but the other big issue is that this thing in your pocket is your device and it absolutely blows that the company who's already making you wake up at 3 in the morning is also making you use your personal property to do it. And if you've made the personal choice not to have a cell phone? Good luck getting a job in this industry, buddy.
From a security perspective, I dearly hope it either runs iOS or stock Android on a Pixel. Friends don't let friends use OEM-crapified Android.
Yes, I know, you can bypass Do Not Disturb for voice phone calls from specific numbers on iOS, and you can bypass it for more kinds of alerts on Android. I have never found the bypasses to be reliable.
Seriously: the on-call selfie thing was the most tone deaf crap I've ever seen from PagerDuty. When I get woken up at 3 in the morning because some other team broke something, I most definitely do not want to take my picture and tweet it.
How many industry outages do you think 5G/mmWave will cause from people losing their cell service because it got foggy?
That's right, I already have to have a second device with me all the time anyway because there are very few issues severe enough to page me but simple enough to fix from a 5" cell phone screen.
Want to comment on this? How about we talk on Mastodon instead?
Share on Mastodon