Vaccinated (part 1)

As of yesterday morning, I've gotten my first shot of the ModeRNA1 COVID-19 vaccine!

I'd been trying to get an appointment since Berkeley opened them up to all adults on April 9th, and had no luck until 3am on Friday morning, when I managed to get one of the spots released by the CVS near my house (they were all gone a few minutes later). I was surprised to find when I arrived at CVS that, despite the intense competition to get vaccinated, supply is so tight that they were only being issued 10 doses (one vial) per day.

It's so frustrating that vaccines are still so hard to get here, and yet doses are sitting unused around the country. I don't know what the solution is to convince rural Republicans to get vaccinated, but I hope someone comes up with something to prevent the ultra-conservative parts of America from serving as a breeding ground for weird SARS-CoV-2 variants.

Anyhow, while I'm writing a post on COVID-19, here are some fun links:

At this point I'm kind of vacillating between optimism that we're all going to be vaccinated and safe soon, and deep pessimism that the right wing media echo chamber has created an insurmountable barrier of misinformed people with a near-pathological antipathy towards science and public health, even if we do make it through this pandemic, the next one is going to be some real Dark Ages shit.

Fun stuff.

Stay safe out there.

1

ModeRNA is such a cool name for a RNA vaccine company; I refuse to use their new boring brand capitalization of "moderna"

Surprising behavior in GNU tar

Here's a fun game for you: what do you expect to be the state of the filesystem after running the following commands in an empty directory on a Linux system?

$ touch foo:bar
$ tar -cpf foo:bar.tar foo:bar
$ rm foo:bar
$ tar -xpf foo:bar.tar

Do you expect the directory to contain the files foo:bar and foo:bar.tar?

What if I told you that instead the directory would only contain foo:bar.tar and stderr would say

tar (child): Cannot connect to foo: resolve failed

Yep! It turns out that GNU tar, if passed a filename containing a colon, treats the part before the colon as a hostname and attempts to connect to that host over rsh to download the part of the file after the colon. If you're not familiar with it, rsh is the completely-insecure predecessor to SSH. It's been at least 20 years since any reasonable system has shipped with RSH. This behavior is documented in Chapter 9.1 of the GNU tar manual but nobody I polled had ever heard of it.

Anyhow, GNU tar has a --force-local option to disable this behavior. If you ever process tar files whose names you do not completely control, or which might for some reason contain a colon, you should pass --force-local before every invocation.

An enterprising security researcher could probably have a lot of fun experimenting with vendor crap like security cameras and routers that take tar files with user-controlled filenames for firmware upgrades and see how many of them can be persuaded to establish a rsh shell to some attacker-controlled device...

I miss working from the office

I went into my office yesterday for the first time in a few months to pick some stuff up. We got notified a couple of days ago to get any personal property out of the office before Thanksgiving or else it'd be thrown out, so I guess we're moving out of the office. It was a pretty eerie place to be; even now, 8 months later, most people haven't been back and it kind of looks like the entire office was abducted by aliens in early March.

Despite how weird it is, I still miss working out of the office.

read more

Summertime California Sky

It's 9:45am on a cloudless Wednesday morning here in summertime California.

apocalyptic orange skies

My light meter reads 14 lux pointed directly at the sky — about as much as the middle of the night (with local light pollution), or about 1/1000 as much as a normal morning. They tell me that the skies are this apocalyptic shade of red due to the smoke from the 16 major wildfires currently burning in California1. Local news has plenty to say on the cause if you want to read up. Damn if it isn't disconcerting to live through, though.

What world did I bring my son into? Remember, this is probably going to be one of the cooler years over the next century... Vote, friends. Vote like your lives depend on it.

1

Including the LNU Lightning Fire Complex at 375,209 acres; the SCU Lightning Fire Complex at 396,624 acres; the CZU August Lightning Complex at 86,509 acres; and the Creek Fire at 163,138 acres. And also the El Dorado fire at at about 9,600 acres, which is notable because it was sparked by idiots who should not have access to pyrotechnic devices... or children.... Collectively these fires are burning an area twice the size of Rhode Island.

A Son!

Hello hypothetical readers; sorry for my absence, but I've spent the last weeks pretty well busy — as I foreshadowed in May, I have a son now!

baby isaac

It's been quite an adjustment. Isaac was born on August 7, 2020 after a very long1 labor, but he's happy and healthy now. I got 3 weeks off of work, which I spent with my wife doing intensive child care. Let me tell you, I have a whole new respect for single parents. Caring for an infant with just the two of us (since none of our families or friends can visit or help out due to COVID-19) is hard work. He needs to be fed every 2–3 hours, changed every 0.1 – 3 hours, and while he does sleep a lot, it's not really the kind of deep sleep where you can just put him in a crib and do other work. As of a couple of days ago, I'm back at work2, but my wife still has six or so more weeks of parental leave. We're still trying to do our best to spread the load, which means I'm spending most nights up all night3. Again, much props to all the single parents out there. Also: much props to grocery delivery. Things would be a lot harder if I had to gird myself against the pandemic and spend an hour getting groceries every time we ran out of something.

I guess you should prepare yourself for lots more baby pictures in the future.

My son's delivery was at the nearby hospital, which is owned by Sutter Health Group. Sutter Health also owns every other hospital in the area except the Kaiser in Oakland, which is only accessible to people on Kaiser insurance. I'm noting this first because, like most people in America, we didn't have any realistic choice about what hospital to go to, unless we were willing to drive to a different county (potentially an hour or more drive with traffic to get to UCSF).

Keeping in mind that Sutter Health has a monopoly on delivery centers in the area, you can imagine my surprise when we received the first bill for the delivery, in the amount of $103,736. No, that's not a typo. Nope, it's not missing a decimal point. One hundred and three thousand dollars. More than the average Californian makes in a year before tax. Yes, I have pretty good insurance and they're covering most of that4, but Jesus Fucking Christ, we were in the hospital for less than a week, in a labor & delivery ward (no ICU, etc) and had no unusual procedures performed. We weren't even in a very good hospital; there was a concerning level of dirt and grime, the cafeteria wasn't open on weekends, and a scary amount of the medical equipment was broken and had to be replaced during our stay. The doctors and nurses (especially the nurses) were good, but we spent maybe a total of four hours over the entire stay interacting with a doctor, and until the very end over the delivery5 we only saw a nurse once every two or three hours6. This isn't exactly concierge care!

I was always in favor of socialized medicine, but after seeing how (a) awful the care is, and (b) how incredibly, comically, expensive it is for a hospital stay that basically every human being goes through, I just want to take a moment to offer an emphatic middle finger to every Roger-Ailes-brainwashed voter and politician who's kept us in this nightmare system of private medicine while the rest of the world moved on to treating medical care as a right instead of a privilege.

1

49 hour

2

with "flexible hours", whatever that means, and still working from home (due to COVID-19)

3

I mean, not technically all night, but he tends to only sleep for 30-90 minute stretches during the night then want to eat and be burped and whatnot.

4

How much of that? Who knows! Some parts of the bill are "out-of-network" even though everything was done at an "in-network" hospital and it's not like anyone gave us choices over which doctors or nurses would perform specific procedures. A bunch of stuff is also missing, and presumably will be on some subsequent bill. I expect to end up paying somewhere between $2,000 and $20,000 out of pocket for the entire thing.

5

They give you a dedicated nurse once you pass the 36 hour mark, I guess?

6

In the L&D room, my wife was hooked up to a bunch of monitors because she was on pitocin and they require continuous monitoring of pulse, fetal pulse, blood oxygenation, and blood pressure when someone's on pitocin. Those monitors were connected to an old-school continuous feed printer and just generated a constant stream of paper containing her stats. That printer ran out of paper every 3 hours or so, at which point it would emit a piercing beep and flash bright red. Every time this happened, we would have to page the nurse to have someone come change the damn paper because apparently it's not a high priority that we're having a very loud alarm go off right next to my wife's head while she's in labor, and they have a policy of not pre-emptively replacing the paper before the alarm goes off. Separately, the pitocin or ringers bag would run out every few hours and need to be replaced. For the first day or so, the only times we saw any medical staff would be to introduce themselves at shift change 3x a day, and to come in after we paged them when these damned alarms were going off every few hours.

It's like being Slashdotted, but it's not 2002 any more

So, my etcd post was cross-posted to Hacker News and Reddit and probably more places. For the good of my own sanity, I'm not going to try to read through the comment threads on those other sites; generally, I try to avoid Hacker News anyway1. Just some brief, off-the-cuff follow-ups:

  • Yes, I recognize that the post from the other day was, uh, inflammatory. I did not write it as a persuasive thinkpiece targeted at the critical 18-25 demo or whatever. It's just some notes while I was in the process of rewriting some software from the etcd v2 API to the v3 API2.
  • This is my damned website and I'm going to be as snarky as I want, Anonymous Coward from 13 hours ago.
  • All nine million of you who wrote to me are absolutely right: as a user of open-source software I have the right to fork it if they decide to pull out the v2 API. I'm not super-interested in becoming the de facto maintainer of a database, particularly not one in a language that don't use very often, but I do have that right and ability. Thank you for writing to me.
  • Non-sarcastic thanks to everyone who pointed out typos and issues in the original post. Fun fact: Layer 3 and Layer 4 in the OSI model are not the same thing.
  • If you agree with me that simplicity is a virtue in software architecture and we embrace too many things that look like 90's-era Microsoft APIs, how about really driving it home by giving some money or time to Black Lives Matter, the SPLC, the ACLU, or other organizations working to simplify and improve our civil lives3?
  • If you vehemently disagree with me and think I'm human scum for not embracing the glorious combination of systemd, kubernetes, and the Registry Hive, why don't you really pwn me by donating money or time to an organization whose website probably runs on some k8s cluster somewhere like Black Lives Matter, the SPLC, or the ACLU?

Many thanks to DreamHost for successfully seeing this static-HTML website through a big traffic spike. I've been a customer for like 15 years and so far so good.

1

I find that Hacker News is really good at attracting people who think technology is cool, which is a particularly dangerous thing for anyone who actually expects to work in the tech industry. Computers are awful. You need to really embrace the hatred before you can be an effective technologist.

2

Yes, dear commenters, I do actually use the tools I complain about

3

This metaphor isn't a stretch at all, why do you ask?

Etcd, or, why modern software makes me sad

etcd icon

Once upon a time in 2013, there was a tool called etcd which was a really lightweight database written around the Raft consensus algorithm. This tool was originally written in 2013 for a bullshit unsuccessful project called CoreOS Container Linux that was EOL'd several years ago, but that doesn't really matter --- etcd was greater than its original use-case. Etcd provided a convenient and simple set of primitives (set a key, get a key, set-only-if-unchanged, watch-for-changes) with a drop-dead simple HTTP API on top of them. I have built a number of tools using etcd as a lightweight consensus store behind them and it's absolutely a pleasure to work with.

Hello **massive influx of new readers**! I see that some person who's out to get me kind soul has cross-posted this to Hacker News, Reddit, and a bunch of other sites. Cool! A few things you might want to know _before_ you send me hate-mail:
  • The word "rant" is right up there in the tags line. This is not meant to be a persuasive argument to the secret cabal that controls API design or a nuanced technical comparison article. It's just some off-the-cuff thoughts. Chillax.
  • If this didn't come across clearly enough in the article: I think etcd is great! I have written a bunch of tools and applications on top of it! I think it's a fantastic little dæmon and its API, even the new janky v3 API, is still a million times better than ZooKeeper

Okay, then. Read on.

In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers). I would go so far as to say that Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd. It's a comprehensive suite that promises to simplify operating clusters of software and give something like the experience of Google's borg cluster manager. What it really does is:

  1. Add hundreds of new failure modes to your software
  2. Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML
  3. Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking

If you are running a truly enormous system and want to have off-the-shelf orchestration for it, Kubernetes may be the tool for you. For 99.9% of people out there, it's just an extra layer of complexity that adds almost nothing of value.

I digress, though; this is a story about etcd. And, unfortunately, our stories come together because Kubernetes was quickly changed to use etcd as its state store. Thus began the rapid decline of etcd.

With the massive influx of Kubernetes users came, of course, a large number of Xooglers who decided to infect etcd with Google technologies, as is their way[^infection]2. Etcd's simple HTTP API was replaced by a "gRPC"3 version; the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys. etcd 3.2 added back a tiny subset of the HTTP API through the "gRPC Gateway", but not enough to implement any of the rich applications built on top of the original API. The v2 API lives on for now, but upstream threatens to remove it in every new version and there will surely come a time when it'll be removed entirely.

That's it. That's the story. Popular modern technology is taken over by expats from a megacorp and made worse in the service of a hyper-specialized (and just plain over-hyped) orchestration platform. That's the world today. Anything that has a simple and elegant feature-set ends up coöpted by people who just want to build big ungainly architecture and ends up inheriting features from whatever megacorp the coöpters came from4. The software development world would prefer to use their multi-gigabyte IDEs running on ElectronJS to build thousand-dependency Java applications targeting ungainly APIs on hard-to-operate systems than support something simpler and better. Quality is, alas, a dying art.

1

Read: "not good"

5

I've worked with a lot of Xooglers in my career (heck, I worked there myself). I now consider it to be a serious negative on someone's resume to have worked at Google. The many ex-Google coworkers I've had have (even when they've been otherwise brilliant) been uniformly less capable of working on non-Google systems than their much more junior equivalents with other backgrounds. All big companies have their own proprietary technology stacks, but the degree to which Googlers never learn how to do anything without involving protocol buffers, bigtable, and a mile-high stack of other proprietary tools is frankly remarkable. And they spread this to everything new they touch. Any open-source project will inevitably get pull requests to switch from JSON or BSON to Protocol Buffers; every web server now needs to support the cancerous user-hostile protocols of HTTP/26 and HTTP/37 that were passed in a mockery of IETF procedures.

2

Yes, I know, gRPC was added to etcd by Xiang Li, the original author of the project. That doesn't give them a pass for being influenced by the bad ideas coming out of Mountain View, or by their project's newfound popularity in the land of the Xooglers.

3

gRPC is Protocol Buffers8 running over HTTP/26. It's a got a "g" at the beginning of the name to remind you that the only time it's acceptable to use is when you are actually working for Google inside a Google-owned building eating Google-branded food and breathing Google-branded air.

8

Protocol Buffers or protobuf is Google's very bad serialization format

6

HTTP/2 a.k.a. SPDY is a comically bloated Layer 5/6/7 mega-combo protocol designed to replace HTTP. It takes something simple and (most importantly!) comprehensible and debuggable for junior programmers and replaces it with an insanely over-complicated9 system that requires tens of thousands of lines of code to implement the most minimal version of, but which slightly reduces page load time and server costs once you reach the point of doing millions of requests per second. I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web.

9

My favorite HTTP/2 interaction has been finding and reporting this bug in haproxy. The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 § Aa is just a list of the 61 most popular headers from Google properties.

7

HTTP/3 is all the badness of HTTP/2, but run over a worse layer 4 protocol named QUIC that totally fucks up networking for everybody in order to get a tiny bit more optimization for Google. That's all it does. It makes the Internet strictly worse for everybody but slightly better for the hugest of huge web properties. Nobody out here in the real Internet gives the slightest shit about head-of-line blocking from TCP, and lots of people want TCP state-aware firewalls and load-balancers to work.

4

I talk a lot of shit about Google, but Facebook and Microsoft are nearly as bad at turning out legions of ex-employees who can't be left alone in the room with a keyboard lest they attempt to recreate their previous employer's technology stack, poorly.

Trackballs

It's been about ten years since I've regularly used that most quintessential of post-1984 computer peripherals, the mouse. The last mouse I had was pretty exclusively used for a gaming PC in college and was a Logitech MX5181; since then I've used a variety of input devices. It's been a while since I did any brief reviews of technology on this ostensibly-technology blog, so what the hey, let's do it! In brief, we're going to talk about:

My first taste of a wrist-pain-free input device was the Logitech TrackMan Wheel Optical, a long-discontinued thumb trackball made in the mid-00's. According to my Amazon.com order history, I purchased this item on September 2, 2007. If you've never used one before, the idea is that you hold this device much like you would a "normal" ergonomic computer mouse, but instead of moving your whole arm your just move your thumb to control the cursor. The TrackMan Wheel (either in this, its "optical" variant, or the older ball-guided version) was a common first introduction to trackballs in the 00's. The tracking was always smooth2, the wheel was refreshingly clicky, and the buttons never failed. This was a great mouse. Unfortunately, it got lost in the move-out from college in 2010, and by that time it was discontinued and they were already going for >$100 on eBay. 😢

I replaced it with the new-at-the-time Logitech M570, which is ostensibly just a Logitech-Unifying-Receiver3-equipped variant of the TrackMan Wheel Optical. A few key notes here:

  • Generally, wireless sucks for mice. Even the Logitech Unifying Receiver, which is generally less flaky than Bluetooth, is still flaky and has human-visible latency
  • I don't know what happened to the controls on this mouse, but the buttons are not nearly as satisfying as the original and the wheel has always felt gummy
  • Forward and back buttons are dumb on a mouse

I used the M570 for a few years because it worked well enough, but it's not a very good input peripheral. I still have it (you can see a picture below, next to a stock shot of a TrackMan Wheel Optical) because there are a couple of games I can only play with it (unless I want to take an unreasonable amount of time to retrain muscle memory, and who wants to do that?)...

In 2012 or so, I had a breakthrough: I purchased a Kensington ExpertMouse. I'd seen these for years4 but never gotten around to regularly using one. The ExpertMouse has a single giant ball that you operate with the tips of your fingers and buttons around the edges. At some point, this trackball gained a label proclaiming it to be "joystick" and it became my standard input device; I took it with me to Uber and rolled many miles on that trackball.

Kensington ExpertMouse marketing shot

Unfortunately, when I left Uber in 2015, the ExpertMouse got nabbed by someone else off my desk and was never seen again. Even worse, some time between 2012 and 2015, Kensington drastically worsened their manufacturing process. I purchased a new ExpertMouse, but the entire thing feels much cheaper. The scroll wheels don't spin cleanly any more, the buttons are sticky, and the ball has to be cleaned twice as much as it used to. There's also a new wireless one that I hear very bad things about. Pre-2013 wired ExpertMouses (ExpertMice?) can sometimes be found on eBay for $150+.

After this, I took a detour and spent some time doing my input with trackpads instead of trackballs. You can see an Apple Magic TrackPad 2 in my new computer post from last year, and I had an Apple Magic TrackPad 1 at work for a while. The Magic TrackPad 2, in particular, is just an absolutely enormous (6¼" x 4½") slab of glass that serves as a multi-touch input for your computer. I still use mine and try to alternate every couple of weeks between a trackpad and a trackball to change up the kinds of stress on my wrists. If you've used a trackpad on any "modern" (unibody) MacBook, you know what a Magic TrackPad feels like. It's great for some things (like horizontal scrolling) and not great for other things (highly precise input; gaming).

But what about trackballs? Isn't that the purpose of this post?

For the last year or so I've been rocking two different Elecom trackballs: an Elecom DEFT PRO at home and an Elecom DEFT HUGE at work. Just today, I went into the office and brought my HUGE home, so I've got them side-by-side on my desk5:

Elecom DEFT Pro and DEFT Huge

Conceptually, they're very similar trackballs: a single large ball that you operate with your pointer and middle fingers, then a scroll wheel and buttons that you operate with your thumb and ring fingers. The HUGE is, well, HUGEr, with a ~25% bigger ball, a couple more buttons, and a built-in wrist-rest. The models I have are a wired-only HUGE (M-HT1UR) and a wired-or-wireless PRO (M-DPT1MR). I initially used the PRO wireless over Bluetooth to my desktop, but after one too many dropouts, I switched it to running wired over the micro USB cable. Unlike the Apple Magic TrackPad, when you plug in the Elecom, it does actually behave as a standard wired mouse instead of just charging.

I only have two complaints about the Elecom trackballs:

  • It's very hard to hold the wheel down. This is not a common operation unless you play a video game that uses holding the wheel down as an important action...
  • When wired, do not use standard USB HID classes so cannot be used on macOS for anything besides tracking and LMB/RMB without third-party drivers. SteerMouse seems to be the only option that supports all of HUGE's buttons on Catalina6. It will be a sad day if they ever drop support. Notably, BetterTouchTool, which I use for other input customization, does not support any of the extra buttons on the HUGE.

Anyhow, that's my brief run through input devices. If all of my mouse-like devices were destroyed in a mysterious laser strike, I would go out and buy another Elecom DEFT HUGE. For now. As I've found the hard way with both Logitech and Kensington, everything seems to get worse with time. Maybe I should just buy a few extra Elecom devices now now and put them away in an airtight box somewhere? No, that's probably crazy. I'm not Jon Siracusa...

For those of you who've made it this far, here's what my desk looks like today:

my desk, today

Stay safe out there, readers.

1

I bought mine in 2005. Remarkably, this is a product that Logitech still makes and sells fifteen years later for sixty American dollars. Capitalism!

2

At least, until gunk got stuck between the wheel and the Teflon nubbins it rolls on, at which point you flip the trackball over, push the ball out, and clean gunk off the nubbins. This is a requirement for all trackballs unless you wash your hands before every time you use your computer, and probably also have yourself surgically altered to no longer sweat.

3

The "Logitech Unifying Receiver" is a proprietary 2.4GHz radio dongle for Logitech keyboards and mice.

4

In fact, the ExpertMouse was one of the "standard" mice we stocked at the Tech Stop when I worked in Google Corporate IT.

5

Yes, these look gross. Hey, it's hard to use an input device for a protracted period without it looking gross. Just ignore the crumbs and stains.

6

I have Fn1 mapped to middle click, Fn2 mapped to show desktop, and Fn3 mapped to Mission Control née Exposé

2<sup>5</sup>

As every company I've ever bought a product from has been reminding me all morning, today is my birthday. 32 revolutions around the sun. Still ticking.

Obviously it's been a pretty busy year; most of that is in my post from last month. According to the accelerometer on my wrist, I've walked 2,904,398 steps in the last year1 but I'm mostly still in the same place as I was. Probably by the time I hit 33 revolutions around the sun, it'll be a very different story. Still got that baby on the way (another ~6 weeks to go), and my company just announced that we're going permanently remote so I guess that'll be a big life change.

Maybe I'll actually post more this coming year? Who knows!

1

Probably about 300,000 fewer than I would've without COVID-19.