It's like being Slashdotted, but it's not 2002 any more

So, my etcd post was cross-posted to Hacker News and Reddit and probably more places. For the good of my own sanity, I'm not going to try to read through the comment threads on those other sites; generally, I try to avoid Hacker News anyway1. Just some brief, off-the-cuff follow-ups:

  • Yes, I recognize that the post from the other day was, uh, inflammatory. I did not write it as a persuasive thinkpiece targeted at the critical 18-25 demo or whatever. It's just some notes while I was in the process of rewriting some software from the etcd v2 API to the v3 API2.
  • This is my damned website and I'm going to be as snarky as I want, Anonymous Coward from 13 hours ago.
  • All nine million of you who wrote to me are absolutely right: as a user of open-source software I have the right to fork it if they decide to pull out the v2 API. I'm not super-interested in becoming the de facto maintainer of a database, particularly not one in a language that don't use very often, but I do have that right and ability. Thank you for writing to me.
  • Non-sarcastic thanks to everyone who pointed out typos and issues in the original post. Fun fact: Layer 3 and Layer 4 in the OSI model are not the same thing.
  • If you agree with me that simplicity is a virtue in software architecture and we embrace too many things that look like 90's-era Microsoft APIs, how about really driving it home by giving some money or time to Black Lives Matter, the SPLC, the ACLU, or other organizations working to simplify and improve our civil lives3?
  • If you vehemently disagree with me and think I'm human scum for not embracing the glorious combination of systemd, kubernetes, and the Registry Hive, why don't you really pwn me by donating money or time to an organization whose website probably runs on some k8s cluster somewhere like Black Lives Matter, the SPLC, or the ACLU?

Many thanks to DreamHost for successfully seeing this static-HTML website through a big traffic spike. I've been a customer for like 15 years and so far so good.

1

I find that Hacker News is really good at attracting people who think technology is cool, which is a particularly dangerous thing for anyone who actually expects to work in the tech industry. Computers are awful. You need to really embrace the hatred before you can be an effective technologist.

2

Yes, dear commenters, I do actually use the tools I complain about

3

This metaphor isn't a stretch at all, why do you ask?

Etcd, or, why modern software makes me sad

etcd icon

Once upon a time in 2013, there was a tool called etcd which was a really lightweight database written around the Raft consensus algorithm. This tool was originally written in 2013 for a bullshit unsuccessful project called CoreOS Container Linux that was EOL'd several years ago, but that doesn't really matter --- etcd was greater than its original use-case. Etcd provided a convenient and simple set of primitives (set a key, get a key, set-only-if-unchanged, watch-for-changes) with a drop-dead simple HTTP API on top of them. I have built a number of tools using etcd as a lightweight consensus store behind them and it's absolutely a pleasure to work with.

Hello **massive influx of new readers**! I see that some person who's out to get me kind soul has cross-posted this to Hacker News, Reddit, and a bunch of other sites. Cool! A few things you might want to know _before_ you send me hate-mail:
  • The word "rant" is right up there in the tags line. This is not meant to be a persuasive argument to the secret cabal that controls API design or a nuanced technical comparison article. It's just some off-the-cuff thoughts. Chillax.
  • If this didn't come across clearly enough in the article: I think etcd is great! I have written a bunch of tools and applications on top of it! I think it's a fantastic little dæmon and its API, even the new janky v3 API, is still a million times better than ZooKeeper

Okay, then. Read on.

In 2015, an unrelated tool called Kubernetes was released by Google (but, really, by Xooglers). I would go so far as to say that Kubernetes (or, as the "cool kids" say, k8s) is the worst thing to happen to system administration since systemd. It's a comprehensive suite that promises to simplify operating clusters of software and give something like the experience of Google's borg cluster manager. What it really does is:

  1. Add hundreds of new failure modes to your software
  2. Move you from writing portable software configuration to writing thousands of lines of k8s-specific YAML
  3. Ensnare you in a mesh of questionably-good1 patterns like containerization and software defined networking

If you are running a truly enormous system and want to have off-the-shelf orchestration for it, Kubernetes may be the tool for you. For 99.9% of people out there, it's just an extra layer of complexity that adds almost nothing of value.

I digress, though; this is a story about etcd. And, unfortunately, our stories come together because Kubernetes was quickly changed to use etcd as its state store. Thus began the rapid decline of etcd.

With the massive influx of Kubernetes users came, of course, a large number of Xooglers who decided to infect etcd with Google technologies, as is their way23. Etcd's simple HTTP API was replaced by a "gRPC"4 version; the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys. etcd 3.2 added back a tiny subset of the HTTP API through the "gRPC Gateway", but not enough to implement any of the rich applications built on top of the original API. The v2 API lives on for now, but upstream threatens to remove it in every new version and there will surely come a time when it'll be removed entirely.

That's it. That's the story. Popular modern technology is taken over by expats from a megacorp and made worse in the service of a hyper-specialized (and just plain over-hyped) orchestration platform. That's the world today. Anything that has a simple and elegant feature-set ends up coöpted by people who just want to build big ungainly architecture and ends up inheriting features from whatever megacorp the coöpters came from5. The software development world would prefer to use their multi-gigabyte IDEs running on ElectronJS to build thousand-dependency Java applications targeting ungainly APIs on hard-to-operate systems than support something simpler and better. Quality is, alas, a dying art.

1

Read: "not good"

2

I've worked with a lot of Xooglers in my career (heck, I worked there myself). I now consider it to be a serious negative on someone's resume to have worked at Google. The many ex-Google coworkers I've had have (even when they've been otherwise brilliant) been uniformly less capable of working on non-Google systems than their much more junior equivalents with other backgrounds. All big companies have their own proprietary technology stacks, but the degree to which Googlers never learn how to do anything without involving protocol buffers, bigtable, and a mile-high stack of other proprietary tools is frankly remarkable. And they spread this to everything new they touch. Any open-source project will inevitably get pull requests to switch from JSON or BSON to Protocol Buffers; every web server now needs to support the cancerous user-hostile protocols of HTTP/26 and HTTP/37 that were passed in a mockery of IETF procedures.

3

Yes, I know, gRPC was added to etcd by Xiang Li, the original author of the project. That doesn't give them a pass for being influenced by the bad ideas coming out of Mountain View, or by their project's newfound popularity in the land of the Xooglers.

4

gRPC is Protocol Buffers8 running over HTTP/26. It's a got a "g" at the beginning of the name to remind you that the only time it's acceptable to use is when you are actually working for Google inside a Google-owned building eating Google-branded food and breathing Google-branded air.

8

Protocol Buffers or protobuf is Google's very bad serialization format

6

HTTP/2 a.k.a. SPDY is a comically bloated Layer 5/6/7 mega-combo protocol designed to replace HTTP. It takes something simple and (most importantly!) comprehensible and debuggable for junior programmers and replaces it with an insanely over-complicated9 system that requires tens of thousands of lines of code to implement the most minimal version of, but which slightly reduces page load time and server costs once you reach the point of doing millions of requests per second. I am filled with rage just thinking about how we took a fundamental part of the Internet, simple enough that anyone can implement an HTTP server, and replaced it with this garbage protocol pushed by big megacorps that doesn't solve any real problems but will completely cut out future generations from system programming for the web.

9

My favorite HTTP/2 interaction has been finding and reporting this bug in haproxy. The compression scheme in HTTP/2 is so shitty that the "compression table" in RFC 7541 § Aa is just a list of the 61 most popular headers from Google properties.

7

HTTP/3 is all the badness of HTTP/2, but run over a worse layer 4 protocol named QUIC that totally fucks up networking for everybody in order to get a tiny bit more optimization for Google. That's all it does. It makes the Internet strictly worse for everybody but slightly better for the hugest of huge web properties. Nobody out here in the real Internet gives the slightest shit about head-of-line blocking from TCP, and lots of people want TCP state-aware firewalls and load-balancers to work.

5

I talk a lot of shit about Google, but Facebook and Microsoft are nearly as bad at turning out legions of ex-employees who can't be left alone in the room with a keyboard lest they attempt to recreate their previous employer's technology stack, poorly.

Trackballs

It's been about ten years since I've regularly used that most quintessential of post-1984 computer peripherals, the mouse. The last mouse I had was pretty exclusively used for a gaming PC in college and was a Logitech MX5181; since then I've used a variety of input devices. It's been a while since I did any brief reviews of technology on this ostensibly-technology blog, so what the hey, let's do it! In brief, we're going to talk about:

My first taste of a wrist-pain-free input device was the Logitech TrackMan Wheel Optical, a long-discontinued thumb trackball made in the mid-00's. According to my Amazon.com order history, I purchased this item on September 2, 2007. If you've never used one before, the idea is that you hold this device much like you would a "normal" ergonomic computer mouse, but instead of moving your whole arm your just move your thumb to control the cursor. The TrackMan Wheel (either in this, its "optical" variant, or the older ball-guided version) was a common first introduction to trackballs in the 00's. The tracking was always smooth2, the wheel was refreshingly clicky, and the buttons never failed. This was a great mouse. Unfortunately, it got lost in the move-out from college in 2010, and by that time it was discontinued and they were already going for >$100 on eBay. 😢

I replaced it with the new-at-the-time Logitech M570, which is ostensibly just a Logitech-Unifying-Receiver3-equipped variant of the TrackMan Wheel Optical. A few key notes here:

  • Generally, wireless sucks for mice. Even the Logitech Unifying Receiver, which is generally less flaky than Bluetooth, is still flaky and has human-visible latency
  • I don't know what happened to the controls on this mouse, but the buttons are not nearly as satisfying as the original and the wheel has always felt gummy
  • Forward and back buttons are dumb on a mouse

I used the M570 for a few years because it worked well enough, but it's not a very good input peripheral. I still have it (you can see a picture below, next to a stock shot of a TrackMan Wheel Optical) because there are a couple of games I can only play with it (unless I want to take an unreasonable amount of time to retrain muscle memory, and who wants to do that?)...

In 2012 or so, I had a breakthrough: I purchased a Kensington ExpertMouse. I'd seen these for years4 but never gotten around to regularly using one. The ExpertMouse has a single giant ball that you operate with the tips of your fingers and buttons around the edges. At some point, this trackball gained a label proclaiming it to be "joystick" and it became my standard input device; I took it with me to Uber and rolled many miles on that trackball.

Kensington ExpertMouse marketing shot

Unfortunately, when I left Uber in 2015, the ExpertMouse got nabbed by someone else off my desk and was never seen again. Even worse, some time between 2012 and 2015, Kensington drastically worsened their manufacturing process. I purchased a new ExpertMouse, but the entire thing feels much cheaper. The scroll wheels don't spin cleanly any more, the buttons are sticky, and the ball has to be cleaned twice as much as it used to. There's also a new wireless one that I hear very bad things about. Pre-2013 wired ExpertMouses (ExpertMice?) can sometimes be found on eBay for $150+.

After this, I took a detour and spent some time doing my input with trackpads instead of trackballs. You can see an Apple Magic TrackPad 2 in my new computer post from last year, and I had an Apple Magic TrackPad 1 at work for a while. The Magic TrackPad 2, in particular, is just an absolutely enormous (6¼" x 4½") slab of glass that serves as a multi-touch input for your computer. I still use mine and try to alternate every couple of weeks between a trackpad and a trackball to change up the kinds of stress on my wrists. If you've used a trackpad on any "modern" (unibody) MacBook, you know what a Magic TrackPad feels like. It's great for some things (like horizontal scrolling) and not great for other things (highly precise input; gaming).

But what about trackballs? Isn't that the purpose of this post?

For the last year or so I've been rocking two different Elecom trackballs: an Elecom DEFT PRO at home and an Elecom DEFT HUGE at work. Just today, I went into the office and brought my HUGE home, so I've got them side-by-side on my desk5:

Elecom DEFT Pro and DEFT Huge

Conceptually, they're very similar trackballs: a single large ball that you operate with your pointer and middle fingers, then a scroll wheel and buttons that you operate with your thumb and ring fingers. The HUGE is, well, HUGEr, with a ~25% bigger ball, a couple more buttons, and a built-in wrist-rest. The models I have are a wired-only HUGE (M-HT1UR) and a wired-or-wireless PRO (M-DPT1MR). I initially used the PRO wireless over Bluetooth to my desktop, but after one too many dropouts, I switched it to running wired over the micro USB cable. Unlike the Apple Magic TrackPad, when you plug in the Elecom, it does actually behave as a standard wired mouse instead of just charging.

I only have two complaints about the Elecom trackballs:

  • It's very hard to hold the wheel down. This is not a common operation unless you play a video game that uses holding the wheel down as an important action...
  • When wired, do not use standard USB HID classes so cannot be used on macOS for anything besides tracking and LMB/RMB without third-party drivers. SteerMouse seems to be the only option that supports all of HUGE's buttons on Catalina6. It will be a sad day if they ever drop support. Notably, BetterTouchTool, which I use for other input customization, does not support any of the extra buttons on the HUGE.

Anyhow, that's my brief run through input devices. If all of my mouse-like devices were destroyed in a mysterious laser strike, I would go out and buy another Elecom DEFT HUGE. For now. As I've found the hard way with both Logitech and Kensington, everything seems to get worse with time. Maybe I should just buy a few extra Elecom devices now now and put them away in an airtight box somewhere? No, that's probably crazy. I'm not Jon Siracusa...

For those of you who've made it this far, here's what my desk looks like today:

my desk, today

Stay safe out there, readers.

1

I bought mine in 2005. Remarkably, this is a product that Logitech still makes and sells fifteen years later for sixty American dollars. Capitalism!

2

At least, until gunk got stuck between the wheel and the Teflon nubbins it rolls on, at which point you flip the trackball over, push the ball out, and clean gunk off the nubbins. This is a requirement for all trackballs unless you wash your hands before every time you use your computer, and probably also have yourself surgically altered to no longer sweat.

3

The "Logitech Unifying Receiver" is a proprietary 2.4GHz radio dongle for Logitech keyboards and mice.

4

In fact, the ExpertMouse was one of the "standard" mice we stocked at the Tech Stop when I worked in Google Corporate IT.

5

Yes, these look gross. Hey, it's hard to use an input device for a protracted period without it looking gross. Just ignore the crumbs and stains.

6

I have Fn1 mapped to middle click, Fn2 mapped to show desktop, and Fn3 mapped to Mission Control née Exposé

2<sup>5</sup>

As every company I've ever bought a product from has been reminding me all morning, today is my birthday. 32 revolutions around the sun. Still ticking.

Obviously it's been a pretty busy year; most of that is in my post from last month. According to the accelerometer on my wrist, I've walked 2,904,398 steps in the last year1 but I'm mostly still in the same place as I was. Probably by the time I hit 33 revolutions around the sun, it'll be a very different story. Still got that baby on the way (another ~6 weeks to go), and my company just announced that we're going permanently remote so I guess that'll be a big life change.

Maybe I'll actually post more this coming year? Who knows!

1

Probably about 300,000 fewer than I would've without COVID-19.

2020 MacBook Air Impressions

For the past few years, I've been using a one-port 2016 MacBook1 as my personal laptop that I carry with me whenever I am traveling or need to work outside the house. It's been a mixed bag -- the size was pretty much perfect and very similar to a thin version of my old 12" PowerBook G4, but the CPU performance was absolutely awful and the battery only lasted about two hours. Also, I was finally bitten by the infamous Butterfly Keyboard Issue and was getting double-entries from my spacebar.

In March, when Apple released the new MacBook Air with a new keyboard and one of the first 10nm Intel CPUs2 on the market, I decided it was time to upgrade, so I bought the middle stock configuration of the new machine3.

2020 MacBook Air

Basic specifications:

Component
CPUIntel Core i5-1030NG7 (1.1GHz base, 3.5GHz turbo, 10nm process, Ice Lake μarch)
RAM8GB4 LPDDR4-3733
Storage500GB Apple AP0512N SSD
Display13.3" 10-bit Retina display @ 2560x1600 (221dpi)

I haven't had it for long, but here are some initial reactions:

  • size isn't as nice as the 12" (37% heavier, and about 20% more volume)
  • new keyboard is more reliable and has a pretty good feel. backlighting is also much stronger.
  • battery life is way better. 5-6 hours of moderate use, and several days of standby
  • screen is a bit brighter and looks less goofy with night shift on. still not nearly as good as my 2019 iMac.
  • CPU performance is nominally a lot better5 and feels less like the limiting factor in day-to-day operations. It still chugs a lot when trying to do medium-intensive things like processing 24MP RAW images from my Ricoh, but at least it finishes eventually now instead of beachballing until I die of old age.
  • integrated GPU is massively observably more powerful than on the 12" (GeekBench says the "compute" score is only 50% higher, but it feels much smoother when doing basic rendering operations like Exposé)
  • the enormous trackpad is really fun. My work MacBook Pro has the same trackpad, but I basically never use it undocked, so this machine is my first time intensively using the giant trackpad on modern 13"-class Macs. I'm pro.

There's a reason that the MacBook Air is Apple's best-selling Mac. If you need a portable Mac, it's probably what you should get. No TouchBar, acceptable price, and now a functional keyboard. Officially endorsed.

PS: I wrote this entire post on the new machine and didn't get any inadvertent double keystrokes at all. Hurray!

1

A.K.A. the MacBook One

2

Only 3 years late. Some day some former Intel executive will write a tell-all about why Intel fell so far behind TSMC in lithography

3

Yes, I got "Space Grey" this time, for the first time in a laptop. It's okay. I think I'll go back to regular old silver next time.

4

I strongly considered getting 16GB, but I mostly just run nvim and firefox. How much RAM could that use?

5

GeekBench says it's more than twice the single-core performance and almost four times the multi-core performance

Flexbox Rocks

Small update: I rewrote the CSS for this website to replace all of the Bootstrap Grid system stuff with straight up Flexbox. It's so much better! We should all use Flexbox for everything!

The "gallery" view (used in, e.g., the end of yesterday's "Life Updates" post) is also now Flexbox and better.

I tested this on a bunch of browsers and screen sizes, but please let me know if you find anything that looks broken.

I also got bored and made the tags page a tag cloud on wide-enough screens because it's like the 90's again.

Trees in the Park

park

There are trees growing out out of the sand at the park near my house, closed for the last few months due to COVID-19. What a weird world.

Life Updates

Hey, so, it's been a while since I posted. I guess I just wanted to post and update. Let's see... what has happened since February 2020....

Family

fetus

Yeah, so, we're having a baby. If you didn't see that on other social media already. He's1 due at the end of July2.

Lockdown

As you and everyone else knows, the world has been in lockdown since March due to the outbreak of SARS-CoV-2 causing COVID-19. My company sent us all on mandatory work-from-home on March 12th and I haven't been across the bridge into San Francisco since. Working from home is... tough. We have a house (unlike many of my workers who have apartments or even just single bedrooms), but it's still hard to get enough space that you can clearly delineate "work time" from "home time". Some parts are nice; I like being able to eat lunch in my backyard with Eva, and I like the fact that I haven't had a cold in two and a half months due to the lack of BART and the constant hand-washing. It's damned helpful that we bought a car when we did, both because public transit is basically gone now, and it's also really hard to buy a car right now.

The baby stuff is also incredibly stressful -- everything about preparing for a new child is a million times harder when the world is locked down. You can't go to any stores to buy clothing or supplies; it all has to be blind over the Internet. I can't go to my wife's OB appointments due to the strict contact procedures. And, of course, they still have no idea how this disease affects pregnant women or newborn children, so we're trying even harder than normal not to get infected.

I can't really write anything about this disease or lockdown that isn't said better by a million other people online, but my sympathies go out to all of those infected and affected. I wish that we had literally any other political leadership at this time, and I hope that we make it through anyway.

Rabbits

We had some sad news recently; one of our rabbits (Alot) was diagnosed with cancer in February and had her ear removed. Unfortunately, she developed a cough and then ended up going into severe stasis and passing away on May 24th. She was a good, brave rabbit and will be missed; she is survived by her bonded partner rabbit Mirabelle.

alot of rabbit

Hobbies

In preparation for having a new (and hopefully photogenic) baby and to entertain myself while locked in the house all day, I did buy a new camera recently. I decided that part of the reason I don't use my OM-D E-M5 enough is that it's pretty big and heavy with a lens on3 and I don't have it with me enough. As they say, the best camera is the one you have with you4. Anyhow, I got a Ricoh GR III after reading several hundred reviews of every high-quality pocketable camera, and for the few days I've had it, it's been absolutely fantastic. An APS-C sensors in a 250 gram package is completely insane, and I'd forgotten how much more light you get with that big of a sensor compared to the Micro Four Thirds sensor on my OM-D or the teensy sensor on my iPhone5. The Positive Film setting on the Ricoh in particular produces a delightful æsthetic. It also produces phenomenal RAW images which someone with time and skill could surely extract a ton out of; I have neither the time nor the skill so I mostly shoot JPEGs.

Stay safe out there, gentle readers.

1

Yes, it's a boy. The sonogram helpfully had arrows to point out where you can tell that it's a boy.

2

He'll have a good head start on all the lockdown babies getting born in January.

3

With my smallest non-potato quality lens, a 17mm prime, the OM-D weighs 588 grams and requires a rectangular prism of 8.57 deciliters. The new camera is 257 grams and 2.16 deciliters.

4

I've heard that maxim for at least 10 years but I cannot find out who said it first. Chase Jarvis wrote a book with that title, but the quote shows up in random places in Google from years before the book was written.

5

APS-C is 368 mm2; m43 is 225 mm2; the iPhone's 1/2.55" sensor is only 25 mm2.

systemd

I've had this sitting on my desktop for months and figured I'd post it. No idea where it originally came from.

systemd