roguelazer's website

It's been five years since the pandemic hit, and three years since the last time COVID hit our family, but it's hit for real now. My five year perfect-streak has been ended¹.

My daughter came home from day-care sick on Monday and I started feeling bit crummy late Thursday afternoon. Lo and behold, not only am I positive for COVID, but so is everyone else in the family. The kids are mostly okay — the baby had a fever on Tuesday but has been fine since then, and my son has no symptoms — but my wife and I both have fevers and aches and fatigue and coughs and the whole nine yards. Let me tell you, having two healthy and rambunctious children home from school while the adults have COVID is the absolute worst.

Anyhow, I got a Paxlovid prescription. So far (< 12 hours in) it's not doing much for the symptoms, but everything tastes like rusty nails now. Yay.

Stay safe out there, nerds.

I don't have time for a lot of hobbies these days, but one of my favorites is preparing food. Whether that's cooking four courses out of Julia Child¹, grilling some meat, or just tossing a salad together, there's something deeply satisfying at turning ingredients into a meal. In some ways, it's not unlike programming; anyone can do a passable job using inferior tools and following other peoples' recipes, but there's almost infinite space to tinker, invest in better tools, and learn to make your own recipes. Also, as long as you follow some simple rules, the worst outcome you're likely to have is something that doesn't taste great.

More than that, though, it's something that's just fundamentally human. Despite what some of my Soylent-chugging former executive leaders might thing, making and eating food is an activity that connects us all. The forces of capitalism are trying as hard as they can to give us a world where everybody eats soulless chain-restaurant slop delivered by underpaid Gig Economy workers, but you can still make something all your own; you can still find joy in something that has no product-market fit, no CSAT, and nothing to optimize².

Some good things

I got a new knife recently (a 200mm Senzo bunka knife). Does it make the food taste better than if I'd cut it with my 15-year-old chef's knife? No, but it makes the act of prepping food a bit more fun. It's wildly too fancy for my skill level, but ¯\(ツ)/¯.

I also replaced my 15-year-old All-Clad stainless steel frying pan with a new 11" Demeyere frying pan³. The old one's rivets were starting to come apart, so I got one without rivets!

The original Moosewood Cookbook continuous to be joyous; any recipe you cook is sure to include some novel ingredients and techniques.

Some good fresh spices can make all the difference to a simple dish — some tellicherry peppercorns and Calabrian chile powder (and oil and salt, of course) is really all you need for some quick sautéed or grilled vegetables.

The Temescal farmer's market is a great place to blow $50 on berries that your 5-year-old will eat all of on the walk home. And lots of other good produce, too. Oh! And the Garden Variety Cheese Hollyhock is so good.

Some bad things

Fuck Samsung appliances. Our house came with two of them: a range and a fridge. They're both stupid.

The range (which doesn't have a model number on it, but looks like the NX60A6511SS/AA) has capacitive touch buttons on top of it. Which are triggered by steam. Which means that if I boil something on the stovetop, it will turn on random settings on the oven. It's infuriating to be making a sauce and suddenly have the oven turn on at 700°F for no reason. Also, if they're damp, then don't respond to finger touches, which means that you then can't turn the oven off without getting a towel and drying the buttons. Such a fucking stupid design.

The fridge is a four-door model with a "Flex Zone", which means you can convert half of the freezer from a freezer into more fridge space. I have no idea why you'd want this (it leaves you with an absolutely minuscule freezer), but it's mapped to a single touch of a capacitive button, which means once in a while someone will brush by it and not notice and then half of your freezer will melt. There's a "control lock" feature, but it also completely disables the ice maker⁴ and water dispenser, so it's not really usable.

Outside the realm of Samsung appliance crap? It's become impossible to find recipes or blog posts about cooking on the Internet that haven't become LLM-slop-filled-nonsense. Thankfully, there are hundreds of years of writing about food that predate Truthiness as a Service.

Auf Wiedersehen and good night

Last week, I wrote bout my experience getting a solar power system installed; since then, I've been spending some weekend and evening time working on improving the little corner of technical nonsense that I have any control over!

Background

First, let's do a quick recap of how my system works:

I have 20 panels, each of which generates up to 460W of DC power and immediately converts it from DC to AC using an Enphase IQ8X microinverter¹. This gives us a peak of 9.2kW possible power split across two phases.

This power comes off the roof to an Enphase IQ Combiner 5 PV combiner. I'm not really sure what the combiner does! I assume it somehow aligns and balances the different AC signals coming off the roof into two 60Hz AC phases, but I have no idea how! Maybe it's just an expensive subpanel with a busbar and some breakers, who knows.

Anyhow, on the other part of the wall we have three Enphase 5P Batteries, storing 5 kWh each (as DC, obviously, since you can't store AC). Each of them has six built-in IQ8D microinverters capable of putting out about 600W, so each battery can provide about 3.6 kW of instantaneous power.

The solar and the batteries both land on an Enphase IQ System Controller 3M, which merges all that power together and directs charge/discharge of the batteries. Unlike older Enphase systems, all of the communication is over powerline Ethernet instead of Zigbee.

The System Controller connects to the main panel in two ways: first, there's a 100A backfeed circuit to power the house with up to 19.2 kW of power; second, the System Controller is connected to the meter with an IQ Meter Collar , which allows the system to operate safely during an outage to provide whole-home backup without energizing PG&E's grid.

Management, Officially

So, how do you see what the system is doing and control it? It's all done through an Enphase app called Enlighten, available both as a webapp and a mobile app. The mobile app is mostly just a wrapper around the webapp; there's almost no native functionality. Both of these tools are incredibly slow. I just benchmarked launching the iOS app on my phone² and it takes 13-21 seconds to start up. Once it starts up, it's not responsive to taps at all and many features take forever to load. It also doesn't support widgets or App Intents or anything else someone might want on a modern Apple platform. The webapp usually loads faster, but has a "do not sell my personal information" banner that can't be dismissed and the layout is broken at lots of screen sizes (including the one in this screenshot).

I don't really need my data hosted in some cloud service anyway. Isn't there a way to get this information directly off the Enphase system gateway (which is known as Envoy)?

Yes, Duh

This would be a short story if there weren't.

It turns out that Enphase provides a documented open API to get data off of the Envoy! The documentation is pretty hard to find, but it's still around at https://enphase.com/download/iq-gateway-access-using-local-apis-or-local-ui-token-based-authentication-tech-brief.

You need to obtain a JWT from some dinky web service (which expires after a year and can't be renewed without an interactive session, alas), but then you can get all kinds of nice JSON data. Sometimes it's oddly slow (like, 15-20 seconds), and there's no authz, so that same token that can read the data can also reconfigure the system, so you definitely don't want to expose the Envoy directly to the Internet.

So, I wrote a small Rust application named envoyproxy³ that polls the Envoy API every minute and serves the most recent result and some other metrics as JSON and Prometheus. I'm running mine in my little k8s cluster here, using the excellent Tailscale Kubernetes operator to expose it to my Tailnet so I can reach it from anywhere.

But why?

The immediate goal here was to have a pleasant iOS experience for seeing system status, so I've also built a SwiftUI app, tentatively codenamed Sunny Days:

It's pretty ugly, but it takes 14MiB of RAM and launches in approximately 300ms. Not too bad for something cracked up on a weekend by someone with absolutely no Swift experience.

I'm not sure what to do with it. It'd never make it onto the App Store — a third-party unsupported client for a proprietary solar system which requires running an open-source server somewhere? Maybe it'll just live on my and my wife's devices through TestFlight forever...

I also, of course, have a Prometheus instance ingesting these metrics, connected to Grafana.

Anything else?

Yeah, the other thing is my car. It's connected to an Emporia EVSE on a fat 60A circuit. However, most of the time I'd prefer to charge it slowly with excess solar. This is... not supported⁴.

There are really two options:

1. Pull out the Emporia EVSE and put in Enphase's, which can talk to the CTs in the Enphase boxes and natively do excess-solar charging. This would require permit and an electrician and a new EVSE and would probably come to around $1000
2. Add the Emporia Vue CTs to the solar subpanel and main panel. This would require two sets (around $300) and should be done by a licensed electrician, so still pretty steep

I didn't want to do that, so instead I wrote a small Python program that reads the Envoy data every few minutes and uses it to adjust the Emporia charge rate up-or-down. It's not as nice, because it's not instantaneous so sometimes it'll draw amps from the batteries or the grid for minute, but that's not a big deal to me, and the cost was $0. It's called emporia-enphase-control and is... pretty rough around the edges⁵. It does work, though! It's running in Kubernetes, right alongside envoyproxy.

That's it, right?

For now. We're still adapting to life with solar; it's a surprising number of changes. For example, we usually ran the dishwasher and the dryer at night (when the grid is the cheapest), but now that power is totally free while the sun's up (the batteries are quite finite, especially when the A/C is running overnight on hot weeks like this one) it makes no sense to run these expensive appliances⁶ at night.

So far, we have only drawn a negligible amount of power from the grid this week⁷.

Did you know that the sun is a mass of incandescent gas, a gigantic nuclear furnace? A source of a constant stream of high-energy photons?

Did you know that PG&E is a gigantic racket with some of the highest rates in the state? We pay 61¢/kWh for electricity during peak times¹, about seven times as much as my wife paid at her old Philly apartment — given the US Energy Information Administration's estimate of 10.5MWh/household/year, that's between $3600 and $6400/year of electricity.

Suffice to say, we've been planning on putting solar panels on the roof since we bought this house. Thankfully, solar technology is amazing. Panels have gotten almost 40 times cheaper (accounting for inflation) in my lifetime! Emissions-free! Long-lived! Quite possibly the most remarkable technological development in my lifetime².

That being said, we had to do a ton of work on this house and it was my hope to hold off on solar for a year or two, reap some benefit of those plummeting prices. Then the country elected Orange Hitler for an improbable second term, and I took him at full faith that he was going to do everything in his power to kill the burgeoning solar industry as a favor to his friends in the earth-murdering industry, so I pushed as hard as I could to get the solar panels on the roof before it was too late.

I started by reaching out to other people who I know who've gotten solar panels installed, and their answers all came down to, We love our panels but our installers were meh so you should look elsewhere. While I did a crash-course on residential solar, I set up an account on EnergySage, which is sort of industry-sponsored aggregator for solar installers. I ended up getting quotes from seven companies through EnergySage, plus three others that I found elsewhere. There were a few dimensions to consider:

• Panels: most of the installers were quoting REC Alpha Pure 2 or REC Alpha Pure-RX panels in between 420W and 480W per panel; a couple were the Hyperion/Runergy HY-DH108P8 series at between 400W and 415W per panel. I'm really not qualified to evaluate the chemistry or physics of the panels, but bigger number = better, and better warranty = better; both of those favored the REC panels
• Inverters: the main dimension is string inverters versus microinverters; basically, string inverters take DC power off the roof and invert it to AC with big inverters at ground level; this is theoretically more efficient but makes it much more complicated to handle unbalanced power (e.g., when shaded); whereas microinverters invert the DC to AC in each panel on the roof and then take the (in-phase) AC off the roof and combine it with cheapish AC combiners at ground level. I can see the advantages of both approaches, and there are an infinite number of infotainment-quality articles arguing in favor of one versus another. I did talk to some electrical engineers who were basically all in favor of minimizing the number of AC/DC conversions.
• Batteries: Under NEM3, you basically have to have batteries, since PG&E will only pay you about 25% of the standard rate for electricity you sell back, so it's no longer cost-effective to overproduce during the day and then consume from the grid at night. The Tesla Powerwall is the 3000-lb gorilla in the room and dominates this market, but there are a few other players³.

Practically, for an integrated system with battery storage in California on short notice, there are two options:

1. Tesla Powerwall + SolarEdge string inverters
2. Enphase microinverters and batteries

I also got one very intriguing quote based on the Lunar Energy system, but we'd be one of the first customers in PG&E and I didn't really want to be a beta tester⁴.

Anyhow, after evaluating all the quotes and reading every BBB and Yelp review I could find, I decided to go with a fairly large company based in Southern California named Sunergy. I didn't (and still don't) want to give Tesla/Elon Musk any money, which meant going with Enphase, and Sunergy was both the best Enphase quote and one of two that was "Enphase Platinum" certified, whatever that means. I signed a contract with them on December 14th, 2024, and the system design was completed on December 19th. Then, we waited.

It took the City of Oakland about 6 weeks to review and approve the permit, but on March 3rd we were approved to proceed. In the meantime, I'd been busy. Installation was scheduled for March 31st. Come March 30th, installation was cancelled due to supply chain issues⁵ and pushed back to April 3rd. Then April 10th. Finally, on April 16th, the panels showed up and were put on the roof — only a few days before Cheeto Mussolini slapped a so-bad-it's-almost-funny 3,521% tariff on solar panels and nuked the entire industry. Not only had I gotten the panels installed, but I was in a beta program⁶ to have the batteries set up in "whole-home backup" mode using the brand-spanking-new Enphase IQ Meter Collar as a transfer switch.⁷

This whole experience so far is pretty frustrating; like basically all home-improvement contracting projects in the US, the homeowner ends up serving as de facto project manager and every site visit involves hours of coordinating between various sub-contractors and laborers. But at this point, when I was ready to declare victory in April, I hadn't even gotten to the worst of it. Let's recap the process of getting solar on your roof:

1. You pay a buttload of money
2. Someone puts panels on your roof that can't do anything
3. Someone with an electrician's license wires them into your panel so they can theoretically do something
4. The city inspects the panels and the wiring to make sure they're safe
5. You pay another buttload of money
6. The utility gives you "Permission to Operate"
7. You can turn your panels on

Do you see the weak link in all of this? It's step 6, where your utility (in my case, PG&E) has to give you "Permission to Operate" or "PTO". There's no particular timeframe in which they must do so, and it's a completely opaque process where your installer files some paperwork and you... wait. It took the city 14 days to inspect and approve the install, and then it took PG&E an additional 63 days to give me PTO, during which time I was legally forbidden from turning the panels on and had to keep paying PG&E out the nose. Having a company whose profits depend on not giving you PTO get to decide at their own leisure when you can turn on your panels seems obviously bad.

But wait, there's more! Remember how I was in that beta program for the IQ Meter Collar? It turns out that meter collars have to be installed by a PG&E electrician. In addition to being a beta for Enphase, this is also still a beta at PG&E (run out of the "SNEM Paired Storage" group), and all of this... beta-ness... is expected to be handled by the installer.

Unfortunately, while I was staring at some photovoltaic depreciating assets and sitting on my hands, Sunergy appears to have been imploding⁸. I got fobbed off to a series of account managers (all of whom ended up leaving the company) and at this point the only contact I still have is their long-suffering founder Chris Hammerstone, whose phone is literally always busy. Eventually I realized that the only way to get attention was to go make a fuss with Enphase, which I did, and then Enphase and Chris somehow joined forces and got PG&E to come plug in the meter collar. That was... last week, August 8th. The final (post-meter-collar-install) setup bits were done this week and the system has been online since Thursday August 14th. Exactly eight months between contract signing and system completion. Eesh!

Anyhow, the power's on. Some takeaways:

• These panels can produce 40 kWh on a sunny summer day⁹ and 25 kWh on a cloudy day
• I have 15 kWh of batteries and I can already tell it isn't going to be enough on winter days
• My house uses like 400W just sitting there empty. I guess that's the fridge¹⁰, the NAS¹¹, and the Ubiquiti gear¹²? Seems high! I bet I could get that down by 50% if I went and put a meter on every power brick in the house.
• Phases A and B are also way out of balance; I think all of the big single-pole appliances I have are on the same phase. Oops.
• Electric dryers are brutal; we (unthinkingly) ran the dryer last night and burned something like 8kWh of battery on two loads. Once this dryer dies (it's a 10-year-old LG, how long could it have), I'm definitely getting a heat pump dryer, since they're supposedly around 80% less energy per load.
• The Enphase iOS app (Enlighten) really sucks¹³. I'm making my own and will be posting about that hopefully in a couple of days.

Will any of these companies be around to support these products in a few years? Are we all going to end up in a wasteland playing a double-necked flamethrower/guitar on the front of a semi truck for the amusement of our god-kings? Who knows!

This site has had both "Categories" and "Tags" since it was on Blogger in 2007. It occurs to me that this doesn't really make any sense, so I'm hiding the categories and won't be populating them going forward. The old category index pages are live (because cool URIs don't change) but otherwise, it's all tags from here on out.

A few other small changes:

• updated some CSS
• fixed missing </li> in page metadata block
• chnged post time to be a <time> so it's machine-parsable, and stopped displaying post time by default
• added signal and hackernews to the Social sidebar, and removed two defunct sites
• archive now shows tag information

Yesterday, I left my job at Instrumentl, which I'd been at since leaving easypost almost three years ago. It was an interesting experience; I don't know that I'd choose to do it again, but it's always good to broaden one's horizons and try new things.

I was only at Instrumentl for about 990 calendar days¹, but I like to think that I got a lot done in that time. When I started, the dev team was only six folks (two people in the US, one in Canada, and three in Türkiye); now it's something like 18². When I started, MFA was disabled in every tool and lots of people were logging into shared administrator accounts; now the company is on much more solid footing and perhaps even on a path to finish a SOC2 in a year or so. I built a few customer-facing features (including the public API, SSO and MFA) from scratch, and a ton of internal tools. I was only there a couple of years, but (as usual) I found myself on top of the all-time productivity leaderboards by whatever metric you care to measure³. Not that these things matter; nobody cares who wrote the most commits or who fixed the most security issues, and anybody who thinks you can objectively measure engineering performance is trying to sell you something that isn't worth buying.

Instrumentl was the first company I've worked at where I was primarily doing Ruby, which was kind of neat. Ruby was the first programming language I really learned, about 20 years ago⁴, and maybe the first programming⁵ job I had was working on HMC CS Staff porting the internally-developed ticket tracking tool⁶ from Rails 1 to Rails 2. I did a fair amount of Ruby/Rails at Easypost, of course, but Instrumentl was all Rails. It's... definitely a lot nicer than it was in 2008! I still think it's practically-perlish in how hard it is to read, though. I'm more-convinced than ever that implicit receivers (that is to say, being able to just type foo instead of self.foo() to call an instance method named foo) is a nightmare for reliability and code maintenance.

Anyhow, some of why I've left comes down to differences over the direction of the company, some because I simply can't stomach reviewing one more bad PR or bloated document that someone clearly let ChatGPT loose on without thinking about the consequences, and because I'm ready to work on something new and exciting. I've got a couple of days before I start at the new place, which is going to be Svix, writing Rust full-time. Let me know if you're reading this in real-time and are bored next week and want to hang out. :-)

I've been using the Happy Hacking Keyboard in various configurations for the last 12 years; it's a great design for a 60% scale keyboard with all the keys that a Unix person needs and nothing they don't; in particular, I credit its lack of arrow keys with finally getting me to use vim motions correctly. Oh, I've dallied with other keyboards; for a while at work I had an Apple Wireless Keyboard, and in 2019 I bought a tenkeyless keyboard from the now-defunct WASD Keyboards¹, but I always come back to the HHKB. That being said, the HHKB isn't perfect; in particular, the Topre switches feel a bit mushy² (and get worse the longer you use them), the bluetooth version is garbage that can't maintain a connection reliably, and the cheaply-printed text on keycaps tends to rub off pretty quickly.

Well, I'm on parental leave now, so I figured I'd do a Project and put together a new keyboard.

Ingredients

• Keychron Q60 Max Barebones: This is a barebones keyboard with the exact same layout as an HHKB
• Kalih White Owl Box Switches: I played with a few switches in key testers and I loved the tactility of these. Reviews are good for the Box line as being more stable than other clicky switches.
• Yuzu Custom Keycaps

This was the first time I'd actually assembled a keyboard (I know, it seems like something that I would be doing regularly); it was pretty easy³. A couple of the switches had bent pins that I couldn't quite straighten out, but I bought plenty of spares. I'm typing on the keyboard right now! I think it's pretty great so far, although obviously I have only been typing with it for a little while. One interesting note is that it's remarkably heavy — around 1500g fully-assembled (about 3x the HHKB). Not that I carry it around much, but I think I'll keep taking the HHKB with me when I go to company offsites.

Now all I need is for Apple to get around to releasing a stand-alone TouchID button...

Important follow-up to A Son!

Two kids and a single-family-detached house; oh no am I a yuppie?

It's January 2025, which means it's time to reflect a little about 2024. Last year, I wrote about some video games I liked in 2023, and I thought that for this year, I might expand and just talk about a few things I liked. After all, the world is currently a burning hellscape looking forward to a fascist overthrow of government and a series of apocalyptic wars on a planet no longer suitable to human life and we don't even have the best president in the last 50 years around to help out any more — seems like a perfect time for some consumerist escapism.

read more

So, I haven't posted here in a while (excepting the tiny CSS post this morning, and for fixing jsSnow earlier today). I promise: I have good reason. It's been a mighty busy few months!

read more