Welcome to James Brown's blagoblag! This contains various thoughts and opinions, mostly wrong, going back a couple of decades. All of the opinions are my own, and probably not my employer's. Feel free to visit the about page for more useless interesting facts about me.

Here comes the sun!

Did you know that the sun is a mass of incandescent gas, a gigantic nuclear furnace? A source of a constant stream of high-energy photons?

Did you know that PG&E is a gigantic racket with some of the highest rates in the state? We pay 61¢/kWh for electricity during peak times1, about seven times as much as my wife paid at her old Philly apartment — given the US Energy Information Administration's estimate of 10.5MWh/household/year, that's between $3600 and $6400/year of electricity.

Suffice to say, we've been planning on putting solar panels on the roof since we bought this house. Thankfully, solar technology is amazing. Panels have gotten almost 40 times cheaper (accounting for inflation) in my lifetime! Emissions-free! Long-lived! Quite possibly the most remarkable technological development in my lifetime2.

That being said, we had to do a ton of work on this house and it was my hope to hold off on solar for a year or two, reap some benefit of those plummeting prices. Then the country elected Orange Hitler for an improbable second term, and I took him at full faith that he was going to do everything in his power to kill the burgeoning solar industry as a favor to his friends in the earth-murdering industry, so I pushed as hard as I could to get the solar panels on the roof before it was too late.

I started by reaching out to other people who I know who've gotten solar panels installed, and their answers all came down to, We love our panels but our installers were meh so you should look elsewhere. While I did a crash-course on residential solar, I set up an account on EnergySage, which is sort of industry-sponsored aggregator for solar installers. I ended up getting quotes from seven companies through EnergySage, plus three others that I found elsewhere. There were a few dimensions to consider:

  • Panels: most of the installers were quoting REC Alpha Pure 2 or REC Alpha Pure-RX panels in between 420W and 480W per panel; a couple were the Hyperion/Runergy HY-DH108P8 series at between 400W and 415W per panel. I'm really not qualified to evaluate the chemistry or physics of the panels, but bigger number = better, and better warranty = better; both of those favored the REC panels
  • Inverters: the main dimension is string inverters versus microinverters; basically, string inverters take DC power off the roof and invert it to AC with big inverters at ground level; this is theoretically more efficient but makes it much more complicated to handle unbalanced power (e.g., when shaded); whereas microinverters invert the DC to AC in each panel on the roof and then take the (in-phase) AC off the roof and combine it with cheapish AC combiners at ground level. I can see the advantages of both approaches, and there are an infinite number of infotainment-quality articles arguing in favor of one versus another. I did talk to some electrical engineers who were basically all in favor of minimizing the number of AC/DC conversions.
  • Batteries: Under NEM3, you basically have to have batteries, since PG&E will only pay you about 25% of the standard rate for electricity you sell back, so it's no longer cost-effective to overproduce during the day and then consume from the grid at night. The Tesla Powerwall is the 3000-lb gorilla in the room and dominates this market, but there are a few other players3.

Practically, for an integrated system with battery storage in California on short notice, there are two options:

  1. Tesla Powerwall + SolarEdge string inverters
  2. Enphase microinverters and batteries

I also got one very intriguing quote based on the Lunar Energy system, but we'd be one of the first customers in PG&E and I didn't really want to be a beta tester4.

Anyhow, after evaluating all the quotes and reading every BBB and Yelp review I could find, I decided to go with a fairly large company based in Southern California named Sunergy. I didn't (and still don't) want to give Tesla/Elon Musk any money, which meant going with Enphase, and Sunergy was both the best Enphase quote and one of two that was "Enphase Platinum" certified, whatever that means. I signed a contract with them on December 14th, 2024, and the system design was completed on December 19th. Then, we waited.

It took the City of Oakland about 6 weeks to review and approve the permit, but on March 3rd we were approved to proceed. In the meantime, I'd been busy. Installation was scheduled for March 31st. Come March 30th, installation was cancelled due to supply chain issues5 and pushed back to April 3rd. Then April 10th. Finally, on April 16th, the panels showed up and were put on the roof — only a few days before Cheeto Mussolini slapped a so-bad-it's-almost-funny 3,521% tariff on solar panels and nuked the entire industry. Not only had I gotten the panels installed, but I was in a beta program6 to have the batteries set up in "whole-home backup" mode using the brand-spanking-new Enphase IQ Meter Collar as a transfer switch.7

This whole experience so far is pretty frustrating; like basically all home-improvement contracting projects in the US, the homeowner ends up serving as de facto project manager and every site visit involves hours of coordinating between various sub-contractors and laborers. But at this point, when I was ready to declare victory in April, I hadn't even gotten to the worst of it. Let's recap the process of getting solar on your roof:

  1. You pay a buttload of money
  2. Someone puts panels on your roof that can't do anything
  3. Someone with an electrician's license wires them into your panel so they can theoretically do something
  4. The city inspects the panels and the wiring to make sure they're safe
  5. You pay another buttload of money
  6. The utility gives you "Permission to Operate"
  7. You can turn your panels on

Do you see the weak link in all of this? It's step 6, where your utility (in my case, PG&E) has to give you "Permission to Operate" or "PTO". There's no particular timeframe in which they must do so, and it's a completely opaque process where your installer files some paperwork and you... wait. It took the city 14 days to inspect and approve the install, and then it took PG&E an additional 63 days to give me PTO, during which time I was legally forbidden from turning the panels on and had to keep paying PG&E out the nose. Having a company whose profits depend on not giving you PTO get to decide at their own leisure when you can turn on your panels seems obviously bad.

But wait, there's more! Remember how I was in that beta program for the IQ Meter Collar? It turns out that meter collars have to be installed by a PG&E electrician. In addition to being a beta for Enphase, this is also still a beta at PG&E (run out of the "SNEM Paired Storage" group), and all of this... beta-ness... is expected to be handled by the installer.

Unfortunately, while I was staring at some photovoltaic depreciating assets and sitting on my hands, Sunergy appears to have been imploding8. I got fobbed off to a series of account managers (all of whom ended up leaving the company) and at this point the only contact I still have is their long-suffering founder Chris Hammerstone, whose phone is literally always busy. Eventually I realized that the only way to get attention was to go make a fuss with Enphase, which I did, and then Enphase and Chris somehow joined forces and got PG&E to come plug in the meter collar. That was... last week, August 8th. The final (post-meter-collar-install) setup bits were done this week and the system has been online since Thursday August 14th. Exactly eight months between contract signing and system completion. Eesh!

Anyhow, the power's on. Some takeaways:

  • These panels can produce 40 kWh on a sunny summer day9 and 25 kWh on a cloudy day
  • I have 15 kWh of batteries and I can already tell it isn't going to be enough on winter days
  • My house uses like 400W just sitting there empty. I guess that's the fridge10, the NAS11, and the Ubiquiti gear12? Seems high! I bet I could get that down by 50% if I went and put a meter on every power brick in the house.
  • Phases A and B are also way out of balance; I think all of the big single-pole appliances I have are on the same phase. Oops.
  • Electric dryers are brutal; we (unthinkingly) ran the dryer last night and burned something like 8kWh of battery on two loads. Once this dryer dies (it's a 10-year-old LG, how long could it have), I'm definitely getting a heat pump dryer, since they're supposedly around 80% less energy per load.
  • The Enphase iOS app (Enlighten) really sucks13. I'm making my own and will be posting about that hopefully in a couple of days.

Will any of these companies be around to support these products in a few years? Are we all going to end up in a wasteland playing a double-necked flamethrower/guitar on the front of a semi truck for the amusement of our god-kings? Who knows!

1

And 35¢/kWh at the minimum-cost time, in the wee small hours of the morning.

2

Other candidates: smartphones, Wikipedia, mRNA vaccines

4

Remember that for later in this story

3

Really, given how much they dominate in the portable-battery space, this is Anker's game to lose, and I'm surprised that I didn't see a single company installing their Anker Solix product.

5

Uh oh

6

You know, the thing I didn't really want to be in

7

How new? Well, I got mine set up in mid-April and they weren't officially approved by PG&E until June.

8

Always good when a business starts getting all 1-star Yelp reviews after you're in bed with them

9

They could produce more if they were installed better; right now, I have ~9000W of panels connected through two 20A breakers, which at the 80% NEC rule means that we're clipping at 7.04 kW. I recognize that it would've been expensive to take some of the panels and run a third string on a third breaker, and that it won't matter during most of the year when the panels will struggle to produce 7kW, but still...

10

Rated for 75W idle

11

rated for the weirdly-specific 21.71W idle

12

currently doing 23.99W of PoE plus however much the non-PoE devices are drawing

13

Even though it's just a webview, it takes around 90 seconds to start and often just hangs forever

No More Categories

This site has had both "Categories" and "Tags" since it was on Blogger in 2007. It occurs to me that this doesn't really make any sense, so I'm hiding the categories and won't be populating them going forward. The old category index pages are live (because cool URIs don't change) but otherwise, it's all tags from here on out.

A few other small changes:

  • updated some CSS
  • fixed missing </li> in page metadata block
  • chnged post time to be a <time> so it's machine-parsable, and stopped displaying post time by default
  • added signal and hackernews to the Social sidebar, and removed two defunct sites
  • archive now shows tag information

Something New

Yesterday, I left my job at Instrumentl, which I'd been at since leaving easypost almost three years ago. It was an interesting experience; I don't know that I'd choose to do it again, but it's always good to broaden one's horizons and try new things.

I was only at Instrumentl for about 990 calendar days1, but I like to think that I got a lot done in that time. When I started, the dev team was only six folks (two people in the US, one in Canada, and three in Türkiye); now it's something like 182. When I started, MFA was disabled in every tool and lots of people were logging into shared administrator accounts; now the company is on much more solid footing and perhaps even on a path to finish a SOC2 in a year or so. I built a few customer-facing features (including the public API, SSO and MFA) from scratch, and a ton of internal tools. I was only there a couple of years, but (as usual) I found myself on top of the all-time productivity leaderboards by whatever metric you care to measure3. Not that these things matter; nobody cares who wrote the most commits or who fixed the most security issues, and anybody who thinks you can objectively measure engineering performance is trying to sell you something that isn't worth buying.

Instrumentl was the first company I've worked at where I was primarily doing Ruby, which was kind of neat. Ruby was the first programming language I really learned, about 20 years ago4, and maybe the first programming5 job I had was working on HMC CS Staff porting the internally-developed ticket tracking tool6 from Rails 1 to Rails 2. I did a fair amount of Ruby/Rails at Easypost, of course, but Instrumentl was all Rails. It's... definitely a lot nicer than it was in 2008! I still think it's practically-perlish in how hard it is to read, though. I'm more-convinced than ever that implicit receivers (that is to say, being able to just type foo instead of self.foo() to call an instance method named foo) is a nightmare for reliability and code maintenance.

Anyhow, some of why I've left comes down to differences over the direction of the company, some because I simply can't stomach reviewing one more bad PR or bloated document that someone clearly let ChatGPT loose on without thinking about the consequences, and because I'm ready to work on something new and exciting. I've got a couple of days before I start at the new place, which is going to be Svix, writing Rust full-time. Let me know if you're reading this in real-time and are bored next week and want to hang out. :-)

1

and obviously far fewer working days, between weekends, holidays, company offsites, and parental leave!

2

that is to say, 17 engineers, one manager, between four and six product managers (depending on how you count execs wearing multiple hats), 3½ designers, and a partridge in a pear tree

3

tickets, commits (about 3600, all squashed), lines of code, number of wiki pages, whatever

4

using a hard-copy of Programming Ruby which I scrimped and saved to buy

5

as opposed to IT or sysadmin work

6

named request; now, alas, replaced by some terribly-enterprisey version of RT

New Keyboard

I've been using the Happy Hacking Keyboard in various configurations for the last 12 years; it's a great design for a 60% scale keyboard with all the keys that a Unix person needs and nothing they don't; in particular, I credit its lack of arrow keys with finally getting me to use vim motions correctly. Oh, I've dallied with other keyboards; for a while at work I had an Apple Wireless Keyboard, and in 2019 I bought a tenkeyless keyboard from the now-defunct WASD Keyboards1, but I always come back to the HHKB. That being said, the HHKB isn't perfect; in particular, the Topre switches feel a bit mushy2 (and get worse the longer you use them), the bluetooth version is garbage that can't maintain a connection reliably, and the cheaply-printed text on keycaps tends to rub off pretty quickly.

Well, I'm on parental leave now, so I figured I'd do a Project and put together a new keyboard.

Ingredients

This was the first time I'd actually assembled a keyboard (I know, it seems like something that I would be doing regularly); it was pretty easy3. A couple of the switches had bent pins that I couldn't quite straighten out, but I bought plenty of spares. I'm typing on the keyboard right now! I think it's pretty great so far, although obviously I have only been typing with it for a little while. One interesting note is that it's remarkably heavy — around 1500g fully-assembled (about 3x the HHKB). Not that I carry it around much, but I think I'll keep taking the HHKB with me when I go to company offsites.

Now all I need is for Apple to get around to releasing a stand-alone TouchID button...

1

Specifically, the WASD V3 87-key with Cherry MX Blue switches on it.

2

To be clear, I actually prefer a clicky keyboard; I had an IBM Model M back when I was a kid...

3

Perhaps my childhood snapping LEGO pieces together prepared me for this?

Some Good Stuff in 2024

It's January 2025, which means it's time to reflect a little about 2024. Last year, I wrote about some video games I liked in 2023, and I thought that for this year, I might expand and just talk about a few things I liked. After all, the world is currently a burning hellscape looking forward to a fascist overthrow of government and a series of apocalyptic wars on a planet no longer suitable to human life and we don't even have the best president in the last 50 years around to help out any more — seems like a perfect time for some consumerist escapism.

this is fine read more

Firefox userChrome Update

If anyone is using the style from my Firefox post a couple of years ago, you may have noted that the userChrome styling broke in Firefox 132; here's the correct new style for the same visual effect:

/* hide the native tabs */
#TabsToolbar .toolbar-items {
	visibility: collapse;
}

/* fix the titlebar color and padding */
.browser-titlebar {
	background-color: var(--toolbar-bgcolor) !important;
	justify-content: space-between !important;
	padding: 8px 0 !important;
	--inactive-titlebar-opacity: 1.0;
}

/* hide the sidebar header so Tree-Style Tabs looks native */
#sidebar-header {
	visibility: collapse;
}

/* Hide the border under where native tabs would be, to get the "unified toolbar" appearance of modern macOS */
#navigator-toolbox {
	--tabs-border-color: transparent !important;
}

/* the coloration of the titlebar to look like a toolbar */
#titlebar {
	background: var(--toolbar-bgcolor);
}

/* hide a single stray vertical line that creeps in if you have tabs hidden */
#titlebar .titlebar-spacer[type="pre-tabs"] {
	border-inline-end: 0 !important;
}

Cheers!

Oh, the humanity

Today was the keynote for Apple's World-Wide Developer Conference 2024, and, as all the pundits predicted, they announced "Apple Intelligence", a new suite of tools building on the current "Artificial Intelligence" hype bubble1. I started out feeling tentatively optimistic that they would continue to use machine learning systems to provide niceties and augmentation — replacing the two-line summary of new mail in Mail.app with auto-summarization, doing automatic segmentation to recolor app icons, improving suggestions of photos from your library that might be fun to look at, better speech-to-text transcription — all of these seemed like totally reasonable features that can be improved by things like vector embeddings, diffusion models, and deep learning transformers.

Then, of course, we actually got to the Apple Intelligence portion of the presentation. In particular, this scene:

Screenshot from WWDC24 of the new text rewriting engine making an email 'Read like a poem'
Fuck. No.

Everybody knows the Steve Jobs line:

Technology alone is not enough. It’s technology married with the liberal arts, married with the humanities, that yields the results that makes our hearts sing.

Well, someone took all of the liberal arts people out of the room when they built this feature and let the Wall Street AI hype-men steer the ship. This isn't a bicycle for the mind, this is a steamroller for the mind. It's a bulldozer that's ready to roll right over any remaining trust you have in the written word; it's taking the tools of disinformation and putting them front and center in our friendships and our personal lives.

Imagine the scene: You're sitting down and you open your email client2. You have a swath of machine-generated spam emails in your inbox warning you that you haven't yet donated enough money to the Biden campaign, that you need to ACT NOW, that a Brand™ you purchased from before is having a sale; somehow, you filter through them all to get an email from your good friend Sole and it's got a quirky poem in it inviting you to a barbecue at his house where you can hang out with your friends. Feels good right? Isn't it heartwarming to have that little touch of humanity, to know that Sole spent a few minutes wordsmithing a fun little poem for this invite? Later, you're talking to Sole and you compliment him on the poem; sheepishly, he responds with "Actually, AI wrote that".

How do you feel in this moment? Is there any scenario where you don't feel cheated? Maybe even lied to? You thought you were getting a personal note from a friend, and it's just more stochastic computer-generated spam. Would you ever read Sole's emails again? I know I wouldn't.

I'm sure this feature got a product manager somewhere a big promotion. Heck, I'm sure there are people who have become so inured to everything being algorithmically-generated spam that they won't even notice or mind that their friends are now sending them messages like this. But, Apple's ethos3 has always been about building tools to empower users to make art, to create, to be original. I don't know what this is, but it sure as hell isn't human creativity.

Let's not even talk about these hallucinogenic "Image Playground" results and the real artists whose work was surely purloined to generate them...4

Apple Image Playground
Apple Image Playground (shudder...)

Maybe I'm alone. Maybe nobody else feels the deep and primal outrage at the thought of parents outsourcing their kids' bedtime stories to artificial stupidity, and at the thought of our friends messages to us being mediated by fancy autocomplete. But, damn! The resounding "who cares" from the tech industry over the last few months, the openness to never being able to tell if written text is actually someone's true thoughts or is just adjacent to their thoughts in a 1536-dimensional vector space... it feels a bit like a betrayal. If you decide you want to use these tools, feel free to leave me off your barbecue invite lists, because I don't think we have compatible values.

1

If you're lucky enough to be out of the loop, this is currently focused on the use of very large neural networks trained on truly enormous volumes of text which are able to do essentially sophisticated autocomplete -- sophisticated enough that people with a really depressing theory-of-mind sometimes believe it's intelligent, even though all it's doing is picking the most statistically-likely next word given the previous words. It's been a real eye-opened watching a lot of business executives give up on the idea that consciousness exists and embrace the idea that a totally deterministic word-picking automaton can replace their workforces.

2

Assuming you're not one of the many folks who has given up on email entirely and just has a counter of hundreds of thousands of unread missives.

3

At least, the ethos that their careful marketing team tries to portray. But, I do know people who work at Apple and do more than pay lip service to it!

4

Let's also not talk about Apple 2030, Apple's pledge to be zero-net-carbon in a scant 6 years, and how this is going to be affected by sending these queries out to OpenAI et al to burn some megawatts on.