Atomic desktops / iot / coreos caching issues

I spent a lot of time looking into some odd issues that ostree users were hitting. It was really hard to track down what was broken. No errors on our end, invalidated cloudfront a few times, did a bunch of tweaks to our backend varnish cashes and... the problem was caused by: me.

Turns out we are getting hit really hard all the time by (what I can only assume is crawlers working to fuel LLM's. It's not just us, see for example this excellent lwn article on the problem

We use amazon Cloudfront to serve ostree content to users, since it allows them to hit endpoints in their local region, so it's much faster and reduces load on our primary cache machines. Cloudfront in turn hits our cache machines to get the content it caches.

How does this relate to ostree issues you might ask? Well, I blocked a bunch of IP's that were hitting our kojipkgs servers particularly hard. It turns out some of those IP's were cloudfront, so just _some_ of the cloudfront endpoints didn't have access to the backend so their cache was out of date. I assume cloudfront also has multiple distributions at each region and it was only _some_ of those.

Removing all those blocks got everything working for everyone again (but of course the AI bots are ever present). I also enabled a thing called 'origin shield' which means cloudfront should only pull from one region and sync to the others, reducing load on our caches.

Longer term we probibly need to split up our kojipkgs cache or add more nodes or rearage how things are hit.

I'm deeply sorry about this issue, I know many users were frustrated. I sure was too. Lesson learned to be carefull in blocking bots.

s390x caching problems

And related to that issue, our s390x builders have been having problems pulling packages for builds. They have a local cache that in turn pulls from our primary one. Sometimes, sporadically, it's getting partial downloads or the like. I've still not fully figured out the cause here, but I did make a number of changes to the local cache there that seem to have reduced the problem.

Longer term here we probibly should seperate out this cache to hit a only internal one so the load on the main one doesn't matter.

Coffee machine fun

Friday my coffee machine ( delonghi magnifica ) got stuck in the middle of a cycle. It had gound the beans, but then stopped before the water step. So, I looked around at repair videos and then took it apart. It was actually pretty cool how it was put together, and I was able to basically manually turn a pully to move it down to unlocked, then I could remove the brew group and clean everything up and put it back together. Working great again. Kudos also to the The iFixit pro toolkit that I got a while back. A weird screw? no problem.

Home assistant

Been having a lot of fun tinkering with home assistant.

After looking, decided that zigbee networking is better than bluetooth and less power hungry than wifi, so I picked up a Zigbee gateway and it works just fine. At one point I thought I accidentally flashed it with esp32 builder, but seems it didn't work, so whew.

Got some smart plugs ( Amazon link to smart plugs ) and these little things are great! pair up fine, HA can manage their firmware versions/update them, lots of stats. I put one on the plug my car charges on, another on a plug that has a fridge and a freezer on, one on the plug my main server UPS is on, and kept one for 'roaming'. It's cool to see how much power the car charging takes in a nice graph.

Got some cheap temp sensors ( Amazon link to temp / humidity sensors ) They seem to be working well. I put one in my computer closet, one in our living room, one in the garage and one outside. (The living room seems to have a 4 degree change from day to night)

I had some old deako smart switches along with a gateway for them. They use a bluetooth mesh to talk to each other and an app, but the gateway is needed for them to be on wifi. I never bothered to setup the gateway until now, but HA needs it to talk to the switches. So I tried to set it up, but it would just fail at the last setup step. So, I mailed daeko and... they answered really quickly and explained that the gateway is no longer supported, but they would be happy to send me some of their new smart switches (that have wifi built in and can act as a gateway for the old ones) free of charge! I got those on thursday and set them up and they worked just dandy. But then I tripped over Chestertons Fence. The 3 old smart switches were all controlling the same light. That seemed silly to me. Why not just have one on that light, use two 'dumb' switches for the other two places for that light and then move the other smart ones to other lights? Well, turns out there are several problems with that: The 'dumb' switches have a physical position, so if you did that one could be 'on' with the light off, another 'off', etc But the biggest problem is that the smart switch is needed to route power around. If you turn the light 'off' on a 'dumb' switch you can have the one smart one with no power and it doesn't do anything at all. So, after messing them up I figured out how to factory reset them and re-pair them. For anyone looking the process is:

Resetting:

• plug in and while it 'boots', press and hold the switch.

• it should come up with a 3 2 1 buttons to press.

• press each in turn

Pairing (you have to pair switches that all control the same lights):

• unplug all switches

• plug one in and Press and hold the switch it should come up with a flashing 1

• If nothing happens, try each of the other two in turn. Only one has 'power'

• press 1 on the first switch.

• Repeat on switch 2 and press 2 on the first switch

• Repeat on the last switch and press 3 on the first switch

I could have saved myself a bunch of time if I had just left it the way it was. Oh well.

Finally I got some reolink cameras. We have a small game camera we put out from time to time to make sure the local feral cats are ok, and to tell how many racoons are trying to eat the cats food. It's kind of a pain because you have to go put it outside, wait a few days and then remember to bring it back in, then pull the movies off it's sdcard.

So replacing that with something that HA could manage and we didn't need to mess with sounded like a win. I picked up a bundle with a Home Hub and two Argus Eco Ultra and 2 solar panels for them.

The Home hub is just a small wifi ap with sdcard slots. You plug it in and set it up with an app. Then, you pair the cameras to it and HA talks to the cameras via the hub. There's no external account needed, setup is all local and you can even firewall off reolink if you don't want them to auto uprgade firmware, etc. I've not yet set the cameras up outside, but a few impressions: The cameras like REALLY LOUD AUDIO. When you first power them on they greet you in a bunch of languages and tell you how to set them up. Thats fine, but when I did this people in my house were sleeping. Not cool. Even powering them off causes a chirp that is SUPER lOUD. The cameras have a 'siren' control that I have been afraid to try. :) Anyhow, more on these as I get them setup.

I had 2 UPSes here. One for my main server and critical loads and another one for less important stuff. With all the home assistant stuff I ran out of battery backed plugs, so I picked up a 3rd UPS. The new one was easy to add to nut, but I had a long standing problem with the two I already had: They are exactly the same model and product and don't provide a serial number on the usb port, so nut can't tell them apart. Finally I dug around and figured out that while I was specifying bus, port and device, it wasn't working until I moved one of them to another USB plug (and thus another bus). I then got all 3 of them added to HA. One thing that confused me there is that since all 3 of them are on the same nut server and are using the same upsmon user, how do I add more than 1 of them in HA? Well, it turns out if you go to the nut integration, add device, enter the same host/user/pass it will pop up a screen that asks you which one to add. So you can add each in turn.

So, lots of fun hacking on this stuff.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114128294620402197