Skip to main content

DC move, riscv, AI and more for early February 2025

Scrye into the crystal ball

Hey, another week, another blog. Still going, still hope things are of interest to you, the reader.

PTO

I realized I had some PTO (Paid time off, basically what used to be called 'vacation') that I needed to use before it disappeared, so I have planned a number of vacation days in the coming month. Including 2 in this last week. :)

I'll also point to an old post of mine about what I personally think about vacations/pto when you are working in a community: time off when you are paid to work in a community

The biggest things for me about this isn't completely disconnecting or ignoring everything, in fact most days I get up at the same time (cats have to be fed!) and sit at my laptop like I usually do, but it's that I don't have to attend meetings, and I can do things I fully enjoy. Sometimes thats still working in the community, but sometimes its like what I did on friday: Look into a battery backup for house and tradeoffs/ideas around that. In the end I decided not to do anything right now, but I had fun learning about it.

My next pto days are next thursday ( 2025-02-20 ) then next friday is a "recharge" day at Red Hat, then I have monday off ( 2025-02-24 ), then march 6th and 7th and finally march 11th.

Datacenter Move

More detailed planning is ramping up for me. I have been working on when and how to move particular things, how to shuffle resources around and so forth. I plan to work on my doc more next week and then open things up to feedback from everyone who wants to provide it.

A few things to note about this move:

  • There's going to be a week (tenatively in may) where we do the 'switcharoo'. That is, take down services in IAD2 and bring them up in RDU3. This is going to be disruptive, but I'm hoping we can move blocks of things each day and avoid too much outage time. It's going to be disruptive, but we will try to minimize that.

  • Once the switcharoo week is over and we are switched, there will be either no staging env at all, or a limited one. This will persist until hardware has been shipped from IAD2 to RDU3 and we can shuffle things around to bring staging entirely back up.

  • Once all this is over, we will be in a much better place and with much newer/faster hardware and I might sleep for a month. :)

riscv secondary koji

Slow progress being made. Thanks to some help from abompard auth is now working correctly. It was of course a dumb typo I made in a config, causing it to try and use a principal that didn't exist. OOps. Now, I just need to finish the compose host, then sort out keytabs for builders and hopefully the riscv SIG can move forward on populating it and next steps.

AI

Oh no! This blog has AI in it? Well, not really. I wanted to talk about something from this past week thats AI related, but first, some background. I personally think AI does have some good / interesting uses if carefully crafted for that use. It's a more usefull hype cycle than say cryptocoins or blockchain or 'web3', but less usefull than virtual machines, containers or clouds. Like absolutely anything else, when someone says "hey, lets add this AI thing here" you have to look at it and decide if it's actually worth doing. I think my employer, Red Hat, has done well here. We provide tools for running your own AI things, we try and make open source AI models and tools, we add it in limited ways where it actually makes sense to existing tools ( ansible lightspeed, etc).

Recently, Christian Schaller posted his regular 'looking ahead' desktop blog post. He's done this many times in the past to highlight desktop things his team is hoping to work on. It's great information. In this post: looking ahead to 2025 and fedora workstation and jobs on offer he had a small section on AI. If you haven't seen it, go ahead and read it. It's short and at the top of the post. ;)

Speaking for myself, I read this as the same sort of approach that Red Hat is taking. Namely, work on Open source AI tooling and integrations, provide those for users that want to build things with them, see if there's any other places that could make sense to add an integration points.

I've seen a number of people read this as "Oh no, they are shoving AI in all parts of Fedora now, I'm going to switch to another distro". I don't think that is at all the case. Everything here is being done the Open Source way. If you don't care to use those tools, don't. If AI integration is added it will be in the open and after tradeoffs and feedback about being able to completely disable it.

ansible lint

We had setup ansible-lint to run on our ansible playbooks years ago. Unfortunately, due to a bug it was always saying "ok". We fixed that a while back and now it's running, but it has some kind of opinionated ideas on how things should be. The latest case of this was names. It wants any play name to start with a capitol letter. Handlers in ansible are just plays that get notified when another thing changes. If you change the name of say "restart httpd" to 'Restart httpd" in the handler, you have to then change every single place that is notified too. This caused an anoying mess for a few weeks. Hopefully we have them all changed now but this rule seems a bit random to me.

fedmsg retirement

In case you didn't see it, we finally retired our old fedmsg bus! We switched to fedora-messaging a while back, but kept a bridge between them to keep both sides working. With the retirement of the old github2fedmsg service we were finally able to retire it.

🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/114009437084125263

Bits from early February 2025

Scrye into the crystal ball

Lets keep the blogging rolling. This week went by really fast, but a lot of it for me was answering emails and pull requests and meetings. Those are all important, but sometimes it makes it seem like not much was actually accomplished in the week.

riscv secondary koji hub

I got some x86 buildvm's setup. These are to do tasks that don't need to be done on a riscv builder, like createrepo/newrepos or the like. I'm still having a issue with auth on them however, which is related to the auth issue with the web interface. Will need to get that sorted out next week.

f42 branching day

Tuesday was the f42 branching day. It went pretty smoothly this cycle I think, but there's always a small number of things to sort out. It's really the most complex part of the release cycle for releng. So many moving parts and dispirate repos and configs needing changing. This time I tried to stay out of actually doing anything, in favor of just providing info or review for Samyak who was doing all the work. I mostly managed to do that.

Datacenter move

Planning for the datacenter move is moving along. I've been working on internal documents around the stuff that will be shipped after we move, and next week I am hoping to start a detailed plan for the logical migration itself. It's a pretty short timeline, but I am hoping it will all go smoothly in the end. We definitely will be in a better place with better hardware once we are done, so I am looking forward to that.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/113969409712070764

Bits from late jan 2025

Scrye into the crystal ball

January has gone by pretty fast. Here's some longer form thoughts about a few things that happened this last week.

Mass updates/reboots

We did a mass update/reboot cycle on (almost) all our instances. The last one was about 2 months ago (before the holidays), so we were due. We do apply security updates weekdayly (ie, monday - friday), but we don't apply bugfix updates except for this scheduled windows. Rebooting everything makes sure everything is on the latest kernel versions and also ensures that if we had to reset something for other reasons it would come back up in the correct/desired/working state. I did explore the idea of having things setup so we could do these sorts of things without an outage window at all, but at the time (a few years ago) the sticking point was database servers. It was very possible to setup replication, but it was all very fragile and required manual intervention to make sure failover/failback worked right. There's been a lot of progress in that area though, so later this year it might be time to revisit that.

We also use these outage windows to do some reinstalls and dist-upgrades. This time I moved a number of things from f40 to f41, and reinstalled the last vmhost we had still on rhel8. That was a tricky one as it had our dhcp/tftp vm and our kickstarts/repos vm. So, I live migrated them to another server, did the reinstall and migrated them back. It all went pretty smoothly.

There was some breakage with secure boot signing after these upgrades, but it turned out to be completely my fault. The _last_ upgrade cycle, opensc changed the name of our token. From: 'OpenSC Card (Fedora Signer)' to 'OpenSC Card'. The logic upstream being "Oh, if you only have one card, you don't need to know the actual token name". Which is bad for a variety of reasons, like if you suddenly add another card, or swap cards. In any case I failed to fix my notes on that and was trying the old name and getting a confusing and bad error message. Once I managed to fix it out everything was working again.

Just for fun, here's our top 5 os versions by number:

  • 237 Fedora 41

  • 108 RHEL 9.5

  • 31 Fedora 40

  • 23 RHEL 8.10

  • 4 RHEL 7.9

The 7.9 ones will go away once fedmsg and github2fedmsg are finally retired. (Hopefully soon).

Datacenter Move

A bunch of planning work on the upcoming datacenter move. I'm hoping next week to work a lot more on a detailed plan. Also, we in infrastructure should kick off some discussions around if there's anything we can change/do while doing this move. Of course adding in too much change could be bad given the short timeline, but there might be some things to consider.

I also powered off 11 of our old arm servers. They had been disabled in the buildsystem for a while to confirm we didn't really need them, so I powered them off and saved us some energy usage.

riscv-koji seconday hub

The riscv-koji seconday hub is actually installed and up now. However, there's still a bunch of things to do:

  • Need to setup authentication so people/I can login to it.

  • Need to install some buildvm-x86 builders to do newrepos, etc

  • Need to install a composer to build images and such on

  • Next week's riscv sig meeting hopefully we can discuss steps after that. Probibly we would just setup tags/targets/etc and import a minimal set of rpms for a buildroot

  • Need to figure out auth for builders and add some.

Overall progress finally. Sorry I have been a bottleneck on it, but soon I think lots of other folks can start in on working on it.

power9 lockups

We have been having anoying lockups of our power9 hypervisors. I filed https://bugzilla.redhat.com/show_bug.cgi?id=2343283 on that. In the mean time I have been moving them back to the 6.11.x kernels which don't seem to have the problem. I did briefly try a 6.13.0 kernel, but the network wasn't working there. I still need to file a bug on that when I can take one down and gather debugging info. It was the i40e module not being able to load due to some kind of memory error. ;(

Chat and work items

One thing that was bugging me last year is that I get a lot of notifications on chat platforms (in particular slack and matrix) where someone asks me something or wants me to do something. Thats perfectly fine, I'm happy to help. However, when I sit down in the morning, I usually want to look at whats going on and prioritze things, not get sidetracked into replying/working on something thats not the most important issue. This resulted in me trying to remember which things where needed responses and sometimes missing going back to them or getting distracted by them.

So, a few weeks ago I started actually noting things like that down as I came to them, then after higher pri things were taken care of, I had a nice list to go back through and hopefully not miss anything.

It's reduced my stress, and I'd recommend it for anyone with similar workflows.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/113930147248831003

Bits from mid late jan 2025

Scrye into the crystal ball

Hello again everyone! Three weeks in a row now. Go me. I'm not really sure what to name these posts, especially given that 'early/mid/late' does not work when there's 4 weeks in a month. I could just put the dates in the title, but that seems difficult to see what is going on. Perhaps I just need to pick out some highlights.

mass rebuild completed

The f42 mass rebuild was completed. This time submitting from the mass rebuild script seemed slower than normal. Possibly due to the item scaper item below. However, things mostly worked out. Instead of submitting everything and koji churning through the backlog, submissions were usually just handled as they came in, leaving koji with really no backlog.

A.I. Scrapers

The A.I. scrapers continue to pound away at things. I mean, I'm assuming thats what they are as they crawl all the content they can get. They are pretty hard to block due to using massive piles of IP's, different user agent strings every time and random access of content. This sort of thing is hitting lots of open source communities too. I think short time we probibly need to look at deploying mod_qos at least, so we can prioritize users/things that need access and let the scrapers get much lower levels of request answering. No idea if that will really stop them though.

Anoying outage on thursday

We had a very anoying outage on thursday. Our rabbitmq cluster started not processing things sometimes and just in general not working right. Restarting it and then a bunch of consumers that connect to it finally got things working again, but it's unclear what the cause was, and cleaning up a bunch of builds and such that were not processed due to missing messages was not at all fun. Currently our rabbitmq cluster is on rhel8 and an older version, we probibly want to move it to 9 and start using the version made by the CentOS messaging sig. If newer/older can talk to each other in a cluster it should be pretty easy. If they can't it will need some downtime.

Mass update/reboot next week

As we do from time to time, there's going to be a mass update/reboot cycle next week. Monday will be staging, then tuesday will be a bunch of things that don't cause any outages (where we can take things out and update/reboot and put them back in), and finally wed will have the actual outage.

riscv secondary koji hub progress

The riscv hub is installed and mostly configured, still need to sort out a few packages it needs to complete the ansible playbook. Next I need to test web interface, then install some x86 builders (that will do newrepos) and a compose host (for composes). Then will come the hard work of importing content / setting up actual riscv builders (which will be external for now until we can find hardware to use locally).

Datacenter move

We announced that we are going to be moving from our main datacenter (IAD2 in ashburn, va) to a new one (RDU3 in Raleigh, NC). It's going to be a ton of work (and lots has already happened, been spending a lot of time on it already), but when it's done we should be in a much better place. More space, better machines, and hopefully things like ipv6 support (that we have wanted for many years).

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/113891021464841039

Bits from mid jan 2025

Scrye into the crystal ball

Hello again, here's some longer form doings and thoughts from from mid january 2025 in and around fedora.

rawhide repodata change

Rawhide repodata has moved over to the createrepo_c default: zstd. This shouldn't affect any dnf use, or fedora createrepo_c use, but if you are using EL8/EL9, createrepo_c there currently doesn't understand zstd. There's a issue to add that in a minor release: https://issues.redhat.com/browse/RHEL-67689 and in the mean time if you are a EL8/9 user there's a copr: https://copr.fedorainfracloud.org/coprs/amatej/createrepo_c/

dist-repos retention

On the dist-repos space issue I mentioned last week, it turns out that the expectation on dist-repos is that you would sync them somewhere you wanted to serve them from, not just serve from koji. So our use case was misalined a bit from upstream here. However, they did adjust to keep latest repos. Should be in a upcoming koji release. Thanks koji folks!

fun email infrastructure mixup between ipa and postfix

There was a pretty curious email issue that came up this last week. fedoraproject contributors (that is folks with an account that is in at least one non system group) are setup with an email alias of theiraccountlogin@fedoraproject.org. This is just a very simple alias. We accept the email and forward it to their real email address. There's no mailbox here or authentication or anything, just a simple alias. We got an alert that disk space was getting low on our mail hub, so I took a look and found that users who were not contributors were getting emails to theiraccountlogin@fedoraproject.org delivered locally to /var/spool/mail on the hub! When we switched away from fas2 to our current IPA based setup, no one realized that sssd/ipa enumerates all users, even if they do not have access to actually login or do anything. There are good reasons for this, but somehow I at least didn't realize that it worked that way. So, since all users 'existed' there, and postfix's default for local users is:

proxy:unix:passwd.byname $alias_maps

It correctly looks them up and thinks they are local. Simply changing this to just be $alias_maps fixes the issue. There wasn't a bug here in postfix or ipa, they were just doing things as they expected. The issue was our misunderstanding how these things interacted.

f42 mass rebuild underway

The mass rebuild for f42 started (all be it a bit later than planned due to a issue getting golang to work on i686 when compiled with gcc 15). This time it seems like our submitting builds is much slower than before. In past years, pretty much everything was submitted in a few days, then koji chewed on the backlog. This time koji is easily keeping up with the submissions and we are only in the 'p's after 3.5 days. Oh well, hopefully we will finish mondayish, which is in line with past mass rebuilds.

forgejo kickoff meeting/discussions

There was a kickoff meeting about forgejo in fedora infra. I'm looking forward to this, but I have so many things going on I am not sure how much work I can do on the deployment. Lots of good ideas/plans discussed. I think it's going to be not a super lot of work to stand up a staging instance, but I think integrating with all our workflows will take a lot more effort. Time will tell.

riscv secondary koji hub

I did finally submit my work in progress PR for riscv-koji hub: https://pagure.io/fedora-infra/ansible/pull-request/2435 Hopefully can finish off things and start deploying next week.

bugzilla and needinfo

I asked on mastodon what folks thought about bugzilla needinfo requests and what they meant. There were a number of opinions: https://fosstodon.org/@nirik/113822583672492457 I think in the end it's a thing that people will use for their own use cases and those will sometimes mis match with recipients. Unless we want to try and make some community wide norm or guidelines (but of course even then not everyone will see those).

Xfce-4.20 and wayland testing

News from a while ago: Xfce 4.20 was released and it's got a bunch of wayland support for various things. However, it doesn't have xfwm4 / a compositor of it's own, so by default you get a X session. If you want to play with the wayland sessions and you are running a rawhide instance, you can install: xfce4-session-wayland-session which will by default pull in labwc as your compositor. You can manually modify the session file to use wayfire if you prefer that compositor. See the testing section at https://wiki.xfce.org/releng/wayland_roadmap I tried both out and they did indeed work, but there are still a bunch of rough edges. Still, great progress!

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/113850897869803740

Bits from early jan 2025

Scrye into the crystal ball

Welcome to 2025. It's going to be a super busy year in Fedora land, all kinds of things are going to be worked on and hopefully rollout in 2025. Back from the holiday break things started out somewhat slow (which is great) but already have started to ramp up.

First up I fixed a few issues that were reported during the break:

  • Our infrastructure.fedoraproject.org host wasn't redirecting http to https (as all our other hosts do). Turns out we disabled this many years ago because virt-install couldn't handle https for some reason. I think this was in the RHEL6 days, and we just never went back and fixed it. This did end up breaking some provisioning kickstarts that had http links in them, but easy (and good!) to fix.

  • Some redirects we had for sites were retirecting to just {{ target }} variables, but if that was examplea.fedoraproject.org////exampleb.com it would redirect to examplea.fedoraproject.org.exampleb.com. A totally different domain. Fixed that by making sure there was a / at the end of the redirect. Unfortunately, I also broke the codecs site for basically everyone. ;( It was broken for a few hours, but easy to fix up after I was made aware.

There's been a ton of f42 changes flowing before/during/after the holidays. Lots of exciting things. Hopefully it can all land and work out ok.

I finally started in on work for a riscv-koji secondary arch hub. It would have been really easy, but we dropped all the secondary things from ansible after the last secondary arch hub went primary, so I am having to go though and adjust a lot of things. Making good progress however. Hopefully something to show next week. This secondary hub will allow maintainers to login/do scratch builds and get things one step closer to primary. There's still a long road for it though, as we need actual local builders and proof of keeping up with primary.

Next I cleaned up some space on our netapp (because see below).

  • I archived some iot composes and marked a bunch more for deletion. As soon as I get the ack on those, that should free up around 5TB.

  • I noticed our dist-repos were really pretty large. This turns out to be two issues: First, we were keeping 6 months of them. We did that because we use these for deployments and before if all repos were older than the last change to them, they just would be missing. I am pretty sure now that kojira keeps the latest one, so this is no longer a factor. I set them to 1 week (as default). This should free up many TBs. Secondly, the flatpak apps repos were not using the latest (they were pulling everything). Adjusted that and it should save us some.

  • Finally, I nuked about 35TB of old db backups. There's no reason to keep daily database dumps since 2022. I kept one from each month, but we have never had to go back years for data. In particular the datanommer and koji db's are... gigantic, even compressed. Unfortunately it will be a while before this is actually freed as we have a lengthy snapshot retention policy on the backup volume. Still should help a lot down the road.

With some freed up space, I now could make another iscsi volume and move our ppc64le buildvm's off to it. It took far longer than I would like to admit to get things working (turns out I had logged in on the client before setting up everything on the netapp and it just didn't see the lun until I ran a refresh). I expected there to be a pretty vast speed improvement, and the vm's are indeed a lot more responsive and install much faster, but I am not sure builds are really that much faster. Will need some more poking around to find out. The local disk on those are 7200 rpm spinning sata drives. The iscsi is a ssd backed netapp over 10G network. Unfortunately also we have seen instablity in the hosts in the last week, which is likely a kernel issue since I updated them to 6.12. Hopefully I can get everything sorted out before the f42 mass rebuild, which is next wed.

comment/reply on mastodon: https://fosstodon.org/@nirik/113811216382678546

Apps I use

Scrye into the crystal ball

Happy New year everyone.

I thought I would do a quick post about apps/software that I use. Of course my requirements may be wildly different from your own, but perhaps you will see something here that you might want to also investigate, or in turn leeds you to something you do want to use.

On the server side, I want to use things that are open source and run on Fedora (my main server at home is Fedora of course). I prefer things packaged up and available in the main Fedora repos, but of course that is not practical for everything sadly.

  • nextcloud: nextcloud continues to be a great solution for a lot of things due to it's library of plugins. I use it for files, uploading photos/videos (see below), organizing photos, recipies, calendar, contacts, deck lists (kanban) and more. It's not packaged (anymore) in Fedora, but it's pretty easy to install and upgrade on the fly. I've also been impressed lately that things like files are... files on the server, not some weird db format that are difficult to add/delete/refresh if you need to from the server side. I also do phone backups to nextcloud (see below)

  • postfix/dovecot/opendkim/opendmark/spamassassin/sqlgrey/saslauthd for email. This seutp has worked for years and years and just works fine.

  • znc for irc bouncer. I still connect to a number of IRC networks (although I am usually much more active on matrix).

  • matrix-synapse for matrix server. I'm just using the packaged Fedora version and it works fine. Someday I will probibly move it to use one of the setups that has more bells and whistles but for now it's fine.

  • postgres for database server for everything that needs one. I was running also a mariadb instance, but I moved the last things off it over the holidays and didn't setup one on the new server.

  • miniflux for RSS. ( https://miniflux.app/ ). miniflux isn't packaged directly in Fedora, but the upstream folks provide a repo and rpms and they work just fine. This allows me to manage/read RSS feeds via web interface, or (usually) just manage/read via an app (see below).

  • I was running calibre in headless server mode to serve ebooks but it's really vastly overkill to do that and it pulls in about 233 rpms which I otherwise do not need. So, I switched to a simple OPDS app: cops ( https://github.com/mikespub-org/seblucas-cops ) (well, a fork of it that works with recent php). It seems to do the job just fine. I can manage things with calibre on my laptop and rsync them to the server where cops serves them to my phone (or whatever reader I want).

On the Linux client side (my laptop):

  • firefox as main web browser, occasionally having to use chrome or chromium. Mozilla hasn't been making good choices of late, and I really hope they find their way again, but I really really dislike the idea of using web browsers or engines that are made by super large companies for their own gain. I'll probibly stick with firefox until it becomes untenable. Perhaps servo will be ready by then?

  • hexchat for IRC. Been using it for ages and ages.

  • discord (flatpak app). I have some friends that I have known since college and we have a discord server to chit-chat on, so reluctantly I connect to that to keep in touch.

  • necko for matrix. I use the Fedora packaged version and it's the best of the matrix clients most of the time. It still has flaws of course, but day to day it's the best one for me.

  • newsflash (flatpak) for rss client reading. Newsflash used to be packaged in Fedora, but it just became too difficult, so I use the flatpak now. newflash looks great, works well, and is a very pleasent reading experence. It connects easily to miniflux (see above). Doing things that way allows me to read things on the laptop (Newsflash), web (miniflux) or phone (see below) and keep all of them in sync so I don't re-read things.

  • foot for terminals. I've used... a lot of terminals over the years and almost all of them are just fine if needed, but I've really taken to foot over the last few years. It's super quick, it allows me to have italic fonts (call me crazy, but I find oblique/italic to be easier on my eyes and vte based terminals no longer allow that). I do use xfce4-terminal on Xfce, because of course foot is a wayland only terminal.

  • calibre for ebook / library management. There's a number of new ebook library managers up and coming, but calibre is still far ahead of all of them in my opinion.

Finally on my phone (a google pixel 8a) running https://grapheneos.org/ Probibly too many apps to note, but ones that interact with my laptop and main server:

  • firefox here as well. The mobile version has gotten much better over recent years. Sadly, installed from google play store.

  • antennapod for podcasts.

  • tusky ( https://tusky.app/ ) for mastadon reading/posting.

  • librera ( https://tusky.app/ ) for ebook reading. While making this post I happened to see I had it installed from google play store, I think because in the distant past there was something that didn't work in the f-droid version, but no longer. I switched it over to the f-droid version and it's working fine. Its a nice reader, it hooks into odps on the server just fine. I've ready so many books with this thing.

  • KISS launcher. I ran accross this a while back somewhere and it's still my main application launcher. It provides your apps as a searchable list, with the ones you use the most at the bottom so you can easily find them. This is so much nicer than paging though a bunch of virtual desktops looking for some icon. Highly recommended.

  • Nextcloud android apps: nextcloud, deck, memories, cookbook all integrate nicely with nextcloud. Deck is nice for shopping lists or organizing things. The main nextcloud app lets you sync things back and forth and setup autosync. I have it syncing my photos and movies up right after I take them. Memories then lets you look at and organzie. cookbook is handy for using in the kitchen when you want to follow a recipe.

  • Element X for matrix. Works fine, has the new 'fast' sync, which seems to work reasonably well.

  • Paseo for step counting. I installed this last year and it's been nice be able to see that I need to get up and walk around more. It hooks into the android steps stuff so walking on the eliptical or the like will show up as steps even though you didn't go anywhere.

  • A bunch of junk non free apps I need for various things sadly, but it's nice to have the option of installing from play store if you absoluetely need some app and can't avoid it.

Im sure there's more things I didn't remmeber or see while looking here but hopefully the list inspires you somehow.

I'm still figuring out comments to blog posts, but if you want to reply, I will be making a mastodon post pointing to this blog post you can reply to: https://fosstodon.org/@nirik/113766175760197132

Holiday Hacking 2024

Scrye into the crystal ball

Like many other folks, over holidays I like to read books, watch movies and tv shows, visit family, bake and eat too much food, drink too many good beers and ciders and meads, and catch up some on around the house projects.

Also, of course I like to catch up on my hacking on things. Often during the normal times I am busy at work and don't have the time or energy to play around with things, or just improve my home infrastructure ( queue up joke about the cobblers children never having new shoes).

So, I thought I would put together a recap of things I looked at/setup in case they inspire others.

There may well be a part two of this, but we will see how much more I get done before the holidays end.

  • Got sound working again on my media PC via HDMI. Sadly, I didn't actually fix it so much as swapped a usbc doc with HDMI in for the frame.work HDMI module that was not working. I'm really not fully sure why it no longer lets me have sound, but I am suspecting a kernel/firmware issue. Might play with it again someday, but it all works fine now.

  • Unpacked and setup the OpenWrt One (see previous blog post for a review). So far it's quite fast and working great.

  • Installed a SNO (Single Node Openshift) in a vm to play with. Found that it's not covered under the Developer subscription (gives you a 60day trial). So decided to ponder more on what k8s thing I want to play with at home. Contenders: k3s, microshift, OKD of some flavor.

  • Moved this blog from wordpress to nikola. I'm sure there's still rough edges, but overall it seems working. Spent longer on trying to get apache redirects working than importing and setting up nikola. (It was a ordering issue, there was a redirect that was basically overriding all the new ones I was trying to put in place).

  • Cleaned up my restic backups as the partition was starting to get full.

  • Updated my Nextcloud instance to 30.0. Added some new apps. The cookbook app is brilliant! You can (usually) just point it to a url and it will scrape the recipe in with all info, media and such. Really nice. The memories app is a nice improvement over the default photos app I think. Cleaned up old trashbin and versions of files that saved a bunch of space.

  • Looked at my piwigo install. I replaced an old gallery2 instance a long time ago with this, but I haven't really done much with it in recent years. It's also the last app I have that uses mysql/mariadb. So, I imported all the files into nextcloud for now and I think I will retire piwigo. Nextcloud memories still isn't great for public sharing, but you can share albums with a public link.

  • Decided it was time to move my main server over to a fresh install. It's working just fine, but /boot is only 185MB, so it can only hold 2 kernels which is anoying. It's also ext3 (yes, ext3!). The / fs is ext4, and I would like to get compression at least from btrfs. This vm was installed with f32 and upgraded to f41 with almost no issues. Did a f41 install and synced data over to it: 309GB becomes 189GB on the new vm. I'm still organizing and moving things. Probibly will cut over around new years with a hopefully short downtime.

  • Got a calendar app on my phone and via the magic of DAVx managed to get my nextcloud calendar, my work google calendar and my home google calendar all syncing in one place. The fossify calendar is not too bad, installable via fdroid and seems reasonably active upstream.

Back to hacking on things and trying to relax some.

OpenWrt one - a short review

Recently the OpenWrt One was announced for sale. This is a wireless access point/router powered by Banana Pi and designed by the OpenWrt project. Additionally, $10 from every device sold go to the Software freedom conservency to help fund OpenWrt efforts.

The device was available on aliexpress, which is a bit weird for us here in the west, but I had no trouble ordering it there and the cost was pretty reasonable. It arrived last week.

OpenWrt One box

The design is pretty nice. There's a NAND/NOR switch. Normal operation the switch is in NAND setting. If something goes wrong, you can hold down the button on the front while powering on and you shoule get a rescue image. If somehow even that image doesn't work, you can switch the switch to NOR mode and it will boot a full recovery from a USB drive. So, pretty unbrickable.

Initial setup was easy. Just screw on the 3 antenna, connect ethernet and usb-c power and everything came up fine. I was a bit confused on what password to use, but then I realized just hitting return would take me to the 'hey, please set a password' screen. A small note might be nice there.

Since I was using OpenWrt on my existing linksys e8450 it was pretty simple to configure the new accesspoint in a similar manner. Upgrade was pretty easy as soon as I realized that I needed to pick 24.10.0-rcN or snapshot on the firmware selector as there are no stable images for the One yet.

I then spent a lot of time playing with the channel_analysis page. This page scans for other accesspoints and shows you what channels are in heavy use or open. On 5ghz, there was basically nothing else, so no problems there. However, on 2.4Ghz there were an astonishing number of aps. I live out pretty far from town, but there's still a LOT of them. Of course some were coming from 'inside the house' like some roku devices or the like. Finally I decided channel 9 was the best bet.

switching things was a bit of a dance. I connected to the openwrt wireless network, logged in and changed the wired network, then powered off the old ap and swapped the network cable to the new one. Then, rejoined the wireless and changed the name/password so all the existing devices would just keep working.

I do notice faster connection rates on my main laptop at least. The accesspoint is also really responsive either via web (luci) or ssh. I may look at adding some more duties to this device over time. It does have a nvme slot so I could do some caching or perhaps some other setup. I also want to play with the usb-c console port and perhaps at some point upgrade my home switch so I can power it via PoE.

All in all a pretty great device. It seems to currently be sold out, but if you are looking for a nice, unbrickable ap that is very open source, this might just be the ticket for you.

OpenWrt One up and routing away

Hello from nikola

Hello again everyone.

After using wordpress for more than 20 years, I finally decided it was time to move off of it. I'm not really happy about the recent turmoil from the upstream wordpress folks, and I didn't think there was too much value over just moving to a static generator as so many have before me.

I did some looking around, and decided to just go with nikola. It uses python and seems pretty well used. It also has a wordpress import plugin which I hoped to use.

The first problem I ran into is that the 'nikola plugin' command didn't work. I couldn't see that I had done anything to break it, and some poking around let me see that this was a bug in 8.3.0 (which is what the current fedora rpm version is), but was fixed in 8.3.1 (released early this year). There is already a PR to update it:

https://src.fedoraproject.org/rpms/python-nikola/pull-request/6

So, I built the new version locally and plugin was back in business.

The wordpress_import plugin worked somewhat, but there were a few issues I hit there too. It tracebacked if I passed '--one-file' to use the new one file format (instead of a content and a metadata file). I looked at it a bit, but couldn't figure out where it was failing. I did have to tweak a bit of the wordpress export, mostly for mistakes that wordpress ignored, like posts with multiple of the same tag on them, etc.

I looked a bit at comments. I have 81 comments on all my posts over the last 21 years, but there are none in the last 5 years. There is a 'static_comments' plugin that lets you serve the old static comments, which looked promising, but it was not very clear to me how to insert it into the theme I picked to use ('hack'). The doc example has jinja2 examples, and just a 'adjust accordingly for mako templates'. I didn't want to spend a bunch of time learning mako templates, so for now I am just going to drop the comments. If I get time or someone wants to help me get static_comments working, let me know.

Builds are quite fast and it's an easy rsync to my main server. Hopefully this all will make me blog a bit more now.

This post will likely cuse aggregators (like fedoraplanet.org) to see all my recent posts again. Sorry about that.