Welcome to 2025. It's going to be a super busy year in Fedora land, all kinds of things are going to be worked on and hopefully rollout in 2025. Back from the holiday break things started out somewhat slow (which is great) but already have started to ramp up.

First up I fixed a few issues that were reported during the break:

• Our infrastructure.fedoraproject.org host wasn't redirecting http to https (as all our other hosts do). Turns out we disabled this many years ago because virt-install couldn't handle https for some reason. I think this was in the RHEL6 days, and we just never went back and fixed it. This did end up breaking some provisioning kickstarts that had http links in them, but easy (and good!) to fix.

• Some redirects we had for sites were retirecting to just {{ target }} variables, but if that was examplea.fedoraproject.org////exampleb.com it would redirect to examplea.fedoraproject.org.exampleb.com. A totally different domain. Fixed that by making sure there was a / at the end of the redirect. Unfortunately, I also broke the codecs site for basically everyone. ;( It was broken for a few hours, but easy to fix up after I was made aware.

There's been a ton of f42 changes flowing before/during/after the holidays. Lots of exciting things. Hopefully it can all land and work out ok.

I finally started in on work for a riscv-koji secondary arch hub. It would have been really easy, but we dropped all the secondary things from ansible after the last secondary arch hub went primary, so I am having to go though and adjust a lot of things. Making good progress however. Hopefully something to show next week. This secondary hub will allow maintainers to login/do scratch builds and get things one step closer to primary. There's still a long road for it though, as we need actual local builders and proof of keeping up with primary.

Next I cleaned up some space on our netapp (because see below).

• I archived some iot composes and marked a bunch more for deletion. As soon as I get the ack on those, that should free up around 5TB.

• I noticed our dist-repos were really pretty large. This turns out to be two issues: First, we were keeping 6 months of them. We did that because we use these for deployments and before if all repos were older than the last change to them, they just would be missing. I am pretty sure now that kojira keeps the latest one, so this is no longer a factor. I set them to 1 week (as default). This should free up many TBs. Secondly, the flatpak apps repos were not using the latest (they were pulling everything). Adjusted that and it should save us some.

• Finally, I nuked about 35TB of old db backups. There's no reason to keep daily database dumps since 2022. I kept one from each month, but we have never had to go back years for data. In particular the datanommer and koji db's are... gigantic, even compressed. Unfortunately it will be a while before this is actually freed as we have a lengthy snapshot retention policy on the backup volume. Still should help a lot down the road.

With some freed up space, I now could make another iscsi volume and move our ppc64le buildvm's off to it. It took far longer than I would like to admit to get things working (turns out I had logged in on the client before setting up everything on the netapp and it just didn't see the lun until I ran a refresh). I expected there to be a pretty vast speed improvement, and the vm's are indeed a lot more responsive and install much faster, but I am not sure builds are really that much faster. Will need some more poking around to find out. The local disk on those are 7200 rpm spinning sata drives. The iscsi is a ssd backed netapp over 10G network. Unfortunately also we have seen instablity in the hosts in the last week, which is likely a kernel issue since I updated them to 6.12. Hopefully I can get everything sorted out before the f42 mass rebuild, which is next wed.

comment/reply on mastodon: https://fosstodon.org/@nirik/113811216382678546