Skip to main content

Bits from mid late jan 2025

Scrye into the crystal ball

Hello again everyone! Three weeks in a row now. Go me. I'm not really sure what to name these posts, especially given that 'early/mid/late' does not work when there's 4 weeks in a month. I could just put the dates in the title, but that seems difficult to see what is going on. Perhaps I just need to pick out some highlights.

mass rebuild completed

The f42 mass rebuild was completed. This time submitting from the mass rebuild script seemed slower than normal. Possibly due to the item scaper item below. However, things mostly worked out. Instead of submitting everything and koji churning through the backlog, submissions were usually just handled as they came in, leaving koji with really no backlog.

A.I. Scrapers

The A.I. scrapers continue to pound away at things. I mean, I'm assuming thats what they are as they crawl all the content they can get. They are pretty hard to block due to using massive piles of IP's, different user agent strings every time and random access of content. This sort of thing is hitting lots of open source communities too. I think short time we probibly need to look at deploying mod_qos at least, so we can prioritize users/things that need access and let the scrapers get much lower levels of request answering. No idea if that will really stop them though.

Anoying outage on thursday

We had a very anoying outage on thursday. Our rabbitmq cluster started not processing things sometimes and just in general not working right. Restarting it and then a bunch of consumers that connect to it finally got things working again, but it's unclear what the cause was, and cleaning up a bunch of builds and such that were not processed due to missing messages was not at all fun. Currently our rabbitmq cluster is on rhel8 and an older version, we probibly want to move it to 9 and start using the version made by the CentOS messaging sig. If newer/older can talk to each other in a cluster it should be pretty easy. If they can't it will need some downtime.

Mass update/reboot next week

As we do from time to time, there's going to be a mass update/reboot cycle next week. Monday will be staging, then tuesday will be a bunch of things that don't cause any outages (where we can take things out and update/reboot and put them back in), and finally wed will have the actual outage.

riscv secondary koji hub progress

The riscv hub is installed and mostly configured, still need to sort out a few packages it needs to complete the ansible playbook. Next I need to test web interface, then install some x86 builders (that will do newrepos) and a compose host (for composes). Then will come the hard work of importing content / setting up actual riscv builders (which will be external for now until we can find hardware to use locally).

Datacenter move

We announced that we are going to be moving from our main datacenter (IAD2 in ashburn, va) to a new one (RDU3 in Raleigh, NC). It's going to be a ton of work (and lots has already happened, been spending a lot of time on it already), but when it's done we should be in a much better place. More space, better machines, and hopefully things like ipv6 support (that we have wanted for many years).

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/113891021464841039

Bits from mid jan 2025

Scrye into the crystal ball

Hello again, here's some longer form doings and thoughts from from mid january 2025 in and around fedora.

rawhide repodata change

Rawhide repodata has moved over to the createrepo_c default: zstd. This shouldn't affect any dnf use, or fedora createrepo_c use, but if you are using EL8/EL9, createrepo_c there currently doesn't understand zstd. There's a issue to add that in a minor release: https://issues.redhat.com/browse/RHEL-67689 and in the mean time if you are a EL8/9 user there's a copr: https://copr.fedorainfracloud.org/coprs/amatej/createrepo_c/

dist-repos retention

On the dist-repos space issue I mentioned last week, it turns out that the expectation on dist-repos is that you would sync them somewhere you wanted to serve them from, not just serve from koji. So our use case was misalined a bit from upstream here. However, they did adjust to keep latest repos. Should be in a upcoming koji release. Thanks koji folks!

fun email infrastructure mixup between ipa and postfix

There was a pretty curious email issue that came up this last week. fedoraproject contributors (that is folks with an account that is in at least one non system group) are setup with an email alias of theiraccountlogin@fedoraproject.org. This is just a very simple alias. We accept the email and forward it to their real email address. There's no mailbox here or authentication or anything, just a simple alias. We got an alert that disk space was getting low on our mail hub, so I took a look and found that users who were not contributors were getting emails to theiraccountlogin@fedoraproject.org delivered locally to /var/spool/mail on the hub! When we switched away from fas2 to our current IPA based setup, no one realized that sssd/ipa enumerates all users, even if they do not have access to actually login or do anything. There are good reasons for this, but somehow I at least didn't realize that it worked that way. So, since all users 'existed' there, and postfix's default for local users is:

proxy:unix:passwd.byname $alias_maps

It correctly looks them up and thinks they are local. Simply changing this to just be $alias_maps fixes the issue. There wasn't a bug here in postfix or ipa, they were just doing things as they expected. The issue was our misunderstanding how these things interacted.

f42 mass rebuild underway

The mass rebuild for f42 started (all be it a bit later than planned due to a issue getting golang to work on i686 when compiled with gcc 15). This time it seems like our submitting builds is much slower than before. In past years, pretty much everything was submitted in a few days, then koji chewed on the backlog. This time koji is easily keeping up with the submissions and we are only in the 'p's after 3.5 days. Oh well, hopefully we will finish mondayish, which is in line with past mass rebuilds.

forgejo kickoff meeting/discussions

There was a kickoff meeting about forgejo in fedora infra. I'm looking forward to this, but I have so many things going on I am not sure how much work I can do on the deployment. Lots of good ideas/plans discussed. I think it's going to be not a super lot of work to stand up a staging instance, but I think integrating with all our workflows will take a lot more effort. Time will tell.

riscv secondary koji hub

I did finally submit my work in progress PR for riscv-koji hub: https://pagure.io/fedora-infra/ansible/pull-request/2435 Hopefully can finish off things and start deploying next week.

bugzilla and needinfo

I asked on mastodon what folks thought about bugzilla needinfo requests and what they meant. There were a number of opinions: https://fosstodon.org/@nirik/113822583672492457 I think in the end it's a thing that people will use for their own use cases and those will sometimes mis match with recipients. Unless we want to try and make some community wide norm or guidelines (but of course even then not everyone will see those).

Xfce-4.20 and wayland testing

News from a while ago: Xfce 4.20 was released and it's got a bunch of wayland support for various things. However, it doesn't have xfwm4 / a compositor of it's own, so by default you get a X session. If you want to play with the wayland sessions and you are running a rawhide instance, you can install: xfce4-session-wayland-session which will by default pull in labwc as your compositor. You can manually modify the session file to use wayfire if you prefer that compositor. See the testing section at https://wiki.xfce.org/releng/wayland_roadmap I tried both out and they did indeed work, but there are still a bunch of rough edges. Still, great progress!

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/113850897869803740

Bits from early jan 2025

Scrye into the crystal ball

Welcome to 2025. It's going to be a super busy year in Fedora land, all kinds of things are going to be worked on and hopefully rollout in 2025. Back from the holiday break things started out somewhat slow (which is great) but already have started to ramp up.

First up I fixed a few issues that were reported during the break:

  • Our infrastructure.fedoraproject.org host wasn't redirecting http to https (as all our other hosts do). Turns out we disabled this many years ago because virt-install couldn't handle https for some reason. I think this was in the RHEL6 days, and we just never went back and fixed it. This did end up breaking some provisioning kickstarts that had http links in them, but easy (and good!) to fix.

  • Some redirects we had for sites were retirecting to just {{ target }} variables, but if that was examplea.fedoraproject.org////exampleb.com it would redirect to examplea.fedoraproject.org.exampleb.com. A totally different domain. Fixed that by making sure there was a / at the end of the redirect. Unfortunately, I also broke the codecs site for basically everyone. ;( It was broken for a few hours, but easy to fix up after I was made aware.

There's been a ton of f42 changes flowing before/during/after the holidays. Lots of exciting things. Hopefully it can all land and work out ok.

I finally started in on work for a riscv-koji secondary arch hub. It would have been really easy, but we dropped all the secondary things from ansible after the last secondary arch hub went primary, so I am having to go though and adjust a lot of things. Making good progress however. Hopefully something to show next week. This secondary hub will allow maintainers to login/do scratch builds and get things one step closer to primary. There's still a long road for it though, as we need actual local builders and proof of keeping up with primary.

Next I cleaned up some space on our netapp (because see below).

  • I archived some iot composes and marked a bunch more for deletion. As soon as I get the ack on those, that should free up around 5TB.

  • I noticed our dist-repos were really pretty large. This turns out to be two issues: First, we were keeping 6 months of them. We did that because we use these for deployments and before if all repos were older than the last change to them, they just would be missing. I am pretty sure now that kojira keeps the latest one, so this is no longer a factor. I set them to 1 week (as default). This should free up many TBs. Secondly, the flatpak apps repos were not using the latest (they were pulling everything). Adjusted that and it should save us some.

  • Finally, I nuked about 35TB of old db backups. There's no reason to keep daily database dumps since 2022. I kept one from each month, but we have never had to go back years for data. In particular the datanommer and koji db's are... gigantic, even compressed. Unfortunately it will be a while before this is actually freed as we have a lengthy snapshot retention policy on the backup volume. Still should help a lot down the road.

With some freed up space, I now could make another iscsi volume and move our ppc64le buildvm's off to it. It took far longer than I would like to admit to get things working (turns out I had logged in on the client before setting up everything on the netapp and it just didn't see the lun until I ran a refresh). I expected there to be a pretty vast speed improvement, and the vm's are indeed a lot more responsive and install much faster, but I am not sure builds are really that much faster. Will need some more poking around to find out. The local disk on those are 7200 rpm spinning sata drives. The iscsi is a ssd backed netapp over 10G network. Unfortunately also we have seen instablity in the hosts in the last week, which is likely a kernel issue since I updated them to 6.12. Hopefully I can get everything sorted out before the f42 mass rebuild, which is next wed.

comment/reply on mastodon: https://fosstodon.org/@nirik/113811216382678546

Apps I use

Scrye into the crystal ball

Happy New year everyone.

I thought I would do a quick post about apps/software that I use. Of course my requirements may be wildly different from your own, but perhaps you will see something here that you might want to also investigate, or in turn leeds you to something you do want to use.

On the server side, I want to use things that are open source and run on Fedora (my main server at home is Fedora of course). I prefer things packaged up and available in the main Fedora repos, but of course that is not practical for everything sadly.

  • nextcloud: nextcloud continues to be a great solution for a lot of things due to it's library of plugins. I use it for files, uploading photos/videos (see below), organizing photos, recipies, calendar, contacts, deck lists (kanban) and more. It's not packaged (anymore) in Fedora, but it's pretty easy to install and upgrade on the fly. I've also been impressed lately that things like files are... files on the server, not some weird db format that are difficult to add/delete/refresh if you need to from the server side. I also do phone backups to nextcloud (see below)

  • postfix/dovecot/opendkim/opendmark/spamassassin/sqlgrey/saslauthd for email. This seutp has worked for years and years and just works fine.

  • znc for irc bouncer. I still connect to a number of IRC networks (although I am usually much more active on matrix).

  • matrix-synapse for matrix server. I'm just using the packaged Fedora version and it works fine. Someday I will probibly move it to use one of the setups that has more bells and whistles but for now it's fine.

  • postgres for database server for everything that needs one. I was running also a mariadb instance, but I moved the last things off it over the holidays and didn't setup one on the new server.

  • miniflux for RSS. ( https://miniflux.app/ ). miniflux isn't packaged directly in Fedora, but the upstream folks provide a repo and rpms and they work just fine. This allows me to manage/read RSS feeds via web interface, or (usually) just manage/read via an app (see below).

  • I was running calibre in headless server mode to serve ebooks but it's really vastly overkill to do that and it pulls in about 233 rpms which I otherwise do not need. So, I switched to a simple OPDS app: cops ( https://github.com/mikespub-org/seblucas-cops ) (well, a fork of it that works with recent php). It seems to do the job just fine. I can manage things with calibre on my laptop and rsync them to the server where cops serves them to my phone (or whatever reader I want).

On the Linux client side (my laptop):

  • firefox as main web browser, occasionally having to use chrome or chromium. Mozilla hasn't been making good choices of late, and I really hope they find their way again, but I really really dislike the idea of using web browsers or engines that are made by super large companies for their own gain. I'll probibly stick with firefox until it becomes untenable. Perhaps servo will be ready by then?

  • hexchat for IRC. Been using it for ages and ages.

  • discord (flatpak app). I have some friends that I have known since college and we have a discord server to chit-chat on, so reluctantly I connect to that to keep in touch.

  • necko for matrix. I use the Fedora packaged version and it's the best of the matrix clients most of the time. It still has flaws of course, but day to day it's the best one for me.

  • newsflash (flatpak) for rss client reading. Newsflash used to be packaged in Fedora, but it just became too difficult, so I use the flatpak now. newflash looks great, works well, and is a very pleasent reading experence. It connects easily to miniflux (see above). Doing things that way allows me to read things on the laptop (Newsflash), web (miniflux) or phone (see below) and keep all of them in sync so I don't re-read things.

  • foot for terminals. I've used... a lot of terminals over the years and almost all of them are just fine if needed, but I've really taken to foot over the last few years. It's super quick, it allows me to have italic fonts (call me crazy, but I find oblique/italic to be easier on my eyes and vte based terminals no longer allow that). I do use xfce4-terminal on Xfce, because of course foot is a wayland only terminal.

  • calibre for ebook / library management. There's a number of new ebook library managers up and coming, but calibre is still far ahead of all of them in my opinion.

Finally on my phone (a google pixel 8a) running https://grapheneos.org/ Probibly too many apps to note, but ones that interact with my laptop and main server:

  • firefox here as well. The mobile version has gotten much better over recent years. Sadly, installed from google play store.

  • antennapod for podcasts.

  • tusky ( https://tusky.app/ ) for mastadon reading/posting.

  • librera ( https://tusky.app/ ) for ebook reading. While making this post I happened to see I had it installed from google play store, I think because in the distant past there was something that didn't work in the f-droid version, but no longer. I switched it over to the f-droid version and it's working fine. Its a nice reader, it hooks into odps on the server just fine. I've ready so many books with this thing.

  • KISS launcher. I ran accross this a while back somewhere and it's still my main application launcher. It provides your apps as a searchable list, with the ones you use the most at the bottom so you can easily find them. This is so much nicer than paging though a bunch of virtual desktops looking for some icon. Highly recommended.

  • Nextcloud android apps: nextcloud, deck, memories, cookbook all integrate nicely with nextcloud. Deck is nice for shopping lists or organizing things. The main nextcloud app lets you sync things back and forth and setup autosync. I have it syncing my photos and movies up right after I take them. Memories then lets you look at and organzie. cookbook is handy for using in the kitchen when you want to follow a recipe.

  • Element X for matrix. Works fine, has the new 'fast' sync, which seems to work reasonably well.

  • Paseo for step counting. I installed this last year and it's been nice be able to see that I need to get up and walk around more. It hooks into the android steps stuff so walking on the eliptical or the like will show up as steps even though you didn't go anywhere.

  • A bunch of junk non free apps I need for various things sadly, but it's nice to have the option of installing from play store if you absoluetely need some app and can't avoid it.

Im sure there's more things I didn't remmeber or see while looking here but hopefully the list inspires you somehow.

I'm still figuring out comments to blog posts, but if you want to reply, I will be making a mastodon post pointing to this blog post you can reply to: https://fosstodon.org/@nirik/113766175760197132