Bits from mid late jan 2025
Hello again everyone! Three weeks in a row now. Go me. I'm not really sure what to name these posts, especially given that 'early/mid/late' does not work when there's 4 weeks in a month. I could just put the dates in the title, but that seems difficult to see what is going on. Perhaps I just need to pick out some highlights.
mass rebuild completed
The f42 mass rebuild was completed. This time submitting from the mass rebuild script seemed slower than normal. Possibly due to the item scaper item below. However, things mostly worked out. Instead of submitting everything and koji churning through the backlog, submissions were usually just handled as they came in, leaving koji with really no backlog.
A.I. Scrapers
The A.I. scrapers continue to pound away at things. I mean, I'm assuming thats what they are as they crawl all the content they can get. They are pretty hard to block due to using massive piles of IP's, different user agent strings every time and random access of content. This sort of thing is hitting lots of open source communities too. I think short time we probibly need to look at deploying mod_qos at least, so we can prioritize users/things that need access and let the scrapers get much lower levels of request answering. No idea if that will really stop them though.
Anoying outage on thursday
We had a very anoying outage on thursday. Our rabbitmq cluster started not processing things sometimes and just in general not working right. Restarting it and then a bunch of consumers that connect to it finally got things working again, but it's unclear what the cause was, and cleaning up a bunch of builds and such that were not processed due to missing messages was not at all fun. Currently our rabbitmq cluster is on rhel8 and an older version, we probibly want to move it to 9 and start using the version made by the CentOS messaging sig. If newer/older can talk to each other in a cluster it should be pretty easy. If they can't it will need some downtime.
Mass update/reboot next week
As we do from time to time, there's going to be a mass update/reboot cycle next week. Monday will be staging, then tuesday will be a bunch of things that don't cause any outages (where we can take things out and update/reboot and put them back in), and finally wed will have the actual outage.
riscv secondary koji hub progress
The riscv hub is installed and mostly configured, still need to sort out a few packages it needs to complete the ansible playbook. Next I need to test web interface, then install some x86 builders (that will do newrepos) and a compose host (for composes). Then will come the hard work of importing content / setting up actual riscv builders (which will be external for now until we can find hardware to use locally).
Datacenter move
We announced that we are going to be moving from our main datacenter (IAD2 in ashburn, va) to a new one (RDU3 in Raleigh, NC). It's going to be a ton of work (and lots has already happened, been spending a lot of time on it already), but when it's done we should be in a much better place. More space, better machines, and hopefully things like ipv6 support (that we have wanted for many years).
comments? additions? reactions?
As always, comment on mastodon: https://fosstodon.org/@nirik/113891021464841039