Once again, things have been real busy and I haven't kept up with sending my rawhide notes, but I will try and do better! Astute rawhide trail travelers would have noted that there was a distinct lack of updates to rawhide recently. There were composes up to December 4th, and then nothing until late yesterday (December 11th). Here's a list of the reasons all those composes failed: * As part of a big infrastructure move (Moving every server in our main datacenter to a new area) we also applied updates all around. The December 5th rawhide failed because the new koji package had a bug with python3 and 'runroot' plugin we use for koji. This has since been fixed upstream and a fixed version applied to infrastructure machines. * Another few runs failed because we were moving the signing infrastructure and didn't have everything back up and working. Recent changes to the rawhide compose require signing because it signs ostrees commits that are made as part of the compose (unlikely packages where they are signed before the compose). * A firefox update came out that did not build on armv7. This was ok until a new hunspell update came out, a bunch of things built against it and the existing firefox had a broken dependency on the old version. Ideally we could have untagged the hunspell update and all the things that had already built against it, but that was a pile of packages to look for. In addition, the firefox update was a security one (of course). So, I filed a bug on it, excluded armv7 (just until the compile is fixed) and did a new build. This was only part of the rawhide fix however, as the Xfce armv7 image is release blocking and it no longer had a firefox. So, I dropped firefox from it for now (until the compile is fixed). * While dealing with the firefox thing, I merged some more kickstart PR's folks had submitted. Sadly, one of them had a syntax error in KDE (which is release blocking), causing another compose to fail. So, we are back on track for now. Hopefully firefox will get fixed soon as we can revert hacks. I'll keep watching things over the holidays.