It's been a few weeks since I posted a rawhide week in review. First I was very busy and there wasn't all that much going on in rawhide, then I was on vacation. ;) Now I'm back and there's a fun little bit of rawhide this week to talk about: Some of you may have noted there was no rawhide composes pushed out saturday or sunday, and the one on monday was later than normal. The reason for this is simple: An update to the createrepo package was made on friday and used in the rawhide composes after that that broke the compose. Luckily the createrepo maintainers pushed out a fix monday and things were back in business. I re-ran monday's compose after that fix landed (and thus the reason it was later than normal). So, how do you go about debugging these compose issues? Well, the first place to look is the logs of the compose, you can find them on line at (for example): http://kojipkgs.fedoraproject.org/mash/rawhide-20131020/logs/ where you can see various log files. The mash.log in that case had the traceback and pointed to the recent createrepo changes. Additionally, detailed output of the cron job that makes rawhide (and branched) goes to the releng list. You can look there for the full output if (for example) something prevents it from even getting as far as running things in the chroot. A short aside here about how rawhide composes currently: It's a cron job, it starts up, makes a base chroot with mock and then runs mash in that chroot with actual rawhide packages. It calls out to make arm, i386 and x86_64 boot images to builders, but otherwise all of it composes on a single machine. It sends out broken dep reports and generates an email about the compose for the devel list, then the contents are synced over to the master mirror and everything flows along. Theres some work on-going now to move this compose into koji and make it a full normal koji task. That will be nice when it lands. :)