This will likely be the last of my week in rawhide blog posts this year. First some thoughts about recent holiday breakage in rawhide, and then some retrospective. Right as everyone left for the holiday's this year, rawhide suffered from a number of bugs. Firstly a selinux issue causing lots of things to fail with execstack/execmem errors. This would cause things to not come up right on boot, or at the very least make it so you couldn't login. Running in permissive mode until this bug is fixed is sadly the workaround at least at the moment. With the release of Fedora 20, a number of QA folks have moved up to rawhide and run into a number of issues in the gnome stack in particular. Adam Williamson has updated the rawhidewatch blog with several of them. We really need to try harder as a distro to be more careful with builds. yes, particularly in rawhide. Any of the following are things we should be working to detect and avoid:

• ABI/API bumps of packages where the maintainer doesn't announce them in advance. The policy is for the maintainer to announce these in advance and (wherever possible) rebuild affected packages or at least get an ack from maintainers of those that they will rebuild them. There's just no excuse for pushing a build with no announcement and wandering off.
• Pushing a critical build or something that requires a bunch of rebuilds you can't do on a weekend or holiday. If your package is minor sure, work whenever you can, but pushing things like a libpng rebuild on a friday afternoon, or pushing a new gnome build before leaving on holiday for 2 weeks when you know it's not ready is not ideal.

Things we should be doing more of:

• Eating our own rawhide-food. If we had more folks running gnome in rawhide they would have seen some of these issues as they happened, not after the fact. In general, more coverage would be great. I think we need to make sure in the fedora.next world to have people actively using the rawhide version of the 'products' day to day.
• Asking for help. If you have a package and it has broken deps and you don't know how to fix it, please ask on the devel list. There's people around who are able and willing to help if they know there's an issue going on that isn't solved.
• Announcing/Coordinating. Tell folks when there's an ABI bump, file bugs when you see broken things.
• Automation. As always we should look at putting some checks in place for rawhide to help prevent some of the issues we run into, it's just a matter of deciding on the tooling and finding time to implement it.

So, looking back on the year there have been a number of hiccups and bugs and issues, but overall rawhide has been very usable day to day for me for the most part. I really hope I can help more folks start using it day to day as well to increase our testing and fix things much faster before they go to any 'stable' Fedora release. With the fedora.next stuff upcoming I think we will probibly see more folks move to rawhide simply because they want the newest things and f21 isn't going to be along as fast as it would have been in the past. Here's looking forward to another exciting year of rawhide. :)