A bit late, but here's some thoughts on rawhide last week. ;) One rough spot was the boost rebuild. Boost has a cycle similar to Fedora, so a new major version comes along about every 6 months or so and requires rebuilding all the packages that use it. In Fedora thats around 170ish packages or so. I communicated with the Boost maintainers and we decided the best way forward was to just commit the new Boost and rebuild everything in one day and then fix up the parts that broke. That was the plan. Sadly, we ran into a few hiccups. The new Boost was built and tagged in, but the newrepo job that should have added it to the buildroot got stalled, and everyone just assumed it was in there. Then, the rebuild started of the other packages and about 1/2 way through someone noticed that the new Boost was not actually getting built against. ;( So, all the rest of the builds were canceled, new Boost was added in for sure this time, and about 1/2 of the packages needed to be rebuilt again. ;( So, the lesson here is to make sure something is in the build root before rebuilding against it. Other than that, not much happened, rawhide has been running along fine here. The mass rebuild for f19 started in on wed, so that will be keeping the build system busy for a few days. Look for a massive update when the rebuilt packages are tagged into rawhide (basically it will be a yum update * on your rawhide machines).

I don't want to single out anyone, but over the last few weeks I've either been struck by a unconnected group of random incidents, or am seeing a disturbing trend. In either case I want to call it out and hope folks try hard to reverse the trend or reduce the number of incidents. (I think I have seen 'default to private' about 6 or 8 times in the last few weeks) To borrow a phrase from Red Hat, "default to open" is something I aspire to always do, especially in the Fedora community. This means simply that you default to sharing everything in the open by default and only after specific consideration do you keep something not open. This means:

• Default to sending to a list when you are replying to something asked there. 
• Default to asking in a irc channel instead of sending someone a private message
• Default to filing a ticket when you wish something done instead of sending private email.
• Default to replying on a bugzilla bug instead of sending private emails.

The point is you should not think "oh, should I send this publicly?" You should Default to that, and only specific, rare things that need to be private can be private. This has any number of advantages, both for you and the community:

• You can get an answer faster from a group of people than just one person. 
• You can get peer review of that answer from the group, where you wouldn't by just talking to one person.
• You can increase the knowledge of the community, instead of hiding it away. Perhaps 5 other people had the same question you just asked and got answered.
• You can start discussions and end up with a much better solution than the one person you were trying to talk to would have given you.
• Your question or comment could lead others in related questions or comments.

In the Fedora world, we really should default to open. So, if you send me something privately that I think should be public, don't be surprised if I ask you to reshare with the community. PS: I think the origin of the phrase "Default to open" is this blog post by Chris Grams: http://darkmattermatters.com/2009/07/16/red-hat-culture-tip-default-to-open/ but I'd love to hear further on the origin of the phrase.

Another week, another rawhide review. ;) This one is painfully boring however. There were some more abi bumps and some broken deps, but nothing very critical. The icu update got sorted out and everything rebuilt for it. I did a more methodical rebuild of broken deps where I scripted it to clone, bump spec and fire off a scratch build. Anything that completed scractch build I looked at more closely and did a real rebuild on those things that made sense. The rest that failed I posted to the devel list. Perhaps that will help folks fix them. Down the road we need to get to a place where this is all scripted for us. The rawhide mass rebuild is coming up at the end of this week most likely, so rawhide users will need to be ready to re-install everything. ;)

Several folks said they find my rawhide posts useful, so I am going to keep doing them for now. They do at least give an idea of what sorts of problems you might run into in rawhide and how to easily debug them. So, this week there was more broken deps than normal. We had:

• lbgraph5
• gcc 4.8.0
• libgsoap
• libseccomp
• icu

All land at some point. We really need to get better at coordinating these. If you (you being the maintainer of the library thats changing ABI) don't know all the packages that depend on your library, you should. Announcing the change in advance is great, but we should really strive to always rebuild things the SAME DAY as the ABI breaking package is built. The icu update deserves a special paragraph. This library has a charming feature allowing you to 'disable-renaming' meaning it produces a ABI compatible library over so name bumps or anything else for a subset of the consumers of the library. I'm really not sure how this is supposed to work, but in the end it doesn't. The linux dynamic library SO naming works fine. Rawhide kernel maintainers have been trying to isolate what debug options in rawhide are causing some people to have slowdowns that make running that kernel unusable. They started out with 0 debug options then have been slowly adding more day to day. I still need to test the most recent one, but overall so far they have continued to be usable. Hopefully we will identify the small options that cause the most problems and can disable those. There was one serious rpm break that (almost) landed in rawhide. It would have resulted in your rpmdb getting wiped out. Luckily it was detected quickly and we stopped the rawhide compose for that day to prevent it going out to repos. So, lesson there is two fold: have backups, and test your packages as soon as you can so we can stop something really broken going out.

Another week with rawhide, another few issues:

• A gtk3 update caused nm-applet in systray to stop working right. The applet runs fine, but right or left clicking on it doesn't provide a menu. Filed upstream as: https://bugzilla.gnome.org/show_bug.cgi?id=691911 Hopefully this is fixed soon. 
• libnl3 had an unannounced ABI break. It turns out that they didn't think the update was changing versions and it got reverted, but by then everything had rebuilt against it. The maintainer answered my emails on this issue fine, but I'd really really really prefer that discussion about these kinds of things takes place on the list. We are a community, we should act like one. ;)
• There were a few packages with broken deps over a poppler update. This was announced however, so it wasn't too bad.
• mdadm updated to a version that needs a newer dracut. That dracut is in f18-updates, but was never built for rawhide. Hopefully this is fixed later today. When f18 updates appear thats your note as a maintainer that you MUST start also building any fixes for rawhide. Rawhide only inherits from f18 base/gold/general availability. It does not inherit from f18-updates.

Not too bad overall, and my laptop was just fine for Fudcon. So, not sure if I will keep doing these blog posts, or just stop now. Do folks find them useful?

Another hackfest day, starting way too early, at least this time I got a bit more sleep. ;) Infrastructure folks gathered in the same room as yesterday to work on various projects. Some of the things we got done/discussed:

• Wrote up a hotfix SOP with a new slightly improved process to allow us to much more easily see the diff of the actual fix. See: http://infrastructure.fedoraproject.org/infra/docs/hotfix.txt
• Talked about revamping our website build with Kévin Raymond and how we handle transations. He had a great plan to make things build much faster.
• Fixed up the hairpinning in phx2 again after 4 machines had problems with the change yesterday.
• Fixed up a new pull translations script for the websites.
• Talked about migrating from puppet to ansible and how best to do that. It's going to be a gradual road, but there's some things we need to do first.
• Decided that things that listen to fedmsg message bus should treat the messages as informational, and always validate/verify the data with the orig application before acting on it.
• Some folks worked on moving openid out of FAS to it's own standalone service.
• Learned about WAT?!? https://www.destroyallsoftware.com/talks/wat
• Again many more things that I can't recall. ;)

Also had a meeting with FESCo folks and feature wrangler to dicsuss features. We made a lot of progress and are going to hopefully have a proposal ready for the next meeting. It's incremental improvements, but thats the best way to move things the right direction.

Day 2 is hackfests. Things started as they do after far too little sleep. A quick breakfast and bus ride over to campus and things started in. Had a short, but hopefully productive talk with Tim Flink from QA about autoqa and their needs around clouds. I think many of their needs will be completely satisfied with the same setup we have been using for copr's. (Basically fire up an instance, run ansible to set it up, run test case, pull results, destroy, repeat). Then we had a great discussion with Matt Domsch our mirrormanager keeper. We are going to try and get Mirrormanager 1.4 out to staging this weekend, and look at splitting it out of our app servers to it's own smaller hosts. Oddly while looking at this, we noticed that mirrormanager hits over the last year have been steadily increasing. It's about twice what it was a year ago. Next, Spot came and we discussed plans for badges and app's. Lots of cool stuff there down the road. It will be a lot of work, but it will be worth it. Lunch was sandwitches again, but they were not bad. ;) After lunch we started diving into implenting things:

• Removed the host alias for admin.fedoraproject.org inside our phx2 datacenter. In the past we had to do that to avoid a hairpinning issue, but it meant the single proxy there was a single point of failure.
• Shut down the no longer being used insight hosts. Will clean them out fully in a few days. 4 virtual servers.
• Some discussion with PPC secondary arch folks about various setups.
• Discussion about fasClient pushing instead of pull.
• Talking about getting external fedmsg producers.
• Talked about projects using github for their projects instead of fedorahosted. It's up to the project. We of course will only use projects that are completely free running into our infrastructure, but what they in turn use for their infrastructure is up to them.
• Probibly about 10 other things that I forgot that other people involved should blog about. :)

I also did some assurances for folks for a CACert.org meetup. Back to the hotel for a very short time and then time for fudpub. :)

Day one pre-dawned early (got some alerts at around 4am after going to sleep at 1am), then got up around 7am local time (6am my time). ;( Breakfast at the hotel was there...nothing really very exciting, but they did have coffee at least. The bus over to the event was amusing... it's a "party bus" with funky lights and all kinds of wacky printing on it. They did the job of transporting people over to campus nicely thought. The normal fudcon setup fired off then: A into to barcamps, everyone pitching a session streaming by the stage, voting on barcamp talks and then state of Fedora talk by our fearless leader. Everything went very smoothly and the setup was great. This time more than in the past even there were slots that had 2 or even 3 talks that I wanted to go to. Alas, I could only go to one at a time. :) I picked the infra demos talk first. There were a fair number of folks there, although most of them were involved in infra and probibly knew about the applications demoed. Still, hopefully we got a bit more exposure for the things we have been working on the last year. Do take a look at http://apps.fedoraproject.org and click around. :) The next slot I took the hallway track and talked to various folks in the hall and in the auditorium. Also picked up lunch (sandwitches). Then lightning talks. Some of these were pretty amazingly cool. :) Next up was the EPEL session. I wanted to try and discuss a few issues we haven't been able to really reach consensus on. There was a good spirted discussion. Nothing was really concluded, but I do have more info to ponder on and propose some solutions for the epel-devel list post fudcon. I then took the hallway track again, and finished out with a "Spins need fixing" and "Fedora Formulas" talk. Lots of good talk there about formulas, spins not so much. We will really have to come up with something better for f19 asap. Now it's time for pizza and cupcakes at the hotel, and then I think a short night for me... Looking forward to hackfests tomorrow, I think we can get a few things done and lots more good discussion.

Thursday was a traveling day. Drove from denver to lawerence for fudcon. It was a uneventful trip, there is just not much on kanas aside from rolling plains and long empty freeways. :) Managed to catch up on planet money and this american life some on the way. Got into the fudcon hotel around 4pm. Just in time to gather in the lobby and then wander off for dinner and beers at freestate brewing company. Then back to the hotel to catch up with folks and chat until late. I didn't get to sleep until something like 1am here, and then had to get up for the first day of fudcon at 7ish. It's going to be a long, but hopefully very fun day. :)

yesterday was Fedora 18 release day. ;) Grab it while it's hot... http://fedoraproject.org/get-fedora-all Overall release day went very smoothly. We did run into 2 minor issues:

• The instrepo in mirrormanager for fedup (the new upgrade tool) wasn't working in the early parts of the morning after release. Matt Domsch (our wonderful mirrormanager maintainer) got in and fixed it up, and it was working before midday. 
• There was a docs issue with pushing various guides, but that got fixed up thanks to the power of git.

Our master mirrors have been handling the load just fine, I think partly because we had so large a pool of other mirrors with content at release time. Anyhow, if you haven't had a chance, go check out Fedora 18 today. :)