It's here! Branch day! This is the time when Fedora 20 branches off rawhide and becomes it's own seperate collection of packages and buildroots and install trees. Rawhide itself marches on with only a passing wave for it's branched child. Starting tomorrow, there will be a seperate branched compose (just like the rawhide one). It will compose from all the packages built that day. This will be the process until Alpha change freeze, which is currently scheduled for 2013-09-03. At that point, bodhi will be enabled and all packages built will need updates to pass into updates-testing for Fedora 20, then promotion into the branched compose/base repo when stable. Finally when the release is all go and ready an updates repo will be created and populated with 0 day updates and the branched nightly compose will stop. Package maintainers will want to do a 'git pull' in any package git repos they have to pick up the new f20 branch. Most packages in the Fedora 20 branched repo will be signed with the Fedora 20 key, however there are likely to be a few stragglers until alpha change freeze when updates via bodhi can always be signed before pushed out. Before the alpha change freeze is also an excellent time to fix any broken deps or fails to build from source problems. If you wait until after alpha change freeze you will have to submit updates and wait for them.   http://fedoraproject.org/wiki/Releases/Branched has more detailed information. Meanwhile in rawhide things have been going along pretty well. There's been a cosmetic bug about lightdm not showing any icons in the greeter for a few weeks and since no one reported it, I decided to stop being lazy and do so. ;) I'm sure it will get fixed pretty soon.

Much of this last week many of the members of the Fedora Infrastructure team were at flock. Many kudos to those that organized and ran things, it was a great time. We had a lot of great face to face discussions on things. Our badges application ( http://badges.fedoraproject.org ) went live right before flock and seems to be quite a hit. It's open badges for doing various things in Fedora (powered by the fedmsg message bus to see events to award on). Pretty fun stuff, and it's working very nicely to get people to do things that perhaps they wouldn't have bothered to do before (like set security questions in fas, or answer questions on ask.fedoraproject.org or add tags to packages in fedora tagger). We hashed out a good plan for a Fedora-Oauth setup. Someday this will allow people to get a Oauth token to run a cron job with only specific permissions you give it, or something you can give to another user to perform some specific action. Should be very handy down the road. Of course, infrastructure never sleeps. On tuesday when many of us were getting ready to head home, our puppet master's certificate authority expired (causing all clients to not be able to talk to the master). Smooge luckily was not traveling and managed to reissue everything for us. :) We've started to setup a mailman3/hyperkitty instance in our staging env. There's a bunch more work to do there, but as soon as things are looking stable we are going to roll out a production one and migrate a few lists to it. I'm really looking forward to it, it should be a much better way to consume our often floods of list emails. The rest of this week has been catching up on day to day tasks and resting up from flock. Next week we have the branching off point for Fedora 20 and tons more fun things to roll out.

Day 4 was more hackfests than scheduled talks. First up was a gpg key signing and cacert assurance event. There was about a 50% turn out from those who asked to attend, which isn't too bad considering all the late nights hacking away we have had. ;) Then we had a good conversation with Michael DeHaan about ansible and how we can get the best out of it. Great stuff. I'm really glad he could make it out, even though I missed his ansible talk (I was giving mine at the same time). The afternoon was a lot of infrastructure hacking. We all wrote down items we wanted to work on or discuss and one by one went through them all. Lots and lots of great discussion. I then did a run showing how to create a new host in ansible, but we had to leave the building as that was going. All in all a great flock. Looking forward to the next one.

Day 3 started with a keynote from Dave Crossland about free fonts and making them. Lots of great background on the free fonts world and some ideas on how to fund things to make it better. Next was spot's talk about changing the updates model. He's proposing that many/most users don't like the constant updates stream that we have now, and proposes that they would get a monthly bundle of the non security critical path updates, with an option to update all those and any additional packages. Security updates would still go out as now, and all the existing yum repos and process would still be in place so users could opt into the old model and just update with yum. This plan needs to get buy in from releng and qa, but it's an interesting idea. It could allow Fedora to be more popular with folks who don't like always being reminded of updates and who now just ignore them. Then over to the other building for Patrick's talk about OpenID. He had given much the same talk to some of us in infrastructure a while back over IRC, but it was good to listen again and I hope good information for others. I really didn't understand OpenID and how it worked before Patrick explained it all to us. Great information to have. My talk on the life cycle of a package was up next in the same room. I think it went pretty well, there were a number of questions and some discussion on how to improve workflows with fedmsg and such. After a lunch at a local mexican food place we all headed back for workshops. I wandered around a bit as there were 3 or so workshops I wanted to look in on. The last session of the day had all the present FESCo members on stage in the auditorium talking about some of the more controversial proposals and sessions and things that they thought were good talks and went well. Finally off to dinner at the aquarium. Very nice venue and food. If you are ever in Charleston, do look it up.  

Another all too early start of the day, and we all had gathered in the auditorium to hear Aeva Palecek from lulzbot talk about 3d printing. Most every disaster you can think of happened during this presentation, but happily Aeva and Flock staff were able to overcome every obstical and have an entertaining, informative and great talk. Aeva went over her history with 3d printing software and showed off the process all the way to printing a fedora coin on a reprap. Great stuff. I really hope Fedora is now and will continue to be a great platform for 3d printer enthusiasts. The biggest danger from this talk is that now I want to go and buy/make a 3d printer to play with it myself. :) After the keynote (and a quick check to lulzbot that their printers were currently out of stock, but expected back soon) it was off to the Fedora at yahoo talk. Yahoo has rolled out quite a bit of Fedora for their engineers/employees and it was interesting to hear how they set things up and and what issues they ran into. They love that Fedora is all free software. They so far deployed Fedora 17 and then have recently started moving to Fedora 19 (they skipped Fedora 18 because they couldn't justify that quickly upgrading the just deployed Fedora 17 hosts). The release cycle has been a pain point for them, along with some issues with 32bit software running on 64bit installs, and reliability of upgrades. Next was a talk back in the main auditorium on Fedora Badges. We just enabled badges a few days ago and it's live and running along now. See https://badges.fedoraproject.org for your badges and information. I knew everything that the talk went over, but it was a good introduction to our badges setup and how to help out with it. I really hope this helps make Fedora more fun for a lot of people and it's really great to see it up and running. Next was task automation in qa. This talk went over the problems and limitations in the current autoqa setup and a general plan for a next generation framework. Things are generic and high level enough at this point that it should hopefully allow things to be flexible. It also seems possible that this setup could be handy for other groups in Fedora besides QA. Since it's a method of setting up tasks that run on triggers it's more of a cross between continuous integration and a scheduler. For example, projects like Fedora docs might well find it useful to build on every commit in this system. After lunch was a infrastructure hackfest on OAUTH. We have several places where we would like a system that allowed users to get a token to have permissions to run things from command line or cron jobs or other automated tools. We spend pretty much the rest of the afternoon hacking away at this and came out with a plan for implementing something that should do what we need. It's not really OAUTH, but it's similar. It should allow people to do things around auth tokens and keep us secure. Look for more information as we get things implemented and call for early testers. Dinner was at a place called Mynt. Very good food, drinks and conversation. Now to try and sleep and get ready for tomorrow. I have my talk on the life cycle of a package tomorrow as well as lots of fun sounding talks. One thing thats been amazing for me this year is that there has not been a single timeslot where there is not at least one talk I want to attend. Usually it's been a hard decision between two (or even three) talks. :)

Day one! Early start as always, and folks started assembling in the auditorium for the world famous state of Fedora talk from our lovely Fedora Project Leader. She talked about love and why we love Fedora and how you follow what you love. There's a lot of passion in the room for making things better and moving Fedora forward. After that a quick group snapshot. I'm not sure this is going to turn out that great. We had a LOT of people packed into a small space, but we will see. The first talk I went to was also in the auditorium, a great overview of our Fedmsg bus. If you've not heard much about it, I invite you to look at the slides and video from the talk. We also discussed some great ideas around roping upstream projects and hosting providers into fedmsg collection and tying debian and Fedora together better. ie, sharing more data and seeing things from the other that could be used to help each other out. Next up was "Why Fedora Sucks" by Christoph Wickert. Pretty much all the things he went over I agree with. They are things we should really try and fix and do better on. Perhaps we could get a group of people interested in solving them, it's just going to take some grunt work for the most part. Finally before lunch, there was Dennis Gilmore's talk on the future of the datacenter. Lots of good discussion about arm in the datacenter and areas that it would end up being used it. Lunch was at a BBQ place. Not bad. Looked for ice cream, but it was too far to walk before we needed to be back. After lunch was my HA databases workshop for Fedora Infrastructure. We had several folks who knew a lot about databases and HA and replication, so we had a pretty good discussion about what might meet our needs. Looks like shared iscsi storage might be the best bet. Just have two database server instances that mount a iscsi volume thats shared. This would mean short downtime when switching instances, but might be enough for our needs. Once that wrapped up and after a drink and some cookies! (yum), it was back to the auditorium for Mitr's Fedora Next workshop. Lots of discussion on testing and rawhide and what level of things we want to try and do to create a more consistent base. Just time to drop laptops off at the hotel then we were off to dinner/drinks at The Blind Tiger. I'd have to say the grits and shrimp were better than I expected. :) On to day two tomorrow...

Like day 0 of any conference, it's travel time. ;) Up at 5am to get ready and get out to the airport for my 8am flight. Due to: traffic, then slow shuttle bus, then long long security checkpoint line I just managed to get to my first flight with a few minutes to spare. Then Off to atlanta, where I came in in terminal A, and my next flight was in terminal F. Just managed to make that one with a few minutes to spare too. :) Met up with friends new and old at the hotel, had a nice dinner at a local pub, a bit of hacking back at the hotel and then to bed. Looking forward to flock day 1!

Another week, another rawhide update. :) This week saw the Fedora 20 mass rebuild finish and land. This time it finished in under 3 days, which I think is a record. Monday's rawhide included all those rebuilds, so if you are running rawhide likely you had to update almost every package on your machine (just barring those that failed rebuild). Often after these there's a number of broken deps and things that didn't rebuild properly. This time is no exception. The only one that hit my laptop was the nm-connection-editor needing a rebuild against the new gnome-bluetooth. Everything upgraded pretty smoothly here after fixing that. The ipv6 NetworkManager crash I have been hitting was fixed today by dcbw. :) Much appreciated and quick work. I was also seeing intel video crashes here. I'm still not sure what the issue is there, but rawhide has sna accel enabled, I switched back to the older uxa and things have been very stable again. I'll likely try and poke at this some more after I get back from flock. If anyone needs to do the same, just add a /etc/X11/xorg.conf.d/20-intel.conf with:  

Section "Device" Identifier  "Intel Graphics" Driver      "intel" Option      "AccelMethod"  "uxa" EndSection

Nightly rawhide boot.img's were not composing right after mass rebuild due to broken deps, but they are working starting today. There was a last minute nfs issue on arm that kept it from making a boot.img, but that should be fixed in tomorrow's rawhide and we should have the full set of boot.img's tomorrow hopefully. Finally if you have a package that uses the 'hardened build' macro, and your package uses libtool, do make sure it's actually working for your package (use the hardening-check package available in repos). We are working on a fix to that macro for this case, but it would be good to look and make sure your package is or isn't affected by this. See bug 978949 for more info.

A few notes of interest to folks coming out to flock ( http://flocktofedora.org ) later this week (and even a thing or two for folks not able to make it out): 

• If you find yourself unable to reach resources out on the net that you need, you can use our handy flock proxy: https://fedoraproject.org/wiki/FlockProxy You will want to install 'sshuttle' before traveling so you can use it. It's a simple ssh proxy that should allow you to get to whatever resources you need if you can reach the proxy on port 22 or 80. 
• The #fedora-flock channel on irc.freenode.net is the primary place on irc to help coordinate with others or find any information about flock you are looking for.
• There are also irc channels for each of the rooms at the conference: #fedora-flock-ectr101 thru 212 and #fedora-flock-auditorium. You can use these channels to discuss talks/workshops as they are happening, help provide notes and links for record keeping for those not present and ask questions in a talk you are not able to attend.
• If you don't already have a ride to/from the airport, look at the RideShare page: https://fedoraproject.org/wiki/Flock:Rideshare and add your information. You can then share a cab or car with others arriving or departing at the same time.
• If you would like to help manage video, and/or irc and help speakers with time reminders, please see the volunteers page: https://fedoraproject.org/wiki/Flock/Volunteers we still need more folks for many of the talks!
• Fedora Infrastructure has blocked off a room for working on things monday after the bulk of the conference is over. If anyone has ideas, comments, suggestions, questions for infrastructure and you didn't find one of the infrastructure team before then, please do drop by monday.
• If there's anyone attending (or available on irc even) that knows postgres and replication/ha solutions, please let me know. I'd love to have some more info for our high available db's for infrastructure workshop.

See everyone at flock!

A bit late this week, but was a bit too busy to get this post out the door until now. ;) So, this last week was a busy one. I was out at our main datacenter. We have some excellent on-site folks that do things for us as needed there, but we still try and get out there a few times a year to do things that need scheduled timing or someone who knows our setup to complete. I was out monday, tuesday and wed this time. I got a number of things done out there:

• Got everything accounted for and ready to label things that were missing labels. 
• We pulled 4 old decommissioned servers out and freed up their space/power/net/etc.
• I got memory added to 13 servers. This should allow us to increase density of virtual guests some. Also, I grew the memory on our main updates pushing machine, so we can hopefully have it generate deltarpms for much larger packages (right now there's a limit we had in place due to memory.)
• Helped the gnome.org folks out some by inventorying their rack and getting stuff labeled there. We also hopefully we will get them a new switch sometime soon so they can better organize their network and get to their management interfaces.
• Worked on bringing our new blade server on line. We were missing a fabric switch, but it seems to have been located just after I left, so hopefully it will be on line soon. This is going to replace our existing bladecenter that is builders.
• Checked all machines for attention lights. This let us know about a drive that failed that we forgot about.
• Lots of other misc running around and checking things.

The mass rebuild started friday afternoon, and looks to be done now. It should land probably in rawhide tomorrow (monday). This time the rebuild was much faster. There's a variety of reasons that could be the case: We rebuilt all our buildvm's using virtio (they were not for the f19 mass rebuild), we now have 48 arm builders which allows the mass rebuild to spread out src.rpm building, noarch builds and tagging to a bunch more machines, and I gave our koji db a bunch more memory now that we had it available (so it could keep up better as well). In any case 2 days for a mass rebuild is great! There's a number of failed builds this time, so do go and clean them up as soon as you can. Next week is flock! Looking forward to it.