• Need to drop our staging git branch in puppet and merge any staging machine changes into the master branch
• Need to work on making out applications more ‘container’ like. ie, reduce the things they depend on and make it easier to spin up an env they run in
• Check out cloud tech and filesystems to see if any will meet our needs
• See if we can automate spinning up a application env with all needed integration testing machines
• Get two factor auth working for sudo for sysadmin-main and deploy. Continue adding types of auth and services once it’s solid
• Talk to the Board about our torrent seeder.
• Make sure the folks who want to work on reviving smolt have all the info and help they need
• Figure out how best to deploy the new community/tagger applications to production
• Find the time to add the nightly live compose to the rawhide/branched scripts so it happens in a totally automated way
• Finish migrating everything to RHEL6

If there’s something I told you I would help you with/do at Fudcon and you don’t see it above, please let me know and I will add it to the list.

Some of us Fedora Infrastructure folks got together to hash out any issues and projects we are concerned about in the coming weeks, months or year. We may have some big challenges ahead with ARM possibly coming in as a primary arch, which will stretch our storage and networks and building code. We are going to take a harder look at torrents and see if they are still worth providing to the community. The new community packages and tagging stuff needs to move into production. Lots of high bandwith talks with involved folks in all those areas.

After a nice little lunch, and a few more discussions it was time to head out to the airport. Matt Domsch was able to take 3 of us to the airport, which was nice. At the airport, there was time to sign gpg keys and mail them out from the keysigning the other day as well as make another pass of the cacert.org assurances that I needed to enter into the system. Hopefully flights home will go smoothly.

Overall it’s been a great fudcon. Lots of things seemed to get discussed and worked out. I heard very little where people were asking for something that they were not willing to champion themselves, which I think is excellent.

Thanks to all the folks who worked so hard to put on a great event and those who attended.

We had some intro and then pitching of barcamp sessions. Lots of nice sounding sessions. Then a short State of Fedora talk from our fearless Fedora Project leader.

There were tons of sessions (as you can see from looking at the wiki: Fudcon wiki page for saturday. I went to the session on making Fedora packaging easier. This was a talk to unveil a few new cool applications that can help folks with maintaining packages.

Pretty slick stuff: Community development packages and community dev tagger. It’s fast and could be very very handy for users.

Lunch was a delightful burger and beefy miracle. Good stuff.

After lunch I met up with some IRC folks to discuss some issues and ideas for improvements there. We also did some cacert.org assurances with a few folks (there’s another session just for that at 4pm, but since we had a few minutes we went ahead and did it to save time later).

Then on to the session on redoing comps and package organization given by Bill Notthingham. There are still lots of details to work out, but comps will split into distro groups (xfce, kde, base, webserver, etc) and categories (web browsers, editors, etc). distro groups will stay in the same place, but categories still need to be hashed out. Some info can come from the package tagging application I saw this morning and some from package maintainers. More info to come, but overall a good plan.

Next was the talk and show and tell on the rasberry pi arm box. It’s a pretty cute little device. I might have to pick up one after they are available.

Then, a lovely cacert event. I assured about 7 people and was in turn assured by 5 more. Then a gpg keysigning which had quite a few folks at it, but we managed to organize and run through checking fingerprints and id’s in short order.

Only a few minutes to drop things off at the hotel and then it was time for FUDPub over at the student center. Lots of food and fun. Finally some more talk back in the hotel where amusingly a wedding party was gathered.

tomorrow will be the last day and travel back home.

Fixing Staging in Fedora Infrastructure.

Some background: Currently we have a some ‘staging’ machines that are supposed to be copies of production instances that we can use to test and integrate new things with. We have a seperate git branch in puppet that handles the staging instances, which seems neat, but turns out to be an annoyance in several ways.

There was a lot of information and debate on what production, dev, staging, integration, or the meant. How we could setup puppet. If we could on demand make a staging instance or a subset of those. How process should work. How we could go from here.

We came up with a plan of attack and some things to consider:

• Drop the ‘staging’ git branch. Everything is in the same git repo. ie, all machines are production.
• Try and make our apps more able to be ‘containers’. Ie, reduce dependence on other parts of Infrastructure so things can be tested in containers easier.
• Look at ways to build containers or integration staging machines on the fly as needed.

After a quick lunch (man the wind was nasty to/from lunch), it was time for a 2 factor auth session.

We’ve been talking about finishing off yubikey as a true two factor authentication method in fedora infrastructure. We had a lot of good input here and hashed out a plan here too:

Short term:

• Fork linotp’s pam module to a new project. This would be just the pam module, and we would enhance it to require a valid ssl cert from the server it’s talking to before sending it anything, prompting for pin and pass and other enhancements.
• First target is going to be sudo for all sysadmin-main users.
• Create a CGI that the pam module can talk to and send auth info to and return ok, bad, broken
• CGI will likely run on fas servers so it can talk to fas and yubikey
• Some quick and dirty way to query pin

Longer term:

• FAS changes to store and set/reset pin
• ADD google auth or OATH to the CGI
• Increase parts thats are covered/where 2 factor is required

All in all some great sessions today. I think we have some lovely plans in fedora infrastructure, ready to dig in and get working in the coming days and weeks.

Looking forward to a great fudcon!

I’ve signed up to give/facilitate 3 workshops/hackfest sessions:

• Fedora Infrastructure: Revamping our staging server setup, then implemeting it. This will be hopefully friday and sunday. We can try and hash out a plan on friday and look at starting implementation or at least timeline for implementing things on sunday.
• Fedora Infrastructure: 2 factor authentication brainstorming. This will hopefully be saturday. I’d like to try and get full 2 factor auth setup for yubikeys and/or a clear plan to do so. Ideally we will setup a framework where we can also optionally use other 2factor setups (google authenticator, etc). I will also have some yubikeys with me to distribute
• Fedora Infrastructure: Getting started/apprentices/Getting involved. This will be a intro session for folks that would like to get involved with infrastructure. We can answer questions, try and find out what would be a good fit for people starting out and work on our apprentice program to try and make it easier for people to join us.

In addition to those, Toshio is going to be running some sessions on Python programming and software releases using our infrastructure as the example. Should be great stuff.

Of course as always in addition to the talks and hackfests, there’s a lively “hallway” track: A chance to meet up with folks and discuss things in a high bandwidth way. I hope to see lots of folks I know, and lots of new faces as well. Happy to discuss any fedora related issues or items there.

If you’re unable to attend this Fudcon, do follow along in #fedora-fudcon on irc.freenode.net. There should be pointers there to whats happening, whats being discussed and allow for remote input. Hopefully we will have irc transcribers in various rooms as well.

This is a nifty little plugin that lets you do codereviews. You can set users to have CODEREVIEWDEV (developer who can submit reviews) and CODEREVIEWMGR (can approve reviews). You can enable the plugin in the admin section of your trac instance, and should then see a “Peer Review” button.

Enjoy.

EDITED TO ADD(2012-01-06): We ran into a problem with the plugin that I missed in my testing of it. ;( I’ve disabled it for now… if anyone has ideas on fixing http://trac-hacks.org/ticket/7034 please chime in there.

• trac-workflowadmin-plugin: This plugin lets you change the ticket workflow on your project. You can setup what states tickets must pass through and in what order. It even gives you a nice little graph of workflow.
• trac-navadd-plugin: This isn’t too visible yet, but I am going to hopefully use it to add handy ‘register’ and ‘forgot password’ links to all projects. This should help with end user ticket reporting.
• trac-watchlist-plugin: This allows any logged in user to watch tickets or wiki pages without having to CC to the ticket or otherwise distrub it
• trac-tocmacro-plugin: This was enabled in the old fedorahosted, but wasn’t packaged up correctly and has been now. This allows you to use a TOC macro to provide table of contents on wiki pages.

As always, these plugins are available in the EPEL repos (currently in testing) for all to use, improve and share. If you need some specific plugin or macro in fedorahosted, just let us know!

We’ve moved fedorahosted to RHEL6 and a newer set of hardware. Everything should be faster and newer. You will probibly most notice the upgrade to trac (from 0.10 to 0.12), but mailman, scms and all other software should also be newer. The backend is now a cluster of two drbd using nodes. We are going to look at decentralizing things much further in the coming year.

Speaking of trac, there’s a ton of new items in the new version along with some new plugins we have enabled:

• The MasterTickets plugin is now installed, so you can have tickets block or be blocked by other tickets, and get a nice graph of the dependency tree.
• Multiple repository support. You can now have your project list and show multiple source repos, using the same or different SCM’s
• A great deal of configuration is available to trac admins on notification emails and permissions.
• Unicode and i18n support is vastly improved
• With a bit of adjustment, the PrivateTickets plugin works fine with the new trac. This allows you to have tickets only some groups can read/see if needed.
• Configurable work flows for tickets. You can set things to go through specific states before being fixed.

Mailing lists (lists.fedorahosted.org) have some small but nice improvements, like properly handling bounce messages from inactive google.com accounts and better utf8 handling.

Our review board instance is still running along, and should be a bit faster now.

We have great plans for 2012 for fedorahosted. If you’re interested in helping us out, take a gander at the Fedora Infrastructure Getting Started page and jump in and join us.

See the fedorahosted.org FAQ for more information about Fedora Hosted and how to host your next open source project there.

First, a bit of background on the setup: We have a storage device that exports raw storage as iSCSI. This is then consumed/used by our main nfs server (nfs01). It’s using the device with lvm2, and has a ext4 filesystem on it. It’s around 12TB in size. This data is then exported to various other machines to use, including builders, kojipkgs squid frontend for packages, koji hubs and release engineering boxes that push updates. We also have a backup nfs server (bnfs01) that has it’s own separate storage with a backup copy of the primary data.

On the morning of December 8th, the connection between the iSCSI backend and nfs01 had a hiccup. It retried current in progress writes, and then decided it could resume ok and kept going. The filesystem had “Errors behavior: Continue” set so it kept going (although no actual fs errors were logged, so that may not matter). Shortly after this, NFS locks started failing and builds were getting I/O errors. A lvm snapshot was made and a fsck run on that snapshot, which completed after around 2 hours. A fsck was then run on the actual volume itself, but that took around 8 hours and showed a great deal more corruption than the snapshot had. In order to get things into a good state, we then did a rsync of the snapshot off to our backup storage (which took around 8 hours), and merged that snapshot back as the master fs on the volume (which took around 30min to complete). Then, a reboot and we were back up ok, but there were some small number of builds that were made after the issue started. We purged them from the koji database and re-ran them with the current/valid/repaired filesystem. After that builders were brought back on-line and queued up builds processed and things were back to normal.

So, some lessons/ideas here, in no particular order:

• 12TB means most anything you decide to do will take a while to finish. On the plus side that gives you lots of time to think about the next step.
• We should change the default on-error behavior to at least ‘read-only’ on that volume. Then errors would at least stop further corruption, and best prevent the need for a lengthy fsck. It’s not entirely clear if the iSCSI errors would have made the fs hit error condition or not however.
• We could do better about more regular backups of this data. A daily snapshot and rsync off of that snapshot to backup storage could save us time in the event of another backup sync being needed. We would also then have the snapshot to go back to if needed
• Down the road some of the cluster fses might be a good thing to investigate and transition to. If we can spread the backend storage around and have enough nodes, the failure of any one might not be as much impact.
• Perhaps we could add monitoring for iscsi errors and note and react to them quicker
• lvm and it’s snapshots and ability to merge a snapshot back in as primary really helped us out here.

Feel free to chime in with other thoughts or ideas. Hopefully it will be quite some time since we have another outage like this one.