Just a tad late on this post this week. ;) The lvm2 issue I ran into last week with /home not mounting on boot was quickly answered by lvm2 maintainers. I had a workaround quickly and then a real fix landed in a few days. The issue also affected f19, so the fix landed there too. Good work. :) See bug 952782 for more info. I continue to see a strange systemd bug where when I update systemd packages, it forgets about the inhibits from xfce4-power-manager. The net effect is that if I do a systemd package update, closing my laptop lid starts suspending (which I really really dislike). See bug 928962 to follow along on that one. Otherwise again, pretty smooth sailing.

Another week of rawhide rolling along and only one really interesting bug hit: Some recent change, likely in lvm2, has caused my encrypted /home lv to not be activated right on boot. Root and swap come up fine and the encrypted volume is unlocked, but for some reason /home doesn't get activated. As a work around you can add a timeout to /etc/fstab: "defaults,x-systemd.device-timeout=30" and then when the mount times out, you login to the emergency shell and 'vgchange -ay' and 'mount /home' and then control-d to continue boot. I've filed a bug on this, so hopefully it will get fixed up soon. Otherwise pretty boring stream of updates this last week... and rawhide keeps rolling along.

Oh look, a this week in rawhide post on tuesday again. :) A pretty boring week in rawhide again. webkitgtk was updated to 2.0.0 after a bit of tweaking on banshee. With 2.0.0, webkitgtk switches to using gstreamer1 packages instead of the old gstreamer-0.10 versions. Banshee was linking against webkitgtk and old gstreamer (which would result in crashes). Luckily there were patches porting it to newer gstreamer and folks were able to get those all working. There's still a number of broken deps in the rawhide and branched reports. I sure wish maintainers would be quicker about dealing with them. Nothing else really leaps to mind, it's been pretty smooth sailing...

Getting a bit lax about sending these out of late. I will try and get back on a tuesday schedule again next week. :) So, looking back on the last twoish weeks there were a few interesting issues: The perl package rebuilt, but something changed in rpm so the new builds started saying that they provided "perl(Carp)". Sadly, there's a seperate package that actually provides that, but perl gets pulled in first, so lots of builds that had a BuildRequires on perl(Carp) failed to build right. Adding an explicit "BuildRequires: perl-Carp" would work around it, but it was a bit anoying. The filtering on the perl package did get fixed up and is back to normal now, so if you were a maintainer that added that work around, please clean it up now. The espeak package ran into a long time rpm issue. They changed a directory to a link, and rpm doesn't handle that well, so it resulted in updates having problems. As always the work around for this is to put some scripting in the package to deal with the directory/simlink before installing the new version. I ran another attempt at fixing broken deps in rawhide last saturday. I did manage to fix up some packages, but there's some that just need maintainer attention or dropping. Please do look to your packages in rawhide and f19. The broken deps reports are just sad. In related f19 branched news, it moved over to using bodhi yesterday as part of the alpha change freeze. Do remember to submit updates now for f19 builds (after you have built for rawhide as normal). Otherwise so far, rawhide is proving to be just fine for my day to day laptop for the most part.

Normally I write up my weekly rawhide post on tuesday, but this week has been crazy, so I only just now got to it. :) First, the branching off of f19 from rawhide happened last week. If you're a package maintainer, make sure you build for rawhide and then (if you want it in f19) the f19 branch. You don't need to create updates for f19 branched until the Alpha Change deadline (2013-04-02), your build just goes out the next day just like rawhide does. There's no longer any inheritence between them, so please remember to build both, don't just build for f19. The first composes of rawhide and branched didn't go to well unfortunately. Due to a mistake rawhide composed, but had 0 packages in it, which meant that there was a good deal of mirror churn. We immediately did another compose, but it wasn't quick enough to save some mirrors the extra bandwith. We are putting some checks in place to make sure it doesn't sync out if this happens again. I ran into an anoying bug with claws-mail's fancy plugin. They had redone how it's preferences worked, and it was opening in my browser any movie or images that were not simple images. Very anoying, but quickly fixed by upstream and pushed out to rawhide. Branched f19 had a bad time with the release name: "Schrödinger’s Cat". Fedora's default grub2 setup helpfully puts the release name in /etc/grub2.cfg, but this one had a ' in it, which messed up it's quoting and left systems non booting. The quick fix was to boot from rescue media and remove the '. A more perm fix has now been pushed out to make the quote a unicode one. Rawhide missed out on that entirely since it's release name is "Rawhide". Yesterday we had our Test Compose 1 for F19 branched, and it's still pretty rough, but things are getting fixed up.

With the recent release of obnam 1.4, I thought it might be a good time to look at it again. I had first looked into it when it was a bit before it's 1.0 release, and at the time it had a number of issues that made me decide to not use it and keep my existing rdiff-backup setup running here at home. Happily, with 1.4, almost all of those issues are cleared up. Some background: (which you can possibly get better at the upstream link above). obnam is a backup system that uses sftp (and optionally gpg encryption). It stores things in efficent 'chunks' on the backup store system. If multiple clients backup to the same store, it only keeps 1 of each 'chunk'. The first backup is full/all files, and after that it's deltas. You can run backups as many times as you like, and 'forget' old generations if you like (by default everything is kept forever). With previous versions you could backup multiple machines to the same backup store, and get dedup, but you could only backup clients one at a time. With 1.4, there's locking so you can indeed have multiple clients backing up to the same store at the same time. They might wait for locks at times, but they won't cause any problems or have to wait for the entire backup to finish. Compression seems to work fine (--compress-with=deflate), although it's not entirely clear how much space is saved by the compression. The pre 1.0 versions were painfully slow on rerunning (after the inital full backup was done). At least with some smaller test directories here, 1.4 really seems to do a much better job of that. This is one of the main reasons I didn't start using obnam, it would have taken more than a day to do a daily backup. We will see how it does with larger backup sets over the next few weeks here. The one issue (which I admit may be a personal one) left to me is that obnam has it's own format. The nice thing about rdiff-backup and some other setups is that you can simply copy files from the backup and look at them they are all there. With obnam you need to use it's command line (or there's a FUSE filesystem being worked on) to see whats in your backups, etc. If the backup store got corrupted, bad things would happen. On the plus side the upstream author really takes pains to make sure things are robust on the backend, so this might be nothing to really worry too much about. So, I'm going to try and do some full backups and see how obnam works out over the next week or two. Look for a followup then. If all goes well it could well replace rdiff-backups here and possibly help with Fedora infrastructure backups down the road.

Another week of rawhide, this time with fun on both ends. Last wed, there was some regretable breakage in rawhide. First there was the release of the 1.14.0 xorg X server. This required rebuild of all the various X driver packages, and at least one of the driver packages didn't finish in time for the rawhide compose. Next was a new claws-mail release that pulled in a bunch of the plugins that had been in an extra plugins package into the main package. The old plugins package was fixed up for this change, but only after rawhide had started composing. Both of these issues were easy to skip by simply waiting a day or using skip-broken to update. The rest of the week was pretty dull until today. I reworked the Rawhide wiki page from the ground up and asked for feedback. I got a number of small typo fixes, but no screams, so I put it in place today. Do look it over and see if there's anything that can be tweaked. I think it's MUCH better than the old one was. I've also floated a proposal to make a Rawhide tracking bug. This would let us track serious rawhide problems and get them attention. I'll wait a bit more for feedback before making that tracker bug, but it seems overall people are happy with the idea. Finally, today was Branching day. What will become Fedora 19 has branched off Rawhide. It's now it's own branch in packages git repos, will have it's own daily compose, it's own Bugzilla component and so forth. I think I am going to keep following Rawhide here, but I know many folks will follow the Branch to help stablize and test things for the upcoming release. One thing for maintainers to note: there is no inheritence between Rawhide and Branched this time. Please build all your packages first for Rawhide, and then for Branched. If you are too busy for this, feel free to drop me a line and I will be happy to help out or find you a co-maintainer who will.

Another week of rawhide that hasn't been terribly eventfull. I'm hard pressed to really note any particular problems this last week. Perhaps I missed them, or my memory is going. ;) I have started to work on cleaning up our rawhide wiki page, which was pretty dire. It still needs more help, in particular the section about installing/using rawhide.

A few fun times in rawhide this last week. The mass rebuild finished and was tagged in last tuesday. Failed to build packages are listed at: http://ausil.fedorapeople.org/f19-failures.html if you are a maintainer of anything listed there, please fix it. Thats just under 500 packages out of about 13,000 or so. Not too bad. There was one chicken and egg sort of problem when things were tagged in: The libmpc package was built late last year, but the new version had an ABI increase and was incompatible, so it was untagged. Then, the mass rebuild rebuilt that version and it suddenly landed again. Since gcc required the old version of libmpc, the buildroot was broken and nothing could build. So, how do you fix this kind of thing? easy: untag the new libmpc so the old one is used. That gets you the buildroot working again so you can build new things. Then, make a libmpc that includes a compat-libmpc with the old version in it. Build that and then you can rebuild gcc against the new one. Fun times. Another minor issue I ran into was that midori (my default browser) suddenly stopped being able to talk https to google.com (and likely other sites). This turns out to have been a issue with the new gnutls that landed. I went back to the old version of the glib-networking package that was compiled against the old gnutls until the bug was fixed. Just a few days. Finally, there was a bit of fun with the new kernels this week... someone was overzelous about trying to get usb to save power and ended up causing all external usb ports to be powered off all the time. (ie, not usable). Luckily the only usb device I use very often is my trusty yubikey. I was able to switch to google authenticator until the patch fixing that landed. I could also have just booted an older kernel until the fix landed. Finally, this last weekend I tried fixing as many of the broken deps on the rawhide report as I could. I managed to fix up 8 or so, and hopefully inspired other people to fix up more. Some of the packages in there have been broken a while, but the report is getting much smaller now. ;(

Another week of rawhide without too much to report. The mass rebuild chugged along all this last week and was tagged into rawhide this morning. Tomorrow's rawhide compose should have almost every package rebuilt, with the exceptions of those that failed to build (maintainers please fix: http://ausil.fedorapeople.org/f19-failures.html ) and those that were rebuilt by maintainers after the mass rebuild started. So hold on to your hats for tomorrows rawhide. I did run into one fun issue sunday morning however. My btrfs installed rawhide laptop was unresponsive when I got up. It seems my root subvolume was in a messed up state and I couldn't boot or mount it rw (I could mount ro). I spent a few hours trying various things to recover it and gather info from it, then finally just reinstalled with ext4/lvm2 again. Hopefully my report will be enough to help btrfs folks track down the bug. I'll wait a while before trying it again. Of course when my laptop needed re-installing, I was glad I had good backups. A backup finished just a few hours before the problem happened on it and since I was able to mount my /home and / (ro), I was able to copy off my mail and anything else that changed in there. A quick netinstall, restore of dot files and then restore from backups of larger data and I was back in business. This might be a good time for everyone reading this to check:

1. Do you have backups of your data? If not, perhaps you should set them up now?
2. If you do have backups, when was the last time you checked them and did a test restore to see that they were actually working and backing up what you think they are?
3. RAID is not backups. Duplicate copies of files on the same physical server is not backups. You should really setup backups. :)