I've seen a number of people ask things like: "Foo is in EPEL-6, why isn't it in EPEL-7?" so I thought I would share a detailed answer: In Fedora, all new releases are branched off rawhide, the ever moving forward release. This means every Fedora release starts out with a version of every package that was in rawhide at the time. This works great for Fedora, but EPEL is just a collection of add on packages, not a full distribution. New EPEL branches/releases appear only after the new RHEL version beta appears. Between RHEL6 and RHEL7 this was something like 3 years. A LOT can happen in 3 years in the open source world. Packages that were in EPEL-6, may make no sense in EPEL-7 (the base things they depend on may no longer exist in the base os, the maintainer may not want to commit to maintaining them there, they may have been renamed, etc). For these reasons, packages are added to EPEL-7 only on actual request of a maintainer. There was no mass branching of all packages in EPEL-6. They are different things and have different package needs. Some examples: EPEL-6 has Xfce 4.8 in it. By the time EPEL-7 was created, Xfce 4.10 was out. Packages like 'Terminal' were re-named xfce4-terminal, new base library packages were added, etc. If we mass branched from EPEL-6 there would have been a bunch of packages we would immediately have to just remove because they could never build or exist. If you want to get some package in EPEL-7 thats not currently there, file a bug on the package and ask that it be added to EPEL-7. If the maintainer doesn't respond in a week and you are yourself a package maintainer, you can request the branch. If you aren't you can ask on the epel-devel list for anyone interested in maintaining it for you. Hope that clarifies things somewhat...

I happened to see somewhere (I cannot actually recall now) someone mention the 'grid-it' line of organizers and picked up one for my laptop bag. It's basically a hard board with lots and lots and lots of straps of various sizes all over one side and a zippered pocket on the other. You put cables, batteries, whatever through the straps and it keeps them in place. It actually is a great help for me as the fedora laptop bag I use has just one big main compartment, so things mill around in there if they aren't connected to something. As with many things I find that it usually looks great at the beginning of a trip with everything in place, but gets less so as the trip goes on and things get tossed back in without being put in their place. For $12 this is a pretty great little device. They of course make them in all sizes and shapes and colors, but they do seem to be pretty handy for organizing that messy laptop bag we all have. ;)

With the release of a new Fedora getting close (Fedora 21 is currently scheduled for release December 9th), I figured this last weekend was a great time to upgrade the 'production' side of my home machines to Fedora 21. I have 2 pretty important machines. One is my production virthost. It also acts as my firewall and router for all my networks (external, wireless, the test network to the test virthost, the internal network, etc). The second is a vm running on that virthost that is my mail hub, vpn end point and other important functions. Of course the first thing to do before any major upgrade is to check backups. I run nightly backups with rdiff-backup to my main virthost, but also I have a encrypted external drive I sync those updates to from time to time, and off site where I can upload them too. I made sure everything was synced up and looked good. The official way to upgrade between Fedora releases is 'fedup'. However, my main virthost is a bit of a weird setup with raid and encryption and such, so I usually just do a yum update on it, and since I am doing that on the virthost, I usually just do the same on the vm as well. This time (like the last few upgrades) everything went very smoothly. No conflicts or issues with the upgrade transactions on either host, took a bit of time to do them, but finished with no issues. Then, a quick reboot and I was up in Fedora 21. Well, mostly. I ran into a issue with my 'internal' bridge. I had not listed the HWADDR for the network interface that should be on the internal bridge, and Fedora 20's NetworkManager was fine with figuring it out, but for some reason Fedora 21's didn't get it added right. I had to set the HWADDR and restart a number of things. I thne ran into a minor issue with my ipv6 tunnel, but got it sorted in short order. So, at least for me, Fedora 21 upgrades were as easy as they have always been.  

ansible is very handy for one-off tasks like gathering quick info or running commands over a group, so I thought I would share some of those today: Filtering on facts:

• ansible -m setup -a 'filter=ansible_distribution_version' -o all (This runs the setup module on all hosts and filters on the ansible_distribution_version, so '20' for Fedora 20, etc. Note also that we are passing -o which means to put all the information on one line, this is very handy for grepping output).
• ansible -m setup -a 'filter=ansible_selinux*' all (Runs the setup module and filters out the selinux info. Note that this is a array with more info, like config and runtime, etc)

Looking for specific information:

• ansible -a 'ps aux | grep puppet' -m shell proxies (This runs the command on all hosts in the proxies group and shows output. Note that here we are using the 'shell' module instead of the default command module. This is because command doesn't understand pipes and such, so you need a full shell module.)
• ansible -m virt -a 'command=freemem' virthost (This uses the virt module and shows free memory on the virthost)

Docs on ansible modules:

• Of course there's google and duckduckgo and the ansible project web pages, but if you want to look up something quickly about an ansible module, use 'ansible-doc' command line. Just 'ansible-doc shell' for example to get all the info about the shell module, it's arguments, etc.

It's super easy to get started using ansible this way. All you have to do is install it on your control host and have ssh connectivity to the hosts you want to run things on.

Some possibly interesting stats for folks about Fedora's Infrastructure (collected this morning by ansible, look for a post about that soon)

• Amusingly, we currently have exactly 400 "instances". Thats hardware servers, vm's, cloud instances, etc. There's more cloud instances that are transitory that aren't covered in this as well.
• 2 instances are down (1's a arm SOC with a bad drive, 1 is a persistent cloud instance we haven't spun up yet).
• OS breakdown: 154 are Fedora 20, 135 are RHEL 6, 105 are RHEL 7, 3 are Fedora 19, 3 are Fedora 21 (Thats right, we have more Fedora instances than RHEL anymore)
• Selinux breakdown: 229 are enforcing, 166 permissive, and 3 are disabled (one of those needs to be, the other 2 should get fixed)
• Hosts still in our old puppet setup: 67 (expect this to drop after our FAD in early december)

Expect RHEL6 numbers to go down to 0 over the next few months. We are already almost at 50% RHEL7, and so far it's been great to work with. Hopefully early next year we can finally retire puppet as well. UPDATE: I meant to say we had more Fedora Instances than any other single OS. (Seeing RHEL6 and RHEL7 as different). Of course total RHEL is still higher than Fedora instances.

A number of folks have been talking about "choice" in open source recently. See the classic post by Adam Jackson: Linux is not about choice and Will Woods followup: two fallacies as well as a google+ post by Greg Kroah-Hartman: here I broadly agree with all of them, but I'd like to talk about a related thing here. Folks are free to choose how to spend their time and energy, free to ask others to spend time or energy (which of course those people are free to ignore or decide not to do) and free to use whatever software they like. Thats all great. I'd like to further posit that the most efficent way to get people to help you with your choices and spend their time and energy on making those choices happen is simple: "Strive to be positive". If someone is working on software you dislike and don't have any intention of using, move on, and instead work on what you DO intend to use or find agrees with your choices. Choosing to spend your time and energy knocking down people doing things (especially if you don't agree with them) just makes you waste your energy on that and slows down the people you are flaming. I'd also posit that it makes your life full of negatives and much less enjoyable for both you and people you are flaming. If you find you are in a discussion where you can't find a way to be positive, it's time to back away and go do something that is.

It may interest some of you to know that we in Fedora Infrastructure use Linux Software Raid (mdraid) on all our servers local disks. Here's a nice list of reasons why:

• Portable: If a machine dies, the disks can actually be swapped into any Linux server and be brought up. If you have hardware raid, you need the same controller, by the same company, with the same firmware, which hopefully works the same way on the new different machine. Good luck with that.
• Consistent: Since we use it everywhere we don't need to worry about weird obscure binary only command lines from N raid vendors that we have to figure out each time we need to do anything with the raid.
• Integrated: When a disk fails, we get email/nagios notification. With hardware raid you can get notification, but you have to setup weird binary only junk and hope that it all works, and hope it lets you know before it's too late.
• Great Performance: True it doesn't have dedicated cache or cpu, but host cpus are so performant these days that mdraid just flies.
• Easy to install: The Fedora/RHEL installer is well integrated to making software raid volumes.
• Easy to discover: You can easily see how many disks a server has, if they are all ok and other info from the command line.
• Hot: You can easily hot remove/add disks and migrate raid levels with no downtime. Hardware raid can sometimes do that with it's binary only command line junk, but often you need to take the machine down into a 'raid bios' thing to do it.
• Upgradable: To get new cool features, all you have to do is upgrade your kernel and userspace tools.
• Completely open: Tons and tons of people use mdadm and the code is open. Most hardware raid vendors have something in the upstream Linux kernel, but it's often behind their development or you need to install binary only tools to really maintain it.

In short, Linux software raid is awesome. :)

Yesterday, Fedora Infrastructure went into "freeze". This happens at the same time in the Fedora release cycle as the development freezes. First, what is a 'freeze'? We mark all our hosts (with ansible variables) as either freezing or non freezing. Hosts are assumed to freeze unless they specifically are marked non freezing. If a host is non freezing, there's no change for it. We could update it's configuration in puppet or ansible, reinstall it, apply updates, reboot it, whatever we normally would like to do with it. However, if the host is frozen, we have to follow a new process to make any changes on it: A patch or description of the change has to be mailed to the fedora infrastructure list and get two people to approve it that are in the sysadmin-main or releng groups. Why do we do this? It's to make sure that we don't make any changes that could possibly impact the upcoming immanent release. For example, if we updated a compose machine and caused it to no longer be able to compose release candidates, the release might be delayed. Or we changed some configuration on the download servers that broke people's ability to download and sync new releases or test composes. We want to make sure everything is very stable to build, compose, test and distribute the upcoming release. Freezes have some other advantages as well, at least for me. They allow us to work on things we sometimes don't have time for, like documentation, or testing new versions of things, or investigating how better to setup existing things, or staging setup, or logs that we don't often get to read, the list goes on.

ok, it's day 18 of the post a blog every day challenge and I am stretching I know, but perhaps some of you will find these feeds/blogs as entertaining as I do:

• popehat - http://www.popehat.com/feed - a blog done by some very articulate lawyers. I like how they manage to explain things so non legal eagles can understand. I also tend to agree with them on free speech issues and the like. Their posts are usually entertaining and enlightening.
• bad astronomy - http://www.slate.com/blogs/bad_astronomy.fulltext.all.10.rss/ - Contrary to it's name, this is GOOD astronomy. Lots of nice discussion about current events in astronomy and explaining how things work. It's really quite fascinating.
• Russel Corkers blog - http://etbe.coker.com.au/feed/ - In particular his 'interesting links for the month' have resulted in some great videos or things I would otherwise have missed.
• Linux Weekly News - http://lwn.net/headlines/newrss - Not just weekly, but daily feed of happenings in the Linux world. If you aren't a subscriber, go and subscribe now!
• National Geographic Found Photos - http://natgeofound.tumblr.com/rss/ - A mix of photos from the National Geographic Society. Sometimes there's some quite striking ones that come accross the feed. Many from long years past, black and white, color, everything.

There's more, but I will leave it at those. Hope some folks find them interesting...

I've reviewed a ton of Fedora packages over the years. I just did some more over this last weekend and thought I would share some tips and thoughts around reviewing packages:

• fedora-review is pretty nice for doing all the tedious checks that can be automated. It fetches the spec and src.rpm for you, builds them, does all the automated tests it can on the package and then leaves you with a nice review template to finish filling out.
• I've always found having a checklist of things to make sure and check is good, even if you notice/check them normally anyhow.
• After firing off fedora-review, I next just go and look at the spec. Not for anything specific, just how the entire thing is setup. Just absorb it and let it flow into my mind. Things like inconsistent macro use just stand out right away, or complex sections that need more scrutiny.
• Then, when I do the specific checks left after fedora-review I can easily think of/see where the problems (if any) are.
• If you are looking for something to review or if it's been submitted, the cached package review page is very handy: http://fedoraproject.org/PackageReviewStatus/

There are some interesting packages in the review queue, do look today and see if you can add some comments to get them further along toward approval.