Kevin's musings

Recently, Fedora infrastructure requested everyone change their password and upload a new SSH public key. We gave folks almost 2 months to make the change. I’d just like to note here the reactions we had, divided into 4 ‘groups’ of users. Of course some people were between two of these groups or some probably had slightly different reactions, but this is what my reading of the feedback leads me to:

1. Group one sees the announcement, skims it, goes and changes their password, generates and uploads a new ssh key, goes back to what they were doing.
2. Group two sees the announcement, reads it, reads the links off it. Adjusts their firewall, checks their backups, re-enables selinux or applies updates, then changes their password and uploads a new key
3. Group three sees the announcement. Complains that their private ssh key is safe and always has been, and they know all about passwords and ssh and encryption and this change is unneeded.
4. Group four doesn’t even see the announcement. They are no longer involved, too busy to read it, or just don’t care

Group one spends about 5 minutes. The advantage to Fedora Infrastructure isn’t great here, but they do have a new password that meets the guidelines and a new ssh key in case they were careless with the old one. Of course they didn’t learn much, so they could be careless with the new one as well.

Group two are the very people we are trying to reach most, and here’s the most advantage to this plan. These people will learn how to improve their security some, how better to handle their ssh private keys and hopefully prevent compromise on their personal machines. They may spend a good deal of time following the links and learning about best practices, but it’s all time well spent.

Group three are the vocal minority. While they already know best practices and keep their ssh private keys safe, they don’t realize we have any way of telling them apart from group 4 (below). Time spent here is large for both them and Infrastructure folks, but advantage is low because it amounts to “I know I am fine and shouldn’t have to do this” vs “We have no way of knowing that”. There are likely a less vocal subset of these folks that show up in Group one (just do it and grumble and move on).

Group four is another group where there is good advantage for infrastructure. These are folks that are gone, don’t pay attention to their Fedora work, or are too busy to spend a few minutes on it. Packages with these folks as maintainers would be better off being orphaned and reassigned to people who use and care for them. Sysadmin groups with these folks are better off not having someone who they think are involved, but really doesn’t have time to be.

So, in the end I think there is still good advantage to us having done this. I hope the folks in group two are large and learn from our documents and best practices. I hope the people in group three realize that this isn’t just about them, it’s about the community, and I hope pruning the people from group four helps improve the health and activity of our community.

Finally, as a side note, the deadline was last night, but we are still assessing exactly how we want to mark inactive folks who haven’t yet changed their password and uploaded a new key. You still have time to go do it now.

Just a friendly reminder: If you are a Fedora account system account holder, and haven’t changed your password and uploaded a new ssh key since we announced the mandatory change, you best do so NOW. The deadline is 2011-11-30 (only 2 days away).

If you don’t, you may no longer have access to groups you currently do (like packager, or sysadmin or ambassador).

Go take a few minutes, read the announcement and security information linked to it, and change your password and upload a new ssh public key.

If you aren’t a Fedora contributor, the information linked in our announcement is still a great read and may just help you be more secure on your machines.

Looking at a bug report today, I saw some pretty poor “bugside manners” on the part of a few people (including the upstream developer). There’s tons of examples out there of good bug report interactions and poor ones. I’d like to urge everyone to take a minute and think before posting that next bug comment.

Good bugside:

• Ask for facts. Program output, versions, exact behaviors
• Provide info if asked for it by others
• Try and see the issue from the reporters view
• If you find yourself unable to say something nice or move the bug along, perhaps you should refrain from saying anything at all

Bad bugside:

• Don’t pile on unrelated bugs. Open a new bug for a new issue.
• Avoid talking about “big picture”/design/philosophy. Those should go to mailing lists or the like, bug trackers are to track and fix bugs
• If you don’t have things to add about the specific bug at hand, don’t post

I’m sure there’s other good/bad rules. Anyone run accross any other common good or bad bugside manners they want to share?

Your blade centers are pretty nifty… however:

a) Isn’t there any way the remote console could just be vnc directly instead of requiring a java applet? Because the java applet has various issues and there are some very nice direct vnc clients available. It would be less hassle for you even.

and

b) Could you pretty please have servers pop up a list of boot options/items right away, even if it takes 10minutes to finish booting to any of them? Having to spend 10minutes staring at a java applet waiting for the 10 seconds I can press “1″ to get to the SMS menu is sure a drag on my time management. Especially when it doesn’t take the keypress and I have to just do it again.

Thanks!

This morning, we announced that we are requiring Fedora contributors to change the passwords and upload new ssh public keys by the end of next month. There’s no breach or cause for alarm, we just thought it would be a good time with all the high profile hacking happening out there for everyone to go look at their security practices and create new keys and passwords (see the announcement for full list).

Please do go and change your password and create a new ssh key at your convenience (but before 2011-11-30).

I’m sorry that the ssh key requirement has caused stress for some contributors, but realize we are not singling anyone out here, there’s good reasons to ask for this change now when it’s not urgent or triggered by outside events. Just a few of them:

• Allow you to revisit your security process and policies and read about best practices
• Allow you to see how to make changes and what machines and places you need to make them in the event you were making a more hurried change
• Allow you to setup a separate ssh key for Fedora matters. This separates out some risk, at the cost of another passphrase and possibly hitting ssh server limits (most allow you to try only 6 keys).

While many of the more savvy Fedora contributors already know these things, and have good practices, we hope that everyone will learn something from this or at least not let it inconvenience them for too long.

Recently, the filesystem on my trusty Dell D820 laptop started having some problems. It’s using btrfs and installed back when btrfs first showed up, so I am no too surprised that something happend with it. Side note: Josef (btrfs maintainer extraordinaire) has been absolutely great in helping track down and fix it, see https://bugzilla.redhat.com/show_bug.cgi?id=726814 for the nitty-gritty.

So, I decided at least for now to move to another laptop here (thinkpad t510). First order of business was to add memory and replace the 1600×900 screen with a nice 1920×1080 one. This went very smoothly. The lcd replacement was easy and made a gigantic difference. My D820 has a 1920×1200 screen in it, and I really didn’t want to move to too much smaller resolution. The higher resolution screen on the 510 also looks MUCH nicer: brighter colors, crisper and all around more pleasant.

Since the filesystem corruption was preventing me from just mass copying everything off the old machine, I just synced over a small list of things I needed: ssh keys, keepassxdb, xchat logs/settings, midori browser history.

This gave me a great chance to (re)setup my mail and it’s filtering. I’m a long time claws-mail user, and adding filters is pretty easy. So easy that I had added tons of them over the years. Seeing my full flood of email to my main mbox gave me a chance to look at things and ask: do I read this? Should I unsubscribe from this list? Do I need to save logwatch emails from my home machines? Do I care when something is auto-discarded from mailman? After just a few days I pretty much had everything filtering away, and in much better shape than I had things before. So, this might actually end up being something I try and do once a year or so: Drop all my filters and re-check my incoming emails for what really matters.

F16 prerelease is running great on the Thinkpad. Everything works out of the box. In fact aside from an anoying glibc resolver bug (which I am running a patched glibc for), this release is looking quite stable and boring so far.

Rebooting machines is a interesting study in the varied opinions of the Linux community. On one end, there are folks who will use ksplice or simply avoid rebooting for any reason short of a hardware failure. On the other you have desktop users who reboot their machines daily. I’m somewhere in the middle: For servers if there is a security update to the kernel or glibc that applies, the server should be rebooted. For my laptop, I usually reboot when there’s a reason (I want to test something related to the boot process, there’s a security update, etc).

Rebooting servers regularly (and it seems doing so for security updates accomplishes this) has several other advantages:

• You can schedule your rebooting. Sometimes power cycling or rebooting a machine puts some stress on the hardware, if it fails, you are able to call for service, etc. If it happens at 2am on sunday morning when you have 9 to 5 business day coverage, you are in much worse shape
• Even with configuration management there are times when things are added to a server, but not set to start on boot, or need some config change to start properly. Better to fix those in a maint window than to not have them come up after an unattended reboot
• You can find out a lot about your servers and how they interrelate, and where the points of failure are by rebooting them
• Do you recall how to get to that serial console / IPMI / pdu / kvm for that server? Scheduling a reboot is a great time to make sure your console access works and shows the boot process.

Fedora Infrastructure has about 150 or so machine instances to manage currently. Until recently the mass reboot process was pretty much just scheduling a block of time and powering through rebooting things. Often we would go over our window as that is a lot of machines to reboot and confirm are back up nicely. So, I worked on changing our process for mass reboots. Now, all hosts are put in 3 different buckets:

• The “C” group. These are machines that only infrastructure folks will notice are down or are machines where there are redundant resources, so they can be rebooted anytime as long as failover or dns changes are made first so no live traffic is still hitting them.
• The “B” group. These are machines associated with Fedora contributors and package manintainers. End users won’t notice these being down, but contributors/package maintainers will. These reboots need to be scheduled, but since there are few machines in this group, the outage is small.
• The “A” group. These are machines that end users may notice being down or slow to respond. Database servers or mailing list hubs would be in this group. Again the number is very small, so outages would be very short

Over time, we are working on moving all the hosts in “A” and “B” groups into “C”. Which would leave us being able to reboot things as time permits with no need for any scheduled outage, but at least with the above setup, outages are much shorter and less frantic. The last set of reboots we did, we used the new method and planning, and I think it went much more smoothly. We also discovered some additional points of failure:

• An nameserver reboot caused some internal machines to stop processing, which allowed us to revamp our nameserver setup and make sure all machines were setup to failover properly to another nameserver.
• A nfs server reboot in the “B” group affected some web servers in the “A” group. This allowed us to revisit why there’s a dependent NFS mount on those web servers.

True to the Fedora way, the details are available at: Mass_Upgrade_Infrastructure_SOP.

If you have not had a chance yet to go and sign in to your fedora account and sign the Fedora Project Contributor Agreement please do so soon! The deadline is approaching this thursday. After that point you will be removed from any groups that require “cla_done”, for example the packager group, or any of the various sysadmin groups or access to fedorapeople.

You can of course be re-added to any groups later, but this could be a annoying and time consuming process. Take a few minutes to look it over today!

EDITED TO ADD: We decided to give another week for a variety of reasons. So, the new cut off is 2011-06-23. We are going to inform group admins and post a list of packages that will potentially be without owners soon.

The release of Fedora 15 is just around the corner (this tuesday, 2011-05-24)!

As always, we will be having a on-line release party over in #fedora-social on irc.freenode.net.

Please join us in celebrating the release…

It’s that time again: Fedora election season. The Fedora Board and The Fedora Engineering Steering Committeeare open to nominations until tomorrow.

I have no desire to serve on the Fedora Board, but I’ve been on FESCo since it was the Fedora Extras Steering Committee. So, thinking about running again brought up many thoughts:

• I think I do a good job keeping FESCo on track and productive
• I think I do good keeping FESCo open and operating in the light
• I do think new blood, energy, ideas would be welcome.
• I for the most part enjoy working on FESCo matters
• I have perhaps less time than I have in the past, what with the new job

Given all that, I have decided to run again, but possibly see if I can’t pawn off the chairpersonship on someone else if I am re-elected. I will let the voters decide if they would like to see me continue or would prefer some new folks joining in.

As always, I am happy to talk with folks about anything. Feel free to comment here, shoot me an email or catch me on IRC. I’m sure I will also be participating in the town halls, so feel free to save up questions for that as well.

If you have time, energy and new ideas for FESCo, do consider running.

The Board race this time should be interesting. There’s a number of new faces and ideas and folks I know and have worked with. I look forward to talking with Board candidates as we move forward in the elections. Good luck to everyone who is willing to step up and make things better!