A few weeks ago when I got back from vacation, I noticed (couldn't help but notice) that one of the fans in my main server was making nasty dying ball berring noises. The computer room/library is down the hall from our bedroom, but it was still anoying. So, first step was to take down the machine and figure out which fan it was. This server is a dell sc1430, made back in 2007 (yes, it's a 6 year old server, but it's done great over the years). Happily, it noted that "MEM" fan had gone out of range, so I knew that was the one to replace. This is a fan that sits over the memory sticks and next to the CPU(s). I looked and saw it was a normal 92mm fan. Off to amazon to order new one. A few days pass, new fan arrives. First problem is that the old fan has these really weird rubber connectors instead of anything like screws. I later figured out that you can pull on them and get them to come out, but at the time I figured that they were put in place, so I cut them off. Much grunting and pulling and I got the old fan out and new fan in. Only then did I try and connect it. Bad move. Turns out that the fan I got was a '3 pin' which is pretty normal, but the dell wasn't 3pin at all. ;( So, then I did my second bad move of the enterprise and put things back and went off to order another fan without actually looking closely to see what the connector was. A few more days pass, next new fan arrives. This time I check the connector first off. It's a '4 pin' fan, which was what I thought this machine actually took. Turns out that it's not a '4 pin' but a dell special '5 pin' one. :( Looking for a more specific fan leads me to ebay and a used one from this same model of machine. Final new fan shows up. Take things down, check old and new fans... they are the same model, pins, etc. Plug in new fan and... nothing happens. Plug in old fan... nothing happens. Something seems to have gotten broken on the connector on the MB or something in the firmware. ;( Now, this server does operate without that fan being there, but the bios sees it's not turning and to make up for it cranks every other fan in the machine to 100%. This makes it a good deal louder than the dying fan was. :( Now, when I got this server, I actually got two of them. The idea is that I would run test instances on one and production on the other, and if production died, I could just move drives over. So, I could do that now, but the machines have diverged over time. I would need to move memory, drives, network cards, etc over and then have no test machines. So, I decided to do what any self respecting geek would do: go shopping for new servers. Digging around on the net, my requirements were: cheap. Ability to run lots of virtuals (so mem and cpu). Ability to reuse my existing sata drives. Good linux support (it goes without saying). This all pointed me to dell sc1100's (or really CS24-ty's). These were 'cloud' servers (1u) that dell made a few years ago by the thousands for large cloud computing places. They were designed to have tons of memory and cpu and not much drive space, but there's a model that had space for 4 sata drives. These things are crazy cheap. I picked up 2 from ebay (it had free shipping, no tax) for about $590/ea. They have no drives (great for me), 72GB mem (yes, thats right, 18 4gb sticks), and 2 hex cpu xenons (24 virt 'cpus'). The new servers showed up yesterday and it didn't take me long to set them up and transfer drives and network connections from the old ones. The only hiccups I ran into were: a SSD on my test box, won't quite connect right in the sata bay on the new server. I've ordered an adapter for it and the boot drive (an old 250GB sata) in my test box is throwing smart errors. I ordered a replacement for it. Now I need to clean up my library/server room and give away/sell the old stuff. :)

Another pretty quiet week in rawhide land. One thing that I noticed the other day was a package that had been updated in the Fedora 20 branched release, but not in rawhide. In the past there was some inheritence between a branched release and rawhide. You could build in branched and if there was no specific rawhide build that build would appear in rawhide as well. This was handy for maintainers, but not so great for rawhide users. Often branched releases would keep older versions of libraries so inherited packages wouldn't work, or someone would rebuild a package in rawhide and break the inheritance when the maintainer didn't want, or a package in a branched release would turn out bad and be unpushed from updates-testing, but then rawhide users would still have it installed forever. So, there's no longer any inheritance between them and if you are a package maintainer, please please build in rawhide first, then do your branched build. Recent rawhide kernels seem to have broken support for my touchscreen on the new laptop. :( It cycles in and out and never works long enough to use. I've posted to the kernel list about it, so hopefully someone will figure out the regression. I wasn't using the touchscreen a lot, but it was still nice to have some, and I wanted to play with the tablet mode some more.

It's been a few weeks since I posted a rawhide week in review. First I was very busy and there wasn't all that much going on in rawhide, then I was on vacation. ;) Now I'm back and there's a fun little bit of rawhide this week to talk about: Some of you may have noted there was no rawhide composes pushed out saturday or sunday, and the one on monday was later than normal. The reason for this is simple: An update to the createrepo package was made on friday and used in the rawhide composes after that that broke the compose. Luckily the createrepo maintainers pushed out a fix monday and things were back in business. I re-ran monday's compose after that fix landed (and thus the reason it was later than normal). So, how do you go about debugging these compose issues? Well, the first place to look is the logs of the compose, you can find them on line at (for example): http://kojipkgs.fedoraproject.org/mash/rawhide-20131020/logs/ where you can see various log files. The mash.log in that case had the traceback and pointed to the recent createrepo changes. Additionally, detailed output of the cron job that makes rawhide (and branched) goes to the releng list. You can look there for the full output if (for example) something prevents it from even getting as far as running things in the chroot. A short aside here about how rawhide composes currently: It's a cron job, it starts up, makes a base chroot with mock and then runs mash in that chroot with actual rawhide packages. It calls out to make arm, i386 and x86_64 boot images to builders, but otherwise all of it composes on a single machine. It sends out broken dep reports and generates an email about the compose for the devel list, then the contents are synced over to the master mirror and everything flows along. Theres some work on-going now to move this compose into koji and make it a full normal koji task. That will be nice when it lands. :)

For the last few years I have been looking around as new laptops came on the market, but nothing really interested me enough to consider moving from my trusty thinkpad T510. This fall there is a new crop of laptops that actually have higher resolution screens and are more interesting. First, what am I personally looking for in a laptop? Well, I use my laptop full time (I don't have any desktop setup here at all, just laptop and servers). This allows me to move around the house, go to coffee shops, travel, etc and have pretty much the same env as always. My usaage is: terminals/shells for administrating servers, irc client for talking to people, email client, rss reading and of course web browsing. I don't tend to run virtual machines on my laptop (I have servers for that), but sometimes it's nice to be able to. I really want lots of resolution in my laptop screens. I look at it all day and want things to look crisp and not hurt my eyes as well as get a ton of stuff on the screen at once when I am doing work on a number of systems. I also wanted to join in all the fun with EFI and secure boot, as my T510 is too old to have either of those. The first of the new laptop crop to come up was the Samsung Ativ book 9+. 3200x1800 screen and looked nice. Before it started shipping however, Lenovo announced the yoga 2 pro (same screen) and then dell announced it's XPS 15 would also have a 3200x1800 screen (although in a 15.1" form factor). I have really been wanting to move to a smaller laptop form factor for a while so the dell didn't interest me as much, and then Best Buy came out with a really great deal on the yoga 2 pro and I could just go pick it up from the store, so I went ahead and pulled the trigger and did so this week. ;) Specs:

• i7 haswell cpu
• 8gb ram
• 250gb ssd ( seems to be a: SAMSUNG MZMTD256)
• 13.3" 3200x1800 lcd
• backlight keyboard (you can enable/disable this with a key, it's not auto, also no levels, it's on or off)
• 1 usb 2 and 1 usb 3
• sd card reader (the card doesn't go in flush with the laptop, so it sticks out)
• micro HDMI out

Not too much in the box, just the laptop, power brick (new style lenovo power), some small bundle of useless manuals. I booted up the installed windows 8.1 to make sure everything worked. Side note: windows now seems to require you to have a MSN account? And uses that for your username and password? At least I couldn't find any way to skip that step. I used the windows disk tool to carve off some space from the main drive for a Fedora install, then got to work. The first gotcha is that there is a backlight issue. If you just boot Fedora media, when the i915 module loads you will get a completely black screen. After poking around a bunch, the solution/work around seems to be that you pass 'acpi_backlight=vendor'. There is however an additional confusion that caused me a bunch of wasted time. If you hit alt-f2 when booting you go to the EFI firmware setup screens. If you do that (say to change boot order or to check something) and then try and boot linux, you will get a wacked out screen that doesn't work (even with the acpi_backlight=vendor). Somehow the EFI setup screen messes with the video. ;( So, power off after making any firmware changes or you will be tearing your hair out trying to see why a working linux install no longer works. ;) The next gotcha I ran into was that the wireless didn't want to come up. I found that I could manually bring it up with a 'ifconfig wlp1s0 up' and on further investigation it seems the ideapad_laptop kernel module was causing some kind of issue with rfkill, making NetworkManager not see the interface. For now I have simply blacklisted the ideapad_laptop module, but I suspect that we should get this fixed as that module might be needed to run some of the special keys on the laptop. Other than those two items, everything seems to work very nicely. :) (after poking around, I then wiped the drive completely, and reinstalled Fedora Fresh. Note that there's no windows disks or way to reinstall windows, so if you do this be sure you are sticking with linux full time) Pros/Good/works:

• The screen is a delight (aside the color issue which doesn't bother me much, see below). I can have pretty amazingly small print and it looks sharp and nice.
• The keyboard backlight is nice. Being able to switch it on/off easily is nice.
• It's well made and very light (around 3lbs)
• secure boot/efi works fine.
• It's nice and fast. ;)
• It was a pretty cheap deal (~1200+tax at best buy)
• It seems pretty sturdy and well constructed. The hinge seems to work fine, you can have it in 'laptop mode' and 'tent mode' and 'tablet mode' easily enough. There's not (yet) any support I could see on the linux side to rotate the display, etc in those modes, but scripting and using xrandr should work fine.
• It's very quiet. Even when cpus are pegged it's hard to hear the fans. There's 0 blinkin lights, so it's also sometimes hard to tell if it's doing anything at all. ;)
• webcam seems to work just fine with cheese. Haven't tried google hangouts, but I suspect it will work fine there.
• With the brightness fix above, brightness up/down keys work great. The lowest brightness is backlight off, which might be handy for some low light situations.
• Suspend/resume works just fine.

Cons/Might be Bad:

• Everything is built in. No replacing SSD/mem/battery. This seems to be the trend in ultrabooks to get them as small as they are.
• The sd card reader has the card sticking out. Might be bad if you use the sd card reader a lot. I don't really.
• There's no trackpoint. It's touchpad only. I hate trackpoints, but if you like them this might not be the laptop for you.
• There's a issue being discussed on laptop forums about the color on the display. Yellows don't show very bright, more 'muddy' or brownish. There's speculation that it's due to the way the LCD is setup. I definitely do see this here, but it doesn't bother me all that much. If you are doing photo's/pre-printing/graphics work this might be a show stopper for you. It's also possible it's a firmware thing and they can fix it in an update.
• The network on the laptop is wireless. There is no dedicated wired connection. You could use a usb network, but if you move large volumes of data around and want full gige speeds, this may not be the laptop for you.
• Some of the hardware keys don't do anything (yet): Mute, Vol up / Down, The little arrows thing, disable monitor, airplane mode. Might be related to the ideapad_laptop module.

Mixed/Needs work:

• Will take some getting used to the keyboard. It's not got as much travel as a regular thinkpad keyboard and the control and Fn are in the wrong order. :)
• You can set in the firmware if the top row of keys is always f-keys or always the laptop specific functions (brightness, etc). Not sure which way I will leave it yet.
• I miss the two buttons next to the direction arrows on my t510, I used them to switch desktops, so will have to get used to control arrows again. ;(
• I've not done much on battery yet. The estimate I get when just unplugging it is between 4 and 5 hours. I was hoping haswell would make this much larger, but the jury is still out.
• The laptop has a touchscreen as well, and it works fine under linux. However, it just works as a mouse, you can't easily use it to drag browser windows around or the like. I think thats just a matter of looking around at how to configure it. Not sure how much I would be using the touchscreen, but in tablet mode (with the screen all the way around) it would be nice to be able to drag browser contents, etc.
• The touchpad is overall great, but the 'buttons' on the bottom of it are really hard to use. Luckily you don't have to most of the time: one finger tap: left click, two fingers tap: right click, three fingers tap: middle click. Side scrolling works great, etc.
• Of course with the resolution that high a lot of stuff is very very tiny. I've adjusted Xfce to mostly be fine with it, but there's still wierdness to addess. In midori, content seems to be stuck in the middle, and some pages text gets cut off vertically. I think it's all solveable tho. Also, I still need to try the gtk3 high dpi settings work.
• There's a little windows logo under the screen (built into it). I might just have to put a Fedora sticker over it. ;)

I've switched over the the yoga for my main/full time laptop, so expect more updates about it as I use it more. If there's any questions about anything I didn't answer, feel free to ask in comments or drop me an email or catch me on irc. ;) I am of course running rawhide on it, so not sure I can report much about older versions, but happy to try. Overall I am pretty happy with it at the moment, we will see if that lasts. ;)

Another week along and rawhide keeps on rolling along. A few things to note from the past week: There was a systemd breakage where logind wasn't correctly showing local users as logged in. This of course caused all kinds of problems with local resource usage. Some quick work by Harald Hoyer fixed it up the very next morning. Thanks for the quick fixes. It's of course better to not break things at all, but if you do, quick fixes are very much welcome. In kernel news, the backlight issue I was running into (where acpi backlight would get renamed after monitor power off/on) was also fixed in the more recent kernels. Good to see things progressing there and getting fixed. I also dropped my snippet to switch to uxa accel. I had to do that when things were crashing all the time on my laptop, but it seems to also have been cleared up. Using a 3.12.0-rc1 kernel without debugging is a dream, very nice and fast here. ;) In the last NetworkManager update there's a nasty ipv6 bug again. It causes NM to cycle here and never completely bring up the network. This is bug 1008104. I still need to gather more complete debug info for NM maintainers to track it down. I hope there will be more folks testing with ipv6 enabled networks. We do seem to hit ipv6 bugs somewhat often. Otherwise a pretty normal week in rawhide land.

Over the years we have had many folks show up and ask us to run their web application in Fedora Infrastructure. Very often they don't realize it's not as easy as they first think to just write, deploy and maintain a full time application that our contributors and users will grow to depend on. We have a process for new applications, called the Request for Resources: https://fedoraproject.org/wiki/Request_For_Resources at first glance it looks a bit long and arduous, but their are good reasons for the process and following it for new applications:

• First, does the application do something that helps Fedora or our users or contributors? Are their places that share our values already doing this thing? An example is when we ran wordpress. Sure it's open source and useful, but wordpress.com does that as it's primary mission, why not let them handle it better than we can and we can focus on Fedora.
• We have a bias (and I am happy to admit it is) to python and flask/pyramid framework applications. This is really due to the fact that our active applications developers and others know these frameworks, know how to deploy and manage them, how to look at logs, common problems or issues and so on. So, if you want to us to deploy and maintain an app that is not using that language or framework, you have a very steep road ahead.
• Do you have a group of folks working on your application, or is it just you? If it's just you, we will likely stop right there until there's more people involved. In addition to the problem of when you are away or unable to look at problems, applications developed by one single person also lack peer review and auditing for security issues or the like.
• Is your application packaged and available in EPEL? If not, that will be something you need to do before you even start the RFR process. The reasons for this: a known place to report bugs, known source and builds, signed packages, known place to contribute, known maintainers/co-mainainers, and easy way for us to watch new bugs as they are filed.
• Are you able and willing to port your application to newer releases as they happen? RHEL doesn't release too often, but when it does we usually pretty aggressively move to new releases. You or your team should be ready to update your application for those new versions.
• Do we have some way to contact someone who knows the application well? If your application breaks at 4am on a Saturday morning, Fedora sysadmins will get paged, but if there's no one available to help fix things, your application could be down a while, and that's not something we want to deploy. Our users and contributors expect a high uptime from us, and we ensure that by making sure our applications are high quality and people are available to fix them.
• Can whatever you are trying to do be added to an existing application? We are usually happy to look at adding functionality where it makes sense, and if your goal is related to something we have already deployed perhaps it could just be added instead of running another application.
• We require SOP's and docs be written as part of the RFR process. This isn't just overhead, it's to allow people to manage the application if they don't have direct experience with it. If there's a known good, documented process to do common tasks with your application anyone who has the ability and can read it can just do it and not be blocked on application authors being available.
• Likewise monitoring is important as part of the process. It's easy to check that a web server is answering, but does that mean the application is up and running and healthy? Or are their other checks we could do that would test that?
• There's some applications we run where we aren't upstream for them. In those cases we really want to make sure that we have good communication with the upstream project and that it's active and open to bug fixes and changes. Also, there should be several people on the Fedora infrastructure team that know those applications and are able to talk with upstream about our needs and fixes.

So, in summary, writing an application is only the very small start of the work involved. Packaging, deploying, documenting, maintaining, adjusting for changes, fixing bugs, fixing security issues, monitoring, all need to happen to and it's up to YOU to plan for and provide those resources so you don't leave Fedora Infrastructure trying to do them long after you are gone. If you have ideas around a new application you expect to run in Fedora Infrastructure, you should come talk with us. The sooner in the process the better for you (no need to re-write an application later or redo a bunch of things to meet our process) and for us (help design new applications to meet our needs from the ground up).

Sorry I've skipped a few weeks of rawhide reporting. ;) Some fun issues in the last few weeks: freerdp was updated, and that seems to have had several issues falling out of it. remmina's rdp plugin is broken which broke composes of the Xfce spin as well. I have removed that plugin from the Xfce group for now, but it still needs fixing. vino was also broken by this, but it's also since been fixed up. Maintainers really should try and coordinate these abi bumps a bit better. ;( The 3.11 kernel was released, so rawhide has rolled on to the 3.12prerelease versions. So far it's been ok here, but a few glitches: Ran into a bug with backlight. Every time you power off your display the number of the acpi_backlight controls increases, making it not work too well for things that expect that to stay the same. This is bug 1005409. Also, for a time there the iwlwifi wireless driver was going very slow again (taking 14 hours for a 1 hour backup of my laptop), but that seems to now be cleared up. The lightdm issue with no icons showing has been tracked down and fixed. Was a minor bug in the gtk3 package (causing it to look in the wrong place for defaults). There were some anoying issues around firefox and a pretrans scriptlet. As a reminder for package maintainers, you SHOULD NOT use shell for pretrans scriptlets. Since on install there's nothing available but rpm and it's embedded lua, you have to write these in lua or expect that any clean install with your package will fail and anoy users. Perhaps we can get rpm to complain about this at build time or reject it. Otherwise things seem to keep rolling along pretty well and rawhide continues to be usable day to day for me. :)

Users of any fedorahosted trac instance may have noticed recently that their gravatar icon has become some weird little retro pattern. The reason for this is simple: We have switched from using gravatar to using libravatar. libravatar is fully open source (any one can run an instance), where gravatar is not. The default libravatar avatar is a little retro icon based on your email address hash (so it's different per user). Of course you can (and should) set your own: Do take a minute to go to: https://www.libravatar.org/openid/login/ and login with your fedoralogin.id.fedoraproject.org and set your default avatar to whatever image you like. Over time I am hoping we see more and more sites using libravatar, so it's a good idea to go and set your avatar now so you will be ready. In addition to fedorahosted trac our https://badges.fedoraproject.org site is using libravatar for all it's avatar needs.

Just a few things to call out this week in rawhide: Our increasing the deltarpm limit seems to have gone well. There's one issue with one package, but it doesn't seem memory related. Hopefully folks will enjoy the deltarpms from all the very large packages we didn't use to deltarpm. Next week the branched Fedora 20 tree heads into it's Alpha change freeze. This is where it will start to use bodhi for updates and in general get more stablized for Alpha release. Please remember to build in rawhide anything you build for Fedora 20 branched to keep the upgrade path sane. This is also an excellent time to clean up broken dependencies and issues in both rawhide and Fedora 20 branched trees. The x86 builder instances are being re-installed this week with Fedora 19 (To match up with our arm builders). Hopefully there will not be any issues with the switchover, but if you run into oddities with building, please let infrastructure and/or releng know about it. Otherwise things are rolling along pretty smoothly in the rawhide world.

Another exciting and busy week in Fedora Infrastructure. I finished getting our rdiff-backup setup in place. We have been using bacula to do backups to tape, but getting things off tape can be anoying. With rdiff-backup we should have easy access to the latest access and better access to historical ones. If things look good over the next month or so, I will likely switch off bacula and just backup the rdiff-backup data to tape to keep longer term. Being a paranoid sysadmin, I always like having good backups going. Applied a patch to move a bunch of our ansible tasks to roles (thanks much misc!). Cleaned up a bunch more ansible things, setup playbooks for our arm-releng instances, still more work to do, but we have made some nice progress. Next week I hope to do a bunch more. Fired up a releng01 instance to run branched bulds (since we branched f20 off rawhide). We have allocated them a bunch more memory now, so they should be able to deltarpms of the larger packages now as well. (There used to be a limit to avoid overloading the guests). Next week we are going to do some updates (although hopefully no reboots needed), and get ready for the Fedora 20 Alpha freeze the week after.