With the recent release of gnome 3.17.3 I decided to take a look at running gnome under wayland again. I run rawhide on my laptop, so it was pretty easy to just select gnome on wayland from gdm and give it a go. The good news is that progress is being made from the last time I tried. For pluses:

• No crashing or hanging. Last I tried my session hung after a short time.
• All the extensions I use worked fine.
• Copy/paste worked fine between apps

However, there's still things to fix up:

• gnome-terminal is basically unusable due to resizing itself all the time on focus in or out events. The window gets smaller and smaller and jumps back to normal, then super small again, etc. I had to run xfce4-terminal to have any kind of sanity.
• There's some weird spacing issues. I usually run with 2 terminals, each taking 1/2 of the screen. If I do this under gnome/X11 they fill up the entire screen, in gnome/wayland there is about a 1/2 in space around each terminal when I side tile them.
• There is some serious focus lag. I switch applications/windows using alt-tab, and there's sometimes a >5 second delay after I have switched before the new app has focus. This makes me send things to the old application I didn't mean to. It may be related to me having sloppy focus enabled.
• No ssh-agent, so you have to start one or type in your ssh passphrase a billion times.
• Some things seemed oddly sized, like the display brightness widget. (Ie, the OSD when you hit up or down brightness). Much larger in gnome/wayland than gnome/X11

Overall things are moving forward to a bright wayland future.

As many of you may have noticed, we have had some issues the last few days in Fedora Infrastructure, in particular with metalinks (the files used by dnf/yum to pull down repository data). First, I'd like to let you know that the mirrorlists/metalink issues should all be fixed now, so please do 'dnf clean all' or 'yum clean all' and retry your operations and let us know (via a Fedora Infrastructure ticket or #fedora-admin on irc) if you still see any problems. If you changed your repo files while this issue was happening, I would strongly urge you to change it back to the metalink. It has a number of great advantages over a specific mirror or mirrorlist. We don't yet have a detailed root cause write up, but should have something next week. The problems seem to have been a number of issues, starting with a network problem that seems to have caused some issue with a virtualhost server that had our main vpn hub and postgresql server on it, then on to a possible mirrormanager bug where it removed the wrong alternative repomd data, breaking metalinks. Most of the metalinks were fixed last night, but we wanted to make sure and have everything back to normal before returning to normal status. We did create some valuable tools out of this unfortunate outage, including scripting to update specific repomd.xml's and not wait for an entire crawl of the tree (which can take many many hours), scripting to check all metalinks against the master mirror and yell when they are not completely in sync, and more. I'd like to thank everyone for being patient while we work on this issue and especially thank all the hard working folks on the Fedora Engineering team that worked on this around the clock! I'm proud to work with such great folks.

Like yum before it, dnf also uses (by default) metalinks served by Fedora Mirrormanager mirrorlist servers. Metalinks are a great thing, but it seems like many people don't understand them or realize what great benefits they provide, so I thought I would do this post to help. A metalink is a xml document that includes checksums and lists of mirrors that have repodata. In Fedora you can see it in the repos provided by the fedora-repos rpm under /etc/yum.repos.d/ like fedora.repo for example: metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch Note that the metalink is fetched over https, meaning if you trust the ssl cert CA setup, you will know that it comes from the Fedora mirrorlist servers and hasn't been intercepted or tampered with. You will also note that it's passing some data to the mirrorlist server. What repo you want, what your arch is, and also your IP address. Once yum or dnf pulls this url, it goes to mirrorlist servers. They take the request and build a metalink xml for that request based on a cache of mirrormanager data that they have. At the top of the metalink is included the current repomd.xml file's timestamp and checksums. Below this there may be serveral "alternate" checksums for repomd.xml from previous repodata. Any of these are considered valid. This is to allow for syncing time for mirrors. If an updates push has just finished, it means only the master mirror is likely using the very newest repomd.xml, allowing the previous one means people will still be able to use mirrors until they sync up to the new one. After some time, the older alternative repomd.xml is dropped, leaving the current one. Next a list of mirrors is generated: Is there some mirror that has marked itself as always preferred for your ip block? Is there a mirror that has marked itself as preferred for your ASN? Is there a mirror geographically near you? What mirrors are currently marked "up to date" (the mirrorlists have a cache, but it's refreshed every hour and the crawler marks mirrors up to date or not all the time)? It weights things by the amount of bandwith mirrors have indicated they are able to handle (so a fast mirror with small BW doesn't get overrun by requests). It then takes the output of this query and makes a list of mirrors in a preferred order (most preferred at the top, then down). Finally the master mirrors are added at the lowest preference to the list, so things should fall through to them if no other mirrors work. Once you have the metalink, yum or dnf then tries the mirrors in preference for the repomd.xml. If it finds one with a matching checksum, it uses that mirror. In the repomd.xml is checksums for all the rest of the repodata, and thus all the packages in the repo. So, if you have the correct/valid metalink, you have the correct/valid repomd.xml, which means even if you download over ftp or rsync or http the packages you get MUST match the valid checksums. Of course additionally after you download and before you update or install the gpg keys are checked on the signed packages, but if you use the metalink you will never need to worry about a corrupt/invalid/unsigned package in the first place. I've seen a number of people lately suggesting you should just replace the metalink in your repos with a direct link to a local mirror you like. This is really a bad idea except as a very short term workaround. If you do that:

• If the mirror you are pointing to is out of date or needs to have some kind of maint on it, mirror admins can remove it or mark it not up to date and all the metalink using folks will no longer hit it, but you will.
• You no longer have a way to tell if the metadata on that mirror has been corrupted/tampered with. True, the packages are still signed, but a evil mirror might give you known vulnerable (but signed in the past) packages. You have no way of telling. Or withholding packages with fixes from you.
• If the mirror you are pointing to is http or ftp someone could man in the middle attack it and tamper with what you get.

Finally, now that we have mirrormanager 2 deployed we are able to start looking at some enhancements for it. Some ideas I have had:

• A way to tell the mirrorlists you only want https (or http/https/rsync ok, but no ftp), etc. This should be pretty easy to do.
• Consider just dropping ftp. It's a horrible protocol, perhaps we don't need to keep it alive anymore.
• More comments in the metalink users get that will help us debug metalink problems. ie, how it decided what was included, etc.
• We have already been working on lots of great improvements on the crawler to do quicker crawls and detect out of date mirrors more easily, including a canary mode (check just repomd.xml), multithreading rsync checks and more.
• I'd like a way to tell what mirrors have iso/qcow2/raw images. We point people to mirrors to download those, but sometimes they get a otherwise updated mirror that doesn't carry images and get a 404. If we kept track of which exact mirrors had that exact image we could direct people more accurately.
• <your ideas here>: please file ideas at https://fedorahosted.org/mirrormanager/

I'd like to thank Pierre-YvesChibon and Adrian Reber for all there recent work writing, rolling out and debugging mirrormanager2. Thanks!

Just a note for everyone/anyone who installed Fedora 22 from anything before the Release Candidate (RC) composes: (If you install from the final release on tuesday you are not affected of course): Before this point the updates-testing repository was enabled and you very likely installed some things from it if you did any installs or updates after you installed. A fedora-release update came along and disabled this repo now, so you have packages from it installed, but that repo is no longer enabled. This can show up as weird issues around mismatched devel packages or other strange looking dependency issues. Please do one of the following if you see any such issues: 1. You can re-enable updates-testing and help us test updates. See: https://fedoraproject.org/wiki/How_to_test_updates or 2. You can run a 'dnf distro-sync' and downgrade your packages to all the correct versions available in updates and the base repo.

I ordered a colorhug ALS right after they were announced. What is that you might ask? Well, it's a USB Automatic Light Sensor, useful for laptops that do not have a built in one. It allows software to see just how bright your environment is and adjust your laptop screen (saving lots of battery life). The device came pretty quickly (considering it was shipped from the UK), and had a nice little instruction sheet, 1GB usb stick with docs and info and the device itself. It sticks out of the USB port a bit more than I would like, but it's reasonably unobtrusive once it's plugged in. Fedora already includes the colorhug-backlight package, so that was simple to install and run. Right now it's tied to gnome, but I suspect it might not be hard to interface into other desktops. After playing a bit with the default level, things seemed to work quite nicely. I did have a few times when I had the laptop on my lap and the sensor got obscured or put in shadow and adjusted the screen when perhaps it shouldn't have, but it was easy to see what was happening and that didn't occur in most normal use. It would be cool if something like this could be added to yubikey nanos or the like (since I already have a port with one of those if it had a sensor it would be great) but not sure how practical that would be. If your laptop doesn't have a light sensor and battery life is important to you, I would definitely recommend picking one of these up. See: http://hughski.com/colorhugals.html for more info and ordering instructuions.

There was quite a bit of activity this weekend in Fedora rawhide, so I thought I would send out a quick post with news for those that might be interested:

• I fixed libcanberra package to work again. Due perhaps to changes in rpm, it was not pulling in it's gtk2/gtk3 subpackages when you installed libcanberra-devel, which meant that anything that tried to build against it failed. I just added manually requires back in there, but there's still a lot of questions about the packaging there. It's... odd. The gtk2 subpackage requires the gtk3 one. In any case that allowed me to:
• Rebuild gdm and pavucontrol so they worked and were synced up with the f22 updates that had been going out. In particular the newer gdm versions were fixing issues around using wayland by default and were quite needed by rawhide users.
• Xfce 4.12 was released saturday morning. nonamedoc built out all the core builds into rawhide and then I pushed out builds for all the plugins and misc associated packages. There were a few minor issues and broken deps, which I mostly got cleaned up and working sunday. See the fedora-xfce list for more detailed status. Right now, IMHO things are looking pretty good, so we may try and build up f22 builds and see if we can get in a late change and freeze break to get it into alpha. As far as Fedora 21 and EPEL7, we will need to wait and see until after everything is sorted in 22 and rawhide. We may be able to do an update there. In any case there will be a copr. I don't have any plans to update Fedora 20. It's nearing it's end of life and such a disruptive change doesn't seem worth it.
• There were missing symbols in x drivers, causing people to get fallback graphics. This was tracked down to the hardened_build for all packages change. It's been reverted for now, so everything should go back to working for folks. Hopefully we can find a solution allowing us to re-enable it and not break things.
• We have started (and nearly finished) signing rawhide packages. Everything won't be 100% signed until we enable out autosign setup, but it's going to hopefully be close and hopefully we will get that enabled soon. ;)
• Rawhide live images are still failing to build due to a systemd change (making /etc/resolv.conf a link). I'm going to try and get that sorted out with systemd folks soon so we can have rawhide images again.
• A vim package update was pushed out with an incorrect comment in it (leading to everytime you run vim you get an error). I notified the maintainer about it in a bug, if I hadn't been busy with other things I would have just fixed it. Another provenpackager came along and did just fix it. :)

I'd also like to add a short note about expectations here. Several times in the last week people have said things like "oh, it's rawhide, not surprised it's broken, see if it clears up in a few days". Please don't tell people this, instead tell them: "looks like you hit a bug, please file it so it can get fixed up". The X issue could have been cleared up more quickly if people had reported it (I didn't notice here because I actually failed to reboot for 3-4days). Bugs and bugs, and we should report them and get them fixed up. If they are serious and don't get fixed in a timely manner, please feel free to let me know and I can try and get them sorted out. We need to keep rawhide working and running so we can integrate new software.

I thought I would post somewhat of a rebuttal to http://marcin.juszkiewicz.com.pl/2015/02/11/rawhide-unwanted-baby-in-fedora-world/ with some of my thoughts around Fedora rawhide and try and clear up some misconceptions. There are indeed people using Rawhide day to day. I myself have for the last few years, and I know there are a number of others (based on IRC conversations and posts to the test list). Regarding the KF5 issues, this is a somewhat unstable time for KF5, as they are just now landing things and integrating them and also gcc just updated to 5.0, causing them some issues. Perhaps some of this work could have been done in a copr or the like, but sometimes it's really hard to anticipate what will happen when you finally build in the official Fedora buildsystem. I don't think the common answer here should be "you should expect that in rawhide", but instead "You should understand that at times various parts of rawhide may be under more work and help them work around those issues". I've definitely run into situations in the last few years where something was broken and I couldn't use it, but I reported bugs on them and people fixed them up. In the mean time it's always good to have alternatives. As for IRC, rawhide issues are 100% on-topic for both the #fedora-devel and #fedora-qa channels. It's very true that people in #fedora will tell you you are asking in the wrong place (they only support stable releases there), but if anyone in #fedora-devel or #fedora-qa tells you that, they are wrong. Also, realize that IRC is a async media. You may well have better luck filing a bug or posting to a mailing list depending on the time and nature of your question. I tried to make sure and capture the audience for rawhide on the Fedora rawhide wiki page: https://fedoraproject.org/wiki/Releases/Rawhide#Audience Finally, in coming months, some improvements/plans for rawhide:

• We hope to finish our automated signing for rawhide. Once this is in place it would allow us to know that everything is signed in rawhide, and as a bonus would give us a gating point via koji tags (ie, right now builds just go to f22 tag, but after we land this things will go to a f22-pending tag or the like, then get signed or have other things run on them).
• Once we have that in place we have a gating point and we can hook automated qa type things into it. Basic problems could be detected and the build never pushed out.
• I'd personally like to revist the idea of shipping previous versions in rawhide (ie, foo-1.0-1.fc23 and foo-1.0-2.fc23) to allow for easier downgrades. Will need to try and convince the rest of releng to do that however, and it would make things larger.
• There's likely to be a mass rebuild in rawhide in the next few weeks. Both for the gcc5 change and also for hardened_build by default.

If anyone has questions about rawhide, feel free to ask me in #fedora-devel on irc, the devel list or via email, happy to help out.

So, what better to do on a lazy sunday than clean up some of the Fedora Infrastructure ansible playbooks to be idempotent. I ran into several common cases in our playbooks for something to not be idempotent, so I thought I would share them in case others run into the same issues. First, for those of you wondering what 'idempotence' is, wikipedia describes it thusly: "the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application". More practically in the ansible world it means after 1 run of a playbook to set things to the desired state, further runs of the same playbook should result in 0 changes. In Fedora Infrastructure we run a ansible-playbook run of each of our host or group playbooks with '--check' and '--diff' every night from cron. Then, a handy report is mailed to our sysadmin-logs members showing all the playbooks that showed something would have changed. In the ideal state this report would be empty, but there's various reasons things show up in there. Of course the most common reason a playbook would show up in our check/diff report is that something actually changed and the playbook has simply not yet been run on the host(s). This could result from someone checking in a change to git and not running the playbook, some variable or global change was checked in and the person doing it didn't realize they needed to run this particular playbook, someone made a manual change on the host (BAD sysadmin!), or perhaps the host was not reachable or up when the playbook was last run. This is the easiest case to fix up: just run the playbook(s) and everything gets back to the steady state. Another common case I see is when a file is changed multiple times in the same playbook run. This can result from a simple copy paste error, or a conditional that isn't correct (a 'when:' in ansible terms), or having the same template or file in multiple roles or directories and not realizing that you are setting it in two places. This can be fixed by only changing the file once, either by collapsing the task into one place, or adding conditionals so it only applies once on each host. Sometimes in rare cases you DO need to change a file multiple times (for example, put a admin config in place, run some admin commands, put the normal user config in place). In these cases you can use ansibles "when_changed" to just tell it this isn't really a change and don't bother marking itself as well. Do be careful using this. Another case I see a good deal is when you run a command or shell (which is fine, --check simply ignores them because they always change) but register a variable thats used later in the playbook. In this case there's a problem when running with --check because the command/shell is never run, the variable isn't set and when you try and use it the playbook fails. In these cases you can use: "always_run: yes" and the command/shell will run. Of course you may also want to add a "when_changed:" so it doesn't always show as changed. I reduced our non idempotent playbooks today a good deal, but there's still more work to be done. I do look forward to the day the report is empty, and I don't think it will be all that far off. ;)

In Fedora Infrastructure, all our applications are Free software. It's one of our base requirements, allowing anyone out there to examine source, improve or modify things. Sometimes, just having the source of an application isn't enough, you need the raw data to figure out some issue or generate some metric or support a theory. Happily in some cases, our applications databases don't contain any sensitive data. In those cases we provide dumps of these databases for any interested parties. Currently available: datanommer - This is the application that consumes all messages on our fedmsg bus. If there were messages in the past, it would have recorded them. You want this db if you want to look at fedmsg metrics or information. fedoratagger - This application allows users to tag and rate packages in Fedora. This db is usefull if you want to look at application ratings or tags. pkgdb2 - This is the application that controls access to Fedora packages for maintainers. You want this db if you want to look at stats about packages or packages related to acls. And finally, new today: koji. This is the db that is used by koji.fedoraproject.org. It's got information about all official builds back to the dark ages. You want this if you want to look at history about builds or buildroots or old fedora release versions. All of these are available at: http://infrastructure.fedoraproject.org/infra/db-dumps/ If you find these useful or see other applications we run where there's no sensitive data stored in the db, please let us know!

In the last few months I've seen several questions around Xfce in Fedora, so I thought I would share some thoughts around versions and other questions related to Fedora's Xfce efforts. Note that I am only one of several maintainers of Fedora Xfce packages, so these thoughts are my own (Although I think my co-maintainers share them too). The current stable released version of Xfce is 4.10, which was released in April of 2012. A rough plan was made for a 4.12 release, but Xfce is an all volunteer project (as far as I know, no one is paid to work on it full time), so timelines have slipped. There's a number of components with 4.11 development releases out, but currently no hard plan for a 4.12 release. There are also from time to time 4.10.x bugfix releases for various components and plugins. Several people have asked why we don't land the Xfce 4.11 builds in rawhide. The problem is that since there's no timeline for 4.12 to be released, we could easily end up shipping those in the next stable Fedora release (Fedora releases branch off rawhide). While in general these builds work fine, they still aren't good to ship out in a stable release. Since these are unstable development releases, Xfce developers could change the way the interface works, drop or add new functionality or otherwise make end user visible changes. While these are fine between Fedora releases (people set aside time and realize that they are going to have to learn the new setup), they would not be acceptable to change in a existing stable release. This sort of thing is an excellent use for the Fedora copr system, and indeed we have a 4.11 copr ready and waiting for those willing to use it. I've also seen from time to time lately people saying that "Xfce is dead". This is completely untrue. There's still a fair bit of development going on. There's bugfixes, new releases and lists activity. True there's not a 4.12 release in sight yet, but I'm not sure thats really such a bad thing. The Fedora Xfce spin is of course alive and well. We did have to finally give up on limiting to 700MB size with Fedora 21. It was just too difficult to trim things and have a good experience. We re-targeted for 1GB usb, which hopefully most everyone can find these days. :) Xfce in EPEL continues to be around for EPEL users: 4.6 in EPEL5, 4.8 in EPEL6 and 4.10 in EPEL7. There have also been some questions around Xfce and HIDPI displays. The support isn't ideal at all, but Xfce an work fine with some tweaking on a HIDPI display. Here (on a 3200x1800 display) I adjust the dpi in Settings -> appearance -> fonts. This doesn't get set completely right on login (sometimes some autostarted apps don't see the change and need to be restarted). Also, you get very small window decorations (hard to resize windows). Otherwise it looks fine and works well. Ideally once Xfce goes to gtk3 they can take advantage of the HIDPI support there.