I've been using claws-mail to both read/compose emails and to read all the vast pile of RSS feeds I try and keep up with for quite a while now and it's for the most part been fine. I never liked claws-mail use of mh folders instead of something more standard like Maildir, and it's calendar integration is... not there, but it did a pretty good job. Unfortunately, claws-mail uses a plugin called 'fancy' to render html (which sees heavy use when loading rss feeds). This plugin uses the old old webkit1 (webkitgtk package in Fedora). This package hasn't been maintained upstream in quite a long while and the number of vulnerabilities in it has just grown and grown. Other distros have dropped it entirely, and Fedora is finally following suit very soon. This has caused the Fedora claws-mail maintainers to drop the fancy plugin to prevent dropping claws-mail entirely. However, without that plugin, RSS reading is... not very pretty. So, after seeing Jiri Eischmann's recent post ( https://eischmann.wordpress.com/2017/03/10/nextcloud-linux-desktop/ ) about using the nextcloud News app, I decided to give that a try. Installing was pretty easy, but note that it's not in stable apps yet. Importing my feeds from claws-mail's RSSyl was pretty easy, but the only want to put feeds in categories is drag and drop. Drag and drop when you have many many feeds is a real pain, I really wished for a 'move to folder' option. The Fedora "feedreader" app looks pretty nice and after a bit of tweaking handled things pretty well. The android OwnNews app also seems pretty nice and functional. I did notice some times when it seemed all three readers were not 100% in sync as to what was read or not, but things stayed pretty close. Being able to use my phone to browse feeds will be handy. I also took this time to cull a bunch of feeds that didn't have any posts in the last few years or I didnt care about anymore or the like. Overall I think this will be a pretty reasonable replacement. Next up was mail. I could have just stuck with claws-mail for just mail, but without the fancy plugin any html emails would not render really at all. So, I figured I would bite the bullet and look at evolution again. I switched over my work and gmail accounts first as they use imap and state is (mostly) remote anyhow. They setup pretty easily. Next I added a account that was all my old claws-mail folders in mh format. In the past when I tried this evolution crashed, but this time it seemed to handle it ok. I later disabled this account for now, as it caused fetching new emails to take a while and I only need it if I need to find an old email and can enable it then. Finally I switched over my main mail stream. Unfortunately, I couldn't find a way to import my old claws-mail filters, but this did give me a chance to cull emails that I get. As things came in saturday and sunday I added filter rules, unsubscribed, or otherwise made a note to stop whatever it was sending me email. There's definitely some oddities I will have to get used to with evolution, but overall I think I can make it work. The biggest frustration is that the shortcut keys to do things don't seem to work for me. (ie, . and , are supposed to go to the next or previous unread message, but they do not here. They don't do anything). There doesn't seem to be a way to quickly go to a particular folder. After a bit of use, I figured out the problem: The shortcuts are focus specific, and I don't really see any indicator which pane I have focused. ;( Right now I think the RSS change is likely to stick, not entirely sure on evolution, but if it doesn't there's a number of other clients to try out (thunderbird, mutt, or back to emacs :)

Well, branching of Fedora 26 off rawhide has come and gone, and it was a bit of a rocky ride this time sadly. The branched composes were failing at first, then working, but not actually syncing to the master mirrors in order for mirrormanager to notice it and people to you know, actually use it. This time we tried to make sure both branched f26 and rawhide were signed the entire time by the right keys. They currently are, but sadly, there's a old/obsolete/bad/nolonger used/no good/varmit of a Fedora 26 key thats in the fedora-release-25 package. So if you are trying to upgrade you may need to find/get the right f26 key before doing so. Also, the resigning meant that mirrors had to pull basically every package again, so it's going to be a bit before everyone is synced up on rawhide and f26 content. If you want to keep following Fedora 26 to it's release, you need to run a 'dnf repo-sync --releasever=26' to make sure you have the right releases and repos. If you want to stay riding on the rawhide trail, you can just dnf update and should get the fedora-27 release and repos. Until next time, ride safe.

Some recent gotchas in rawhide:

• firefox doesn't seem to compile correctly with gcc7 (and the optimization levels Fedora uses by default). The current rawhide version will install and run fine, but looks horrible. As a work around you can install a flatpak or a binary version from upstream or downgrade to firefox-51.0.1-2.fc26 from koji. This is tracked in: https://bugzilla.redhat.com/show_bug.cgi?id=1422532
• A few weeks ago, python2-jinja2 was updated to 1.9.4, then 1.9.5. Unfortunately 1.9.4 broke ansible templating, and all the problems were not fixed by 1.9.5. As a consequence, ansible hasn't been runable the last few weeks. Until today: I pushed ansible-2.2.2.0-rc1 into rawhide. This isn't the final 2.2.2.0, but it hopefully fixes the jinja compatibility issues and gets rawhide ansible users back on track.
• After my last post about composes failing due to an unsigned package, we ran into 3 other issues we had to fix to actually get a compose working: First, an nss update caused lorax to break. Turns out it was due to the mock chroot used not having /dev/urandom and nss noq requires that to be there. Next was the rdma-core package breaking things. It was untagged a while back to fix, but then the mass rebuild rebuilt it again and it got pushed out again. Finally policycoreutils-python started pulling a ton of more deps which broke the minimal installer chroot.
• Even after we now have composes, livecd's are not working due to an anaconda bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425827 Hopefully we will have a fix for that soon and livemedia will be back.
• There was a pungi bug preventing the ostree composes working, there's a proposed fix for that one.
• i386 images and media are all failing due to a weird dep solving issue where it says "will not install src.rpm". https://bugzilla.redhat.com/show_bug.cgi?id=1416699

Lots of bugs, but we are moving forward on quashing them now. I am hoping we have some good composes later this week and then we can try and keep them that way.

In the final run-up to the Fedora 25 release, we slipped a week because there was a bug in installs on apple osx (now macos again) hardware. This was (and is) a use case the Workstation working group cares about, as they would love for folks with apple hardware to install Fedora and use it on that hardware. Sadly, we don't have too many testers with this hardware to help our testing cycles, and many community members with this hardware also are using it day to day and cannot afford to reinstall and test at the drop of a hat. I use my personal yoga 900 for my main machine, so my work laptop has always been for me a test machine or a backup in case of failure. There are a number of reasons for this: I prefer to pick and use laptops that aren't standard corporate offerings, I like to know that I can do anything I choose with the laptop, and if (god forbid) I moved to a new job I wouldn't have to give my primary laptop back. So, when I was up for laptop refresh and I saw that a macbook pro 13" model (12,1 apparently) was available, I decided to choose that and help out with Fedora testing efforts on this hardware. I do feel a bit bad about this as I am not a big fan of Apple and this does mean giving them money, but on the other hand, hopefully I can test Fedora and help avoid slips due to lack of hardware. Also, I can run rawhide on it and see if I can get everything working fully. The macbook arrived the other day and yesterday I unpacked it and did some initial testing: First, the hardware: Seems nice enough, but I am not sure why anyone would get one of these over a Dell XPS 13 or a yoga 900/910. It's got a lower resolution screen than those, 8gb memory instead of 16, and only an i5 cpu instead of an i7. The feel is very solid, but it makes it just makes it seem too heavy to me. Otherwise the screen is bright and nice, the keyboard backlight is nice (The yoga 900 only has off, dim, bright for the keyboard backlight, but this macbook you can set it to whatever you like), the power connector is neat in that it has a light on it telling you if it's charging (orange/amber) or fully charged (green). The keyboard and trackpad seem fine. I decided I would go through the normal macos setup first and then try and setup Fedora to dual boot, as I imagine that would be a common setup. Part of the setup instructions that came from my corporate overlords was mention of enabling File Vault full disk encryption. So, I did that and got everything installed and seemed to be working fine (at least as far as I could tell, not being a macos user normally). Still following what I would think would be the more traveled path, I went to https://getfedora.org and downloaded the Fedora Media Writer. Download went fine and it was no trouble to run it, but it did give me a warning that this was something I had "DOWNLOADED" from the internet. I don't guess we have much way around that warning as we are signing the binary fine and it's not saying it's unsigned, just that it was downloaded and are you sure you want to run it. Download and burn to usb went just fine, no problems at all with FMW. It might be nice if there is an option to download Rawhide images if you really wanted them. Hold down the option key and power on and you get the boot selector thing. Choose FedoraMedia and there's the Fedora Live USB. Everything booted up nicely and I poked around to see what was working or not working. Turns out so far the only thing that doesn't appear to be working out of the box is the webcam. It seems to be a broadcom model that some folks are working on reverse engineering, but haven't gotten that done enough to merge into the mainline kernel ( https://github.com/patjak/bcwc_pcie ). I might try that out sometime down the road. The power connector activity seems a bit odd as well: when you plug or unplug the power it seems to take a minute or two before it notices and starts updating. Next I pulled up the Anaconda installer and looked to install Fedora along side the existing OS. I needed some space to install in, so I selected the largest partition which I knew was the main macos volume (but oddly was showing up as Unknown), and told anaconda to shrink it. That seemed to work fine, and I installed Fedora in the newly freed space. The install finished fine and I rebooted. On reboot, I now got grub and Fedora booted and worked fine. The macos entries however, errored and didn't work at all. Turns out this is a long standing, known bug: https://bugzilla.redhat.com/show_bug.cgi?id=893179 I tried various things suggested in the bug to chainload, but with no luck. Luckily you can still hold down option and get the native bootloader. So, I did that and tried to boot the macos install, but it would think for a while and then error with a picture that was a circle and crossbar and fail to boot. After poking around it seems I now hit: https://bugzilla.redhat.com/show_bug.cgi?id=1033778 ( https://fedoraproject.org/wiki/Common_F24_bugs#apple-core-storage-wipe ). So, my macos partition was unknown to anaconda, so it let me shrink it, but it messed it up completely and nuked all data that was on it. Had I not encrypted things it would have worked, but since I did it was gone. Luckily apple has the advantage of controlling the hardware platform in this case, so I just needed to boot with option command r to get a network rescue. This let me wipe the drive, repartition it, reinstall macos, then boot the Fedora USB again and install Fedora in the space I left for it. So, it wasn't the end of the world, but it would sure have been anoying if I had data there. After all that installing, I moved the Fedora install from F25 to rawhide. No issues there, everything seems to work and be fine after that. So, if we could somehow fix the two bugs I ran into for f26 I think it would help macbook folks a fair bit. If anyone needs me to install or test anything on this platform, just let me know. I plan to keep the macbook ready to (re)install most anytime and hopefully can provide more test coverage for upcoming cycles.

Greetings everyone, lets dive right into the latest changes in the rawhide world:

• The Fedora 26 mass rebuild ran and finished last weekend. 16,352 successfull builds happened, along with around 1,000 that failed to build. Now we have a few weeks until we branch f26 off to fix things up.
• The mass rebuild did disrupt signing of normal updates. Perhaps next mass rebuild we should look at standing up another set of signing servers to just sign the mass rebuild.
• Composes for the last few days have failed. Turns out its due to an unsigned package. But how could that happen? We passed all the builds through the regular signing process. Turns out when builds were tagged in, there were a few builds that overrode newer versions already in rawhide, so releng ran a custom script to retag newer builds back in. However, there was a package where the maintainer built a new version, decided for some reason it was unusable and untagged it. Thats fine, but the custom script mistakenly tagged this "newer" build in and it was long enough ago that it's signature was removed. Just a short note here about "newer": koji has no concept of package versions. To koji if you ask for all the 'newest' builds in a tag, it will give you the most recently tagged ones. This importantly has nothing at all to do with the package-epoch-version-release, thats just not on a level koji knows or cares about.

I find tracking down obscure bugs interesting, so I thought I would share another tale of one I just tracked down. Ever since coming back from the holiday break this year, I found my bluetooth headsets had stopped working. (I have 2 very different models: a plantronics M50 that I have had for a long time, and a Bose QuietComfort 35 (yes, the headphones have a headset profile!)). At first when I noticed the problem I tried the obvious things:

• Downgraded pulseaudio
• Downgraded bluez
• Downgraded and tried various kernels.
• Looked though tons of upstream pulseaudio and bluez bugs.

No change at all. I added myself to a existing bluez bug and provided the bluez maintainer a bunch of debugging output, but it all looked pretty normal, there was just no sound. So I gave up and moved on to more urgent things. Fast forward to today: what better way to spend a sunday afternoon that more debugging. I repeated my former tests and had the same result. Then I decided to try some LiveUSB's to isolate it. I downloaded and booted a F25 updated workstation iso and it still had the issue. So, I thought, Ah ha it has to be the updated pulseaudio since that landed in f25 updates. But to be sure, I downloaded and booted the stock f25 release iso. Still had the issue. Boggling. Then I went looking around in the kernel bugzilla and found a report that talked about issues with bluetooth and firmware. That was just the hint I was looking for. So, I downgraded the linux-firmware package and tried things and... it still didn't work! There was one more trick: The firmware only reloads on cold boot. If it's already loaded (or misloaded) it doesn't even try again. So, cold boot and then everthing started working. The key kernel boot message on the failing firmware: Bluetooth: hci0: Failed to send Intel_Write_DDC (-22) and the working firmware: Bluetooth: hci0: Applying Intel DDC parameters completed For bonus points of course the failing firmware file is: ibt-11-5.ddc and the working one is: ibt-11-5.ddc (yes, thats right, they are changing the file around without changing the version any). Next of course is to figure out where to report this to get it fixed, but some poking around and testing found that http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/commit/?id=581f24500138f5e410d51ab63b205be9a52f4c77 already had a fix. I just need to wait for a update to the linux-firmware in Fedora and everything should be back to normal. So, troubleshooting lessons for today: Sometimes cold boot matters. When you think you have downgraded everything that could cause a problem, remember firmware. Reading around on bug reports will sometimes give you ideas on how to solve yours.

Some more notes from the rawhide trail... Astute readers may have noticed that there was a Mass Rebuild scheduled for last week ( Feb 1st) and it's not yet happened. There were a few reasons for the delay: The gcc folks wanted a few more days to quash some more bugs in gcc 7.0 and additional time was allocated to a change in how debuginfo packages would be generated. Hopefully this rebuild will fire off tomorrow (the 7th) and result in a 1 week push in the Fedora 26 schedule. Just a reminder about Mass rebuilds: These are done in a side tag by an automated release engineering script. It simply uses the rpmdev-bumpspec script to increase the release tag by 1 and add a changelog entry about the mass rebuild and then initiate a build. Packages that fail to build will have FTBFS (Fails to build from source) bugs filed on them after the mass rebuild is over. There's two weeks alloted to cleaning up and fixing these FTBFS bugs and any other fallout from the Mass Rebuild. If you are using rawhide, note that there will be a day after the Mass rebuild where most all of your packages will be updated. Allot extra time to download and upgrade on that day. The last Mass rebuild took around 1.5 days to complete. This one may be slower due to all the additional alternative arches we now have. Another thing that needed to land before the Mass Rebuild was the targeted boost rebuild. This has completed and the side tag merged back into rawhide. Tomorrow's rawhide should have a bunch of packages that use boost updated. There will also be a few that still need fixing for the new boost version by their maintainers. After the mass rebuild is over and cleaned up after, we have the branching off of Fedora 26 from rawhide on the 28th, and then rawhide will march on to Fedora 27.

The last week or so I have spent a ton of time on kojipkgs (sorry again for any build failures this may have caused), so I thought it would be good to outline what it is and how it's used and finally the working setup we have now. kojipkgs is a concept that koji has of a host/url to go to to download packages. On small koji installs this host/url can be simply the koji hub host. Or it can be a seperate host or hosts, as long as it has access to all the packages koji wants. So in practice this means it has to share a NFS mount or other shared storage with the koji hub. For many years Fedora infrastructure had a kojipackages instance that NFS mounted the koji storage, ran a apache web server locally on port 8080 and then had a squid instance in front of that, listening on port 80 and 443 and talking to apache for any items that were not in it's cache. This worked out well because squid could cache commonly used objects and save access to the NFS server as well as return more quickly. However, it also has drawbacks: It's a single point of failure and it's exposed to the outside world and squid's SSL code has not gotten anywhere near the same scrutiny that apache's has. A bit of a side note, but related, lets talk about how koji does builds for a minute. If you tell koji to do an official build of some package, first it makes a mock chroot and downloads and installs the packages in the srpm-build koji group (you can see these and what packages are in them with a 'koji list-groups' command). Note that this install is done with the HOST dnf into the chroot. The src.rpm is then built from pkgs git and lookaside cache and then koji starts builds for any needed arches. These builds first download the src.rpm via a koji urllib2 call and the make a mock chroot and download and install all the packages in the 'build' group. Then rpmbuild is called, etc. So, week before last I updated builders (as I often do) and also upgraded the kojipkgs01 squid server (running rhel7) to the latest versions. However, we started seeing downloading problems at build time. Sometimes (but rarely) dnf would just fail to download a package it needed and fail the build. I figured this would be a great time to try and remove the single point of failure of that one server. I spun up a second kojipkgs instance (kojipkgs02) setup just like the first one (rhel7, squid, apache, NFS). Then I setup our two proxies in that main datacenter to handle kojipkgs requests and use haproxy to send them in to kojipkgs01/02. Then, a switch of dns and it was switched over. Now, however we hit a new problem. Sometimes the src.rpm download via koji was not working. It was silently downloading only part of the src.rpm and then failing the build. Additionally, the build download issues were still happening. I tried isolating to just one proxy. Just one backend. Various haproxy balancer setups so that requests that went to one backend kept on that backend. I replaced both kojipkgs instances with Fedora 25 (much newer squid version). Koji has the ability to pass several urls to the mock setup it uses for builds, so we passed it multiple urls to kojipkgs there and that seemed to get rid of that problem, but the src.rpm download issue was still there. Finally, we tried disabling squid's "smp mode". Suddenly everything started working nicely. So in the end of this saga we have:

• Nice Highy Available kojipkgs. (2 proxies using 2 backends, so we can update, restart, reboot a proxy/backend anytime we like without causing problems).
• A bug report on librepo to retry downloads the number of times it claims to in dnf.conf default even if there's only one baseurl and it was slow/timed out: https://bugzilla.redhat.com/show_bug.cgi?id=1414116
• A koji ticket to make the src.rpm more robust: https://pagure.io/koji/issue/290 and some PR's to change it use requests and validate the downloaded rpm already.
• I still need to file an upstream bug on squid about smp mode, but I don't have too much info to give them (there were no errors anywhere, just some small number of connections would hang/not finish).

I recently saw a post from my co-worker Kushal Das: tools-i-use-daily and I thought it would be fun to share the same information:

Operating System

I use Fedora rawhide on my laptop, which is my primary client machine. I don't have a desk and desktop and monitors, just the laptop and a comfy chair. At home I have two dell CS24 servers (one is a test instance and has all my test instances for Fedora maintainers on it, the other is my 'production' server and is the firewall and hosts the primary server vm that handles all my email and other services for scrye.com). I have various small arm devices, some of which are on and some of which are off at any given time.

Desktop

Since I run rawhide and update and reboot often (usually 1x a day during the week), I usually switch off between Xfce and Gnome / Fedora Workstation. I do admit Gnome has been growing on me over time and I might give it the edge these days, but Xfce is like an old friend that I can easily also do all my work in. I did play around some with sway (wayland based i3 clone) over the break, but it seemed like it would be too much work to build up a config that would be fully usable.

Mail

I still use a pretty crazy mail setup, but it keeps working so I don't replace it. Mail comes into my main server (using postfix/sqlgrey/spamassassin, etc) then goes to UUCP over vpn (openvpn) tcp to my laptop. Then, on the laptop I use claws-mail to read the email which is locally delivered by postfix. You may think I am completely crazy to use UUCP, but it really fits this use case nicely: I can compose and send emails on my laptop even if I don't have any internet, they will simply queue and send the next time I connect. Emails wait for my laptop/vpn and then deliver. I can set larger emails to smaller priority so I get the smaller ones sooner. It resumes just fine. It handles slow links just fine. I keep meaning to just switch to imap, but it ends up not really being enough win for me to care. :)

Editor

vim all the way. I used to be a xemacs/emacs user, but I haven't touched either in years. vim is just so nice and quick and works everywhere there's no contest.

Browser

firefox now. I was using midori full time, but upstream has really not had much time of late and things have gotten pretty dire. I did move the Fedora packages to webkit2 (via an unreleased upstream branch), but many things still don't work right there. I would love to go back, but it would need some real releases and people working on it.

IRC

hexchat. Anyone out there still using xchat really should switch. I have a znc server on my main server at home I connect to.

Blog

wordpress. It's just easy and works. Sure it has security issues from time to time, but as long as you keep updated you are usually in pretty good shape. I use the Fedora packages and they tend to do well at keeping up to date.

Misc

Just a few misc items: I use pass all the time (password-safe), it has most of my passwords in it. ssh, tmux, task (taskwarriror) and ansible are also things I use all the time.

Just a few rawhide notes to end out the year:

• Python 3.6 rebuild completed (mostly) and was merged back into rawhide. There were a number of things that needed cleanup, but we seem to have gotten them all handled in a few days. Next time we should really allow a day or two before we merge the side tag in and fix up things that are important for composes before we merge into rawhide.
• Astute folks may have noticed: Builds in koji now will complete on all arches, even if one fails. In the past if one arch failed, the rest were canceled and the build came back as failed right away. Now all the arches will complete before the failure comes back. This allows you to see not only the problem that failed the build, but any other problems it would have hit on other arches as well. This should be of help especially to larger builds.
• There is still a bit of fallout from the rawhide always signed changes we need to fix up in the new year: chain-builds no longer work, the rawhide fedora-release needs to be modified to always expect repos are signed, and we need to discuss what sort of testing we want to add in now that we have a handy place for adding it.

Have a smooth ride the rest of the year, no matter where your trail takes you.