The first part of 2020 (in addition to all the horrible things happening in the world) was pretty heads down in our datacenter move. Now that that is over, and Fedora 33 is out the door (hurray!), I'm slowly trying to catch up on other things I normally do that are now a bit backlogged. This weekend, I decided to try and catch up (at least some) on my package maintaining and bugzilla bugs.

In the distant past (before 2020) I basically tried to work on this stuff mostly as it came in. Update packages when release monitoring let me know there was a release, ask questions of bug reporters as they reported and try and get something actionable. Of course now I'm behind, so lets try a different process to catch up.

First, some of you may be aware that there is a python module for bugzilla (python-bugzilla), but did you know that it also includes a handy command line interface?

Lets try some initial queries and see if we can triage things to work on things in (hopefully) some semblance of their priority.

% bugzilla query -a kevin@scrye.com -s NEW,ASSIGNED  | wc -l      
102

So, 102 bugs. Not bad at all! Oh, but wait, those are only the ones I am directly assigned to, I am in a number of groups that get assignment and I am just CCed. How bad are those:

% bugzilla query --cc kevin@scrye.com -s NEW,ASSIGNED  | wc -l
412

Well, thats not great. But some of these might be bugs that are since solved or someone else is busy working on them or they have been made moot somehow. So, I will dump that query into a text file (The default output is to give you bugnumber, state (NEW,ASSIGNED), and the summary of the bug. Amusingly, I see the oldest bug I am assigned to was filed in 2011. :) So, I have those to look through and do something with, but perhaps we can make some smaller lists:

bugzilla query -a kevin@scrye.com -s NEW,ASSIGNED  | grep CVE | wc -l
3

Cool, Only 3 CVE's are assigned to me directly. 2 of those are tracking (fedora,epel) for a ansible bug thats not fixed yet and one is for a libntlm bug. Hurray, a CVE I can check on.

So, what I usually do is read the CVE bug, looking in particular for versions (this one affects versions through 1.5). I then look on https://src.fedoraproject.org/rpms/libntlm to see what versions are in what releases. All 1.5 here. Next, I look at other libntlm bugs by going to the handy: https://bugz.fedoraproject.org/libntlm In addition to the 2 CVE tracking bugs (fedora,epel), I see there's a release monitoring bug about 1.6 being available. Great. Time to do some packaging work.

So, this is a library, so we need to be carefull about changes and what might depend on it. Looking at the NEWS file for 1.6 however, we see: "** API and ABI modifications. No changes since last version." Great. Thats an update and 3 less bugs. :)

ok, on to messing with a few packages/components that more 'system wide'. epel-release, distribution, etc.

Then, lets look at packages that are needing updates:

bugzilla query --cc kevin@scrye.com -s NEW,ASSIGNED --outputformat '%{id}: %{component}: %{summary}'  |sed -e 's|^\#||' | sort -n | grep 'is available'
1271027: python-stevedore: python-stevedore-3.2.2 is available
1757650: python-feedparser: python-feedparser-6.0.2 is available
1771091: python-mechanize: python-mechanize-0.4.5 is available
1801531: awscli: awscli-2.0.61 is available
1813734: hiredis: hiredis-1.0.0 is available
1815307: bodhi: bodhi-5.5.0 is available
1829595: python-redis: python-redis-3.5.3 is available
1839836: blueberry: blueberry-1.3.9 is available
1846161: iperf3: iperf3-3.9 is available
1848739: ansible: ansible-2.10.1 is available
1862178: python-eventlet: python-eventlet-0.29.1 is available
1862531: python-vcrpy: python-vcrpy-4.1.1 is available
1880754: calibre: calibre-5.4.2 is available
1881455: python-greenlet: python-greenlet-0.4.17 is available
1886993: mxml: mxml-3.2 is available
1888249: python-pygal: python-pygal-3.0.0.dev1 is available
1890435: koji: koji-1.23.0 is available
1890954: GitPython: GitPython-3.1.11 is available
1891597: libnftnl: libnftnl-1.1.8 is available
1891769: nftables: nftables-0.9.7 is available
1893366: wordpress: wordpress-5.5.3 is available
1893453: iptables: iptables-1.8.6 is available

Some of those are blocked (calibre needs sip5 packaged, ansible needs 2.10.x packaged (which I have done, but need to clean up and submit)), but some of the others should be pretty easy. :) Lets see how far I can get today!

A few hours later...

% bugzilla query --cc kevin@scrye.com -s NEW,ASSIGNED  | wc -l
380

Dealt with 32 bugs. Not a bad afternoon I guess. :) I hope the command line bugzilla use and triage helps others deal with bugs as well.

With everything going on in the world, sadly the normal "in person" Fedora conference, flock, had to be canceled. In it's place a 'all virtual' conference was scheduled for Aug 7th through 9th: "nest with fedora"

Sadly the keynotes were all way too early for me to make it to live (4-5am my time is just too early), so I am going to catch those via recordings as soon as they are up. I did manage to make it to some sessions however.

First a quick note about the platform that was used: hopin.io. I've been to a number of virtual events of late, and most of them have not been that great. However, this time I was pleasantly surprised! hopin is actually pretty well done. There was a 'reception' area and a live chat for conference wide discussions, talks appeared a few minutes before their scheduled time to allow people to join and each talk had it's own chat. Speakers could invite other people to join in the video chat part if they wanted. There was a conference wide 'poll' section with lots of silly polls in it for people to take. Mixed in the talks were some 'hallway' track places where everyone who joined could join via video. There was also a timed thing where you could get 5min with a random other person then it would switch everyone to the next person. Not as good as a real hallway track, but not bad given the constraints. Video and audio worked pretty well.

There was a number of nice talks I went to, as well as some fun ones (CPE put on a pub quiz and a bingo, both were great!).

Smooge and I gave a really quick talk about our recent Datacenter Move. I also thought the 'meet your FESCo' talk was interesting, there were some great questions.

All in all, not as good as a real conference, but a lot safer and easier and not too bad overall. :) Many thanks to all the organizers!

I ordered one of the ubports editions of the pine64 pinephone and after a small amount of playing around with it, I'd like to share my thoughts.

First a little background. I'm very big on open source for many many reasons. I use Fedora rawhide for my laptop day to day and in general try and use free software wherever else I can. My phone has been a annoyance to me for many years now. Being completely closed source, apple/i-phones are right out for me, which basically just leaves android. Now you might think "Thats great, android is open source", but it's really not. While the source is indeed available, development is done by google in secret and dumped into the open after release. This means you don't get a lot of the advantages of open source for android. Other forks/projects do take that android source and clean it up and make it nice, but they too are at the mercy of upstream that may change things in a new release drastically, leaving them to try and catch up for months after a new release. I've been using /e/ on my trusty one plus 3t for the last 3-4 years. They are based off lineageos and 'de-google' things from there. I've never found myself very excited by it, they too are trapped by the android development all taking place elsewhere. I've looked at other possible software, but they all have their issues.

3 or so years ago, Librem announced they were going to make a phone that was as open as they could make it, with high end specs. As far as I know some few batches have been made/distributed, but it's still not a realized product. As part of this push however, software was created that could run on most normal linux distributions that could handle phone specific workflows. See https://source.puri.sm/Librem5 for a long list.

Fast forward to late last year: The pine64 folks, who have made a number of aarch64 based products successfully announced the pine phone. They produced a prototype like developer version and then, early this year announced the ubports version (some $'s of each phone would go to the ubports folks), cost: $150. The ubports version sold out and they have now announced a postmarketos version, also with a usb-c "dock", more memory and a circut board fix to allow usb-c to work right. cost: $200 with dock, or $150 without.

First lets talk about the hardware itself. It's sturdy and well built. The specs are pretty low end, but thats smart (since it's older hardware linux support is better) and not too bad considering the cost. The screen is actually pretty bright and looks nice. The back cover is a bit of a pain to take off, but I'd prefer it be that way to being too easy to remove. The phone uses a usb-c type connector for power/etc. As mentioned the ubports and braveheart (the developer release before ubports) have a pcb bug that prevents usb-c from working right. ;( There is a headphone jack, and front and rear cameras. Taking the cover off you can see: removable battery, sim and micro-sd slots and 6 "pogo" pins. These last are for future cover replacements that might have additional functions, like wireless charging or a larger battery or IR sensors. There's also 6 dip switches allowing you to enable or disable various features. I'm not sure I would use these, but I guess it's a nice feature for some folks. The ubports version has 2GB ram, the newer postmarketos one will have 3GB. There is 16GB internal eMMC. (32GB in postmarketos version). Bluetooth 4.0, wireless, and modem for LTE/3G/etc. You can get a serial console by switching the dip switch for the headphone jack 'off' and then it will do serial console. The speaker, mic, screen, wireless, bluetooth, modem, have all worked nicely for me under various software configs. Overall if the software were up to snuff, I think this would be a ok (if a bit slow) daily driver, but that brings us to software...

The pine phone has a Allwinner a64 cpu (aarch64). In theory any linux distribution that supports that cpu/arch would work. There is of course hardware specific to the pine phone that needs drivers or existing drivers that need tweaking to work with it. Due to that, most distributions are shipping a custom kernel with patches, etc. I sure hope with all the interest this stuff can get all upstreamed.

"Normal" desktops work, but they have drawbacks. For instance running Gnome on a Fedora install, many of the dialogs or windows appear too large to use and there's no way to scroll around them. One help here is the librem software mentioned above. There's a replacement for gnome-shell called 'phosh' and a replacement for mutter called 'phoc'. They are more suited to small screens and have features like forcing everything to resize to screen, etc. There's also phone and messaging apps of various stages and types. It's really nice to be able to 'dnf install' something from rawhide, all that software ready to use. :)

The pine phone will try and boot from the micro-sd card first, then the eMMC, this allows you to pretty easily try other distros out without really committing to them. :) Of course the eMMC is faster, and if you like you can overwrite it with a tool called jumpdrive. Just boot a micro-sd with jumpdrive, connect your laptop via usb and it will let you flash the eMMC.

Here's a smattering of the ones I have looked at:

• The phone shipped with ubuntu touch on the eMMC.
• The first one I flashed to a micro-sd was of course Fedora! (more below)
• Mobian (debian for mobile)
• postmarketos (I actually couldn't get this to boot for some reason)
• Arch
• Manjaro
• Probibly others I have now forgotten...

There's some places some do better than others, but mostly they seem all at the same place: some things working, others not yet and pretty rapid changes.

The Fedora (unofficial) image is linked off the pinephone fedora wiki page, I did a few local tweaks here after setting up a micro-sd card with it. The image uses lightdm, but that makes gnome or phosh use X11 instead of Wayland, so I switched back to gdm and set it to autologin as the pine user and use phosh. For some reason you have to run 'pineaudio' from the command line once before the phone speaker will work right. I think it's just in the wrong mode by default. dnf works fine. gnome-software works fine to install things.

In order for me to use a pine phone as my daily phone I need at least these things (your list might be completely different):

• Phone calls (this in general works, even bluetooth headsets seem fine)
• SMS and MMS (sms works, mms doesn't)
• ebook reading (works fine, number of apps to choose from)
• RSS feed reading from my tt-rss setup. (Newsflash works nicely!)
• picture taking/upload (Some folks claim to have working camera, but I have yet to see it with any of the images I have looked at).
• Battery life a least a day (This is claimed now depending on what you do of course, see more below).
• maps/gps for when I travel (pretty rare these days). Gnome maps works surprisingly well here.
• Twitter (I should stop doomscrolling... but cawbird has a bug to be better on mobile sized screens).
• Light web browsing (check something, settle bet in bar, look up some info). Firefox is a bit sluggish, but it works.
• OTP app ( There has to be one right? right? I guess OTPClient if someone wants to package it for Fedora)
• Music playing (usually in car over bluetooth to the car speakers)
• Alarm clock (I think this should be doable, but not sure there's a good linux app to manage it)
• Podcasts (also usually while traveling which... yeah. gPodder seems to work fine though)
• Weather (lots of options here, too bad about darksky tho)
• Very rare email checking (I would think most any email app would work here)
• IRC (oddly, hexchat has horrible problems resizing and input in dialogs is all messed up, I am not sure why. If nothing else, irssi would work fine in a tmux)
• Encryption support. The only distro I tried that had mention of this so far as postmarketos. It supposedly has a installer than can encrypt. Of course there's workarounds here too. Could boot/use the eMMC for the os and encrypt a micro-sd for data. Then in the event of theft, at least the data could be safe.

Then current blockers for me are: MMS, camera working, OTP app, possibly battery life and possibly encrypted storage.

MMS needs development work. The librem folks asked ModemManager to implement it, but they feel it should be a seperate app that interfaces with MM. Hopefully this will move forward. In the US at least, lots of things are MMS (anytime it's more than 1 person, anytime it's over a size limit, etc)

The camera is clamed working, but It obviously needs more work. Cheese can't get anything from it. Also it would be very nice if we had something like opencamera for linux. It has tons of adjustments and ability.

For OTP apps I did find OTPClient. Looks like it might fit the bill, but it's not in Fedora yet. Any packages want to take it on? :) If not, I might when I get some time.

I haven't really run the phone with a sim in it and using it off power so battery life is a bit of a question. On the plus side however, folks have gotten crust ported to it. As I understand it, this is software that runs in a secondary CPU that can handle power management outside the main OS/cpu. So, it can suspend on idle, power things off that aren't being used, etc. With this folks are claiming 100+ hours of complete standby and less of course with modem active. Suspend/resume does seem to work fine on most all of the distros I tried.

So, I am encouraged. I went ahead and ordered one of the postmarket os ones too, since I don't think I am up to the soldering fix for the usb-c and another GB memory is likely to be nice. I really hope over the next months we see things get upstreamed to the linux kernel, things like MMS and camera get sorted out, and always performance and battery life could be improved. Once the kernel stuff is mostly worked out, I wonder if there is interest/desire to make a official image for the phone? With that we could curate a selection of apps we know works and get things setup right and additionally down the road ship a version based on a stable release instead of rawhide. ;) It will be a lovely day I can run Fedora on my phone day to day.

First a bit of background before we get into reviewing. I've used a laptop as my primary computer for 20+ years, and since I do things on-line most of my waking life, this means I spend a lot of time in front of my laptop typing or reading away. So, it's pretty important to me that my laptop works well, is nice to use and is under support in case anything happens.

For the last 3.5 years or so, my laptop has been a Lenovo Yoga 920. It's been a great laptop and I have enjoyed using it. Unfortunately, it's support is going to be up in a few months and I really don't like my primary laptop to be out of support. In the last 3.5 years, Lenovo has: replaced the LCD panel when it fell off a table and broke, Replaced the motherboard when a sound connector became loose, replaced the keyboard when it became mushy, and most recently replaced the battery because it started to swell up. So, warentee is pretty important to me.

I was starting to worry that none of the current crop of laptops would really be any better than my 3.5+ year old yoga 920, but dell managed to announce their xps 13 9300 and it had some better stats, so I decided I would jump to it and see how things went. One kind of anoying thing was that dell announced the new laptop in January, but the model with the good specs I wanted wasn't available to order until April, and the "developer" edition still isn't available with the high end specs (I got the normal windows one).

On to specs. My old yoga 920:

• 16GB memory
• 512GB nvme 'disk'
• 13.9" 3840x2160 display
• ~3lbs
• 2 usb-c ports, 1 usb-2 port
• intel i7 8th gen cpu (8 "threads")
• intel UHD graphics

The new dell xps 13 9300:

• 32GB memory
• 2TB nvme 'disk'
• 13.3" 3840x2400 display
• ~2.8 lbs
• 2 usb-c ports
• intel i7 10th gen cpu (8 "threads")
• intel Iris plus graphics

So, you can see, the specs are all a little bit better (except for number of usb ports). Ordering on dells site was fine, and it took almost exactly 2 weeks from order to arrival, which isn't too bad overall. Unboxing was fine. I don't really understand why people spend a lot of time talking about that, you only do it once. :)

In order to install linux, you have to go into the firmware (power on and hit f2) and set the disk interface to AHCI from intel raid. Doing so will of course make windows unbootable, but in my case I had no reason to keep windows around so I just wiped it out. If you need to preserve windows, you may want to look at how to boot it in safe mode so it can boot with AHCI or come up with some other solution. I switched that and then booted the latest Fedora 32 RC that I had already laying around. Wiping windows and installing went fine with no issues. After booting up on the install, I then upgraded to to rawhide with 'dnf -y install fedora-repos-rawhide; dnf -y --releasever 33 --disablerepo=\* --enablerepo=rawhide distro-sync'. No problems with that either and a quick reboot later I was on rawhide. A check in gnome-software and I see a firmware update ready to go. Applied and the laptop rebooted, updated firmware and rebooted back to rawhide. Everything went very smoothly and the upgrade lookd great (progress bar, explained what it was doing, etc). Next was a copy of /home from my old laptop and then I could switch over to using the new dell day to day.

I've only run into 3 linux related issues so far, and one of those is now solved:

• Doing a shutdown/power off doesn't fully work. systemd gets everything shut down and the screen goes blank, but you can see it's still on and the fans usually come on. This is being tracked in: https://bugzilla.redhat.com/show_bug.cgi?id=1825298
• I wasn't able to get bluetooth headsets 'headset' profile to work. I have 3 bluetooth headsets here and they were all doing it. a2dp worked fine, but 'headset' would appear to switch the profile, but no sound would come out or input. I looked around on this one for a while until I noticed:

Apr 20 05:45:20 mandor.scrye.com kernel: Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-19-32-4.ddc
Apr 20 05:45:20 mandor.scrye.com kernel: Bluetooth: hci0: Failed to send Intel_Write_DDC (-22)

Turns out it was a firmware file issue, a check upstream found a commit that fixed it: https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/intel/ibt-19-32-4.ddc?id=677930a7d7422930df71adace36f17e996658f45https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/intel/ibt-19-32-4.ddc?id=677930a7d7422930df71adace36f17e996658f45

Copying that ddc file in and all my headsets started working in headset mode. :) The Fedora linux-firmware package has an update with that fix in it now in rawhide/testing for stable branches.

• Finally there's a fingerprint reader on the power button, but there's no linux driver (yet). Dell has promised that there would be something later this year (I guess they are leaning on the vendor). I sure hope it's upstreamed / usable on Fedora and not just some binary only Ubuntu driver/application. :(

So, overall so far I am pretty happy with it. In the happy list:

• The screen is lovely! Its smaller than the one in the yoga and higher resolution. I find myself switching between 150% and 175% in gnome scaling, it's a bit easier when my eyes are tired at the lower resolution, but it's really nice to have more things on screen at the higher.
• Firmware updates with no windows! fwupd is awesome. I had to keep a windows install on my yoga for firmware updates. It's really nice to not have to do that now.
• The keyboard is pretty large and easy to type on. It might take a small adjustment time to get used to, but it's not a bad keyboard at all. It's actually a bit bigger than the yoga one since it goes almost from edge to edge.
• It's fast! Things are quite zippy with the 10th gen cpu and better intel graphics.
• The wireless seems quite nice and fast. The yoga came with a really crappy atheros wireless card that I had to replace with an intel. This dell just has a good card out of the box. It's a "Killer(R) Wi-Fi 6 AX1650i 160MHz Wireless Network Adapter (201NGW)"
• Battery life seems ok. I ran it for about 3-4 hours on battery doing normal stuff (ssh sessions, reading news feeds, irc, etc) and it dropped down about 1/2, so 6-8 hours actually doing work seems not too bad considering the 4k display. The estimates did jump around a lot while I was using it, so I took the 'real world' measurement to mean more than whatever estimates it came up with.
• There is a micro-sd card reader on the left side. It seems to work just fine out of the box. I am not sure what I would use it for normally, but I suppose its a little bit of extra storage and perhaps some use making cards to boot arm devices or something.

Advantage yoga (or at least things that seem better in yoga land):

• The dell xps 13 is a regular laptop, not a 2-in-1/convertable like my yoga was. To be fair however, I almost never used this on the yoga. It's gigantic and unwieldy as a tablet. I did from time to time tilt the screen way back, which perhaps I will miss a little. I do wish the xps did tilt back more than 45 degrees or so.
• Also related to 2-in-1 vs regular, the yoga had sensors so it could rotate the display if you rotated the laptop. The xps doesn't have those. You can however do it manually, and I didn't really rotate the yoga much anyhow, so no big loss there.
• The "home" and "end" keys are on f11/f12 on the dell. This is going to take some getting used to, as the yoga sanely had them on the right and left arrow keys. This made sense to me since the up and down arrows are up and down a page. Perhaps I can remap those.
• There doesn't seem to be any way to monitor the fans. They aren't too anoying, but sometimes I wonder if they are on or not and it's hard to tell.

Misc things:

• There's a nice light at the front of the laptop where you open it to indicate when it's charging. It also can blink to show various hardware faults (no idea what that looks like yet, thank goodness). The light goes completely off when it's fully charged, white when it's charging and battery level is about 5% and amber if the battery is down to 5%.
• I don't have any thunderbolt-3 devices, but 2 usb-c docks I have seem to work fine (ethernet, various display adapters, usb, etc).
• The intel "i7 10th gen" sticker is easy to remove. :) Does anyone who uses their laptop at all keep those around?

Finally there's a few minor ways dell could improve things (IMHO):

• home and end should be on the Fn right/left arrows. :)
• more usb-c/thunderbolt 3 ports. Even one more would have been nice.
• A way to get fan speeds would be nice.
• A brightness level even lower than the minimum one they offer would be nice. (For dark rooms, etc)

So, to sum up: I hope this will be a nice, reliable laptop for me to run for the next 3-4 years. :)

I've been busy and remiss in blogging and this is something I was hoping to publish a while ago, so I give you... a review of the onlykey.

The onlykey is a USB2 connectable hardware security key. Many of you may know yubikeys, this is a competitor to those, with some advantages and disadvantages.

Ordering was simple, I just ordered one via amazon and had it 2 days later. You can also order directly from the onlykey website.

Right away you will note that this doesn't have just one key like the yubikeys do, but instead has 6 of them. Additionally, it's got a colored LED underneath, which shines green when it's unlocked, flashes red when you enter an incorrect pin, etc. The idea here is that the key will be completely locked and useless to others until its plugged in and a correct pin is entered (yes, there's more than one correct pin :) This key only comes in USB2 form as they say USBC would be too fragile. You can of course use a USB2 -> USBC adapter.

There's Several ways to manage the key:

• A chrom(imum) "app". (Although chrome is discontinuing these)
• A "snap"
• A debian .deb package

So, thats not great for Fedora. I tried to get the snap working, but it failed. I tried to use the chrome app, but thats how I found that they are dropping those, so I went with the debian package unpacked and just using the application from there. The app is open source if anyone wants to package it up: https://github.com/trustcrypto/OnlyKey-App (It's npm based)

Right off the bat I hit an issue. My onlykey had old firmware and it was old enough that the app was unable to update it, so I had to monkey around with "Teensy loader" to upgrade the firmware. At one point the key stopped even lighting up and I asked for help on the onlykey discussion group. They had someone answer me pretty quickly and I found that I wasn't properly shorting the two contacts on each end of the key to put it in 'upload firmware mode'. After I did that I managed to update to the new firmware and everything was smooth sailing after that. I really hope they have all their existing stock updated now so no one else should have to go through this. :) Just look at these little contacts you have to short together while pressing the upload button on Teensy Loader! I managed to do it with a hanger from a xmas tree ornament finally:

There's a sort of soft rubber case around the key, you can get all kinds of colors (I just stuck with black). It also comes with the handy little carribeener to attach it to your keychain or whatever.

So, once you have the firmware somewhat up to date, you can run the app. It will also update firmware as long as it's not too old. The firmware is open source: https://github.com/trustcrypto/OnlyKey-Firmware

On your first run (or if you factory wipe it), you have to do a bit of setup. You can enter 2 profile pins (sequences of buttons). They suggest that this might be 'work' and 'home', but you could use them for whatever you like. You can also enter a 'self destruct' profile pin, which wipes back to factory settings if you enter it. You can also tell it to do this if someone enters the wrong pin 10 times, but it will flash red and stop taking input after 3 failed pins. So to wipe it this way you have to enter 3 wrong pins, remove, insert, 3 more wrong pins, remove, insert 3 more wrong pins, remove, insert, 1 more wrong pin. You can also load a firmware called the "International Travel Edition" that has no encryption at all (it's only protected by the pin).

Once you have your profiles setup you can configure what you want in the slots. There's actually 12 slots because (just like yubikeys) there is long press and short press on each button. You can assign whatever you like to those 24 slots (12 for each profile). You can do TOTP, U2F, Yubikey HOTP, username/password, tabs or returns, all kinds of things:

You can do encrypted backups of the contents of the key and of course restore them. There's also some misc settings like how bright the LED is or how fast the keyboard types or what keyboard layout it uses. There's some integration with keybase.io to do encrypted files/messages transfer.

Finally, there's some advanced prefs about yubikeys and U2F tokens. I was confused by these at devconf, but the manual explains: https://docs.crp.to/usersguide.html#Yubico-one-time-password basically you have to run yubikey-personalize and have it generate "Public Identify, Private Identity, and Secret Key" which you then enter into the onlykey app. I can only assume they couldn't just do this due to legal concerns. The U2F prefs are there because:

For the attestation certificates OnlyKey comes with a default attestation certificate and signing key and also allows users or enterprises to import their own attestation certificate/key. This feature allows organizations to only permit FIDO2 keys issued by the organization to be used. Importing attestation certificates and signing keys can be done in the OnlyKey app.

The manual is pretty easy to read and covers all this better than I can probibly. So, on to the actual reviewing:

First the good:

• Open source firmware and app!
• Nice to know that if someone stole it, they would not be able to access anything using it.
• Seems to work fine for U2F/webauthn, TOTP, HOTP and user/pass.
• The LED light is nice to know what state it's in.
• I like being able to do encrypted backups.
• Having a bunch more slots is nice (If you forget which is which, you can press and hold down the 2 button for 5+ seconds and it will print out the labels you gave the slots).

Now the not so good:

• You can set a timeout where the key will lock after X minutes. It works and the LED goes off but the application if you have it open will happily think it's still talking to the unlocked key (until you try and do something then it errors).
• You can sort of use it for keeping your ssh private key, but not easily. You have to import a key (using only very specific ecc curves) and then use a 'onlykey-agent' instead of normal ssh agent. This is a fork of another project and hasn't been updated from that fork for like 4 years, which is really not encouraging. :(
• It's a bit odd trying to type in your pin at say a coffee shop or table with a bunch of co-workers around. It seems like you can't really hide what you are pressing. Still it is more secure than just any unlocked always token.

For me personally, the lack of a nice way to use it for storing my ssh private key is somewhat of a deal breaker. I really really don't want that key to get out (even though it is passphrase protected). My current yubikey was able to generate it on key and keep it always stored there. If I didn't care so much about ssh keys I might well move to using the onlykey day to day. If I traveled much I definitely would look at using it while traveling.

This year again I had the honor of being able to attend devconf.cz. Many thanks to Red Hat (My employer) for sending me to the conference (it also allowed me to attend some work meetings after the conference).

The trip out to Brno was much as it has been for me in the past, except this time it was even longer since the Portland to Amsterdam flight I used to take is no longer offered, so I had to go from Portland to Seattle and then Amsterdam. Due to various scheduling issues I also went to Vienna this time instead of Prague. No particular problems on the trip, just a long haul. The train in Vienna was nice and clean and fast and comfortable.

Devconf.cz runs over 3 days (friday/saturday/sunday) and is jam packed with talks and people to talk to. A number of talks I wanted to attend this year were full and I couldn't get in. :( Tickets also "sold" out really fast this year (in days) after they were available. I was amazed in the opening session how many people raised their hands that they were attending for the first time (seemed like >50%?). Thats really a great sign for the health of open source to have that many new people coming in and so many in general.

Friday opened with a keynote about Open Source SRE (Site reliability Engineering) in several Red Hat OpenShift teams. They did a good job explaining how they divided things up and how they tried to find issues before their customers even knew about them.

I then took the hallway track for a few sessions. So many people I work with over IRC/email there to talk to face to face. Lots of new folks as well.

The next session I hit was "Ansible by pull request: a gitops story". Some good practical story of using git and ansible and ansible tower to manage and deploy things. A good introductory talk on the concept of 'gitops' (using git for operations settings.

I then ran out for a lovely lunch at a local pizza place. Was a great chance to catch up with teammates. Unfortunately it took a while and I missed the talk on Rawhide gating that I was hoping to go to. I hear it went really well however. :)

More sessions and hallway meetings and then off to a lovely team dinner. Had a great time there and then some of us went out for more beers after dinner and had an even more amusing time.

Saturday I missed the keynote talking to various people, then off to a talk by Ben Cotton: "We won. Now what?". He talked about things going on in the open source movement these days and what they might mean or not for the future. I had actually heard of all these events, but hadn't really thought much about what they might mean, so it was some good thought provoking fodder.

I managed to miss the coffee lovers meetup sadly, but then did manage to get into the packit talk. Great stuff there: automating your package builds and tests in Fedora from upstream projects. Lots of nice questions too, which they actually had answers for! :)

Tim Burke, long time Red Hatter gave a very fun talk called "Teamwork lessions learned in 3500km hiking" about his hiking the Appalachian Trail. There were really a lot of great things in this talk, I recommend watching the video of it. Some highlights: Try and do some random act of 'trail magic' kindness each week. Don't worry about things away, just live in the moment. Lots of great life advice.

Then it was time for slideshow karioke. Always a ton of laughs. The premise: You get a set of random, advancing slides and you have to come up with some coherent talk using them. I think people are learning that it's best to come with some idea what you want to talk about, tie it into the first slide and then try and let the audience figure out how the other slides fit in. Some of them were pretty well done this time.

This year there was a group of us that wasn't too interested in the devconf party, so we just went for a nice quiet dinner and more relaxing evening.

Sunday I once again missed most of the keynote by getting stopped talking with folks and getting coffee, etc. But I managed to make the "Alternatives to modularity" talk. This was basically the same proposals as were floated on the devel list, with more detail. There unfortunately wasn't much Q&A time and the time that they did have seemed pretty harsh feedback to me. I personally think the hardest part of their alternative is getting packages to be parallel installable, sometimes thats really really not easy. Aside from that though, I think their proposal would work and be much more simple. I do hope they try and persue the approach and convince module maintainers to switch. If we see after a while there's no one using modules they will have won.

All in all another fun time at devconf. Lots of people, great talks, great area/venue. I recommend it. Of course get your tickets as soon as they go on sale as I am sure 2021 devconf.cz will sell out as quick as this year did. The number of new people really encouraged me, open source is contuing off to the next generation, really great to see!

With the opening up of EPEL8, there's a lot of folks looking and seeing packages they formerly used in EPEL6/7 not being available and wondering why. The reason is simple: EPEL is not a fixed exact list of packages, it's a framework that allows interested parties to build and provide the packages they are interested in providing to the community.

This means for a package to be in EPEL8, it requires a maintainer to step forward and explicitly ask "I'd like to maintain this in EPEL8" and then build, test and do all the other things needed to provide that package.

The reason for this is simple: We want a high quality, maintained collection of packages. Simply building things once and never again doesn't allow for someone fixing bugs, updating the package or adjusting it for other changes. We need a active maintainer there willing and able to do the work.

So, if you do see some package missing that you would really like, how do you get it added to the collection? First, open a bug in bugzilla.redhat.com against the package. If it has a Fedora EPEL product version, use that, otherwise use Fedora. Explain that you would really like the current Fedora/EPEL6/7 maintainers to also maintain it for EPEL8. If they are willing, they will answer in the bug. If no answer after a few weeks, you could consider maintaining the package yourself. Consult with the epel-devel list or #epel-devel on IRC for further options.

Do note that mailing maintainer(s) directly isn't nearly as good as just filing a bug. They would get the bug info anyhow in email, Other users might see that and add that they too want the package, the maintainer might hand off the package and the new packager could see the bug request but have no idea about private emails, some other packager might see the bug and offer to maintain it. All wins for a bug over private emails.

As the collection grows, these sorts of questions will likely die down, but it's important to remember that every package needs (at least) one maintainer.

About 18 months ago or so, I posted about switching my Mail User Agent / mail client from claws-mail to thunderbird: https://www.scrye.com/wordpress/nirik/2017/03/24/mua-or-on-to-thunderbird/ last week, I moved on to a new setup.

First let em talk a bit about why I am moving on from thunderbird. Since moving to it thunderbird seems to have gotten slower and slower (no matter how much I compacted mail folders). They made some, IMHO, anoying changes (like in thunderbird 52: ""When replying to a mailing list, reply will be sent to address in From header ignoring Reply-to header", which is just dead WRONG). Recent thunderbird versions had taken to pausing randomly and just not doing anything at all for like 20-30seconds. I had become annoyed more than once because thunderbird handled all my filtering, so if I was offline for a while and reconnected, it would take a long time for thunderbird to filter my emails and during that time I couldn't really do anything else. The final straw was thunderbird 68's changes to add-ons. Since they were moving to the newer engine, all add-ons have to be reworked to be "webextensions". Firefox went through this last year. However, thunderbird seems to have not handled it well. I didn't see any press or announcement, just updated and suddenly all my add-ons were gone. Additionally, going to the thunderbird addons site, there's a filter for version, but at the time, it didn't even have 68 listed! Pretty much non of my addons were ready for this change.

So, finally I decided I would look at mutt. I had avoided it in the past for a few reasons: I wanted to be able to see html emails easily in the same application I was using to read the rest of my emails, and I just liked the idea of a application that didn't depend on a terminal. With no real GUI MUA's left, I decided to get over those and look at mutt.

Doing some reading and pondering, I ran into a number of places talking about patches to mutt ("If you want to use this, you need patch X...") which lead me to neomutt. I don't know the details, but my understanding is that mutt development slowed way down for a number of years, and a lot of patches piled up. neomutt is a fork of mutt with all those patchsets applied, with the goal of cleaning them up and getting them into the mainstream mutt package. They already got the sidebar patches in, and hopefully they will get others in over time. Since I like living on the edge, I went with neomutt, which has a handy copr made by the main developer.

Thinking about the filtering annoyance I looked at a number of possible process changes, but finally just went with an easy one: procmail on my mail server. It's pretty old school, but it was already setup a long time ago, can filter things as they arrive and is easy to add rules to.

neomutt/mutt can of course use imap, but I like to have my laptop setup so I can read and answer emails even when off line (although that state is increasingly rare). So, I looked at syncing my imap store to my laptop. Many people seem to prefer offlineimap, but it seems it's on life support now and the developers have moved on to a newer tool that isn't really ready yet to be used. Additionally, offlineimap in rawhide (what my laptop runs) is currently broken because it uses python2 and a module it needs has been retired. So, I settled on mbsync. This is a fast, written in C program to sync between local folders and imap server (both ways). It took just a few minutes to configure and then a bit to sync all my imap folders down.

The final part of the new setup was searching. I really liked the ability in thunderbird to search everything, find threads if I just had one email, etc. The winner here is 'notmuch', a xiapan database for emails. After installing the package a simple 'notmuch new' and answering a few questions and away it went. After the initial indexing, it can (and should) be run after each mbsync invocation. You can then search from the command line or (if using neomutt) virtual folders and search queries. It much faster to come back than thunderbirds virtual folders. As a bonus all the emailed I had 'flagged' in thunderbird were still flagged to mutt, so I can continue to flag things I want to reply to and search for and reply to them later.

neomutt (and now mutt too apparently) has a sidebar setup, which is much like thunderbirds accounts/folders setup. I set mine to show any folder that has unread mail (and how many) and any folder that has flagged messages (and how many). Using that I can decide which one(s) I want to deal with. If I just want to crunch through them all, doing c (change folder) will default to the next folder with unread messages.

I of course have multiple accounts, but thats not a problem for neomutt or mbsync. Just need to define the local and remote folders and tell neomutt to read in those folders when you change to them. One issue that I was stuck on for a bit was how to filter the gmail accounts I have. (Since there's no way to use procmail on those). Luckily it seems you can get google to do it. Just need to go to the account on the web interface, select the message(s), click on the little 3 dots "more actions" at the top, filter messages like these, set it to archive automatically (skip mbox) and set a label (which makes google put it in a imap folder with that label).

So, it's been about a week with this setup so far, and I have to say, I really like it. I was off power for about 5 hours last week (power outage) and was impressed how much more battery life my laptop had without thunderbird spinning up fans and taking up CPU. Re setting up all my filters allowed me to find some things that need to be dealt with, which is nice. I think it's made me more caught up/responsive to emails than before. It's definitely more responsive/faster than thunderbird and I never really find myself waiting on the client.

There are a few things I miss from the old/previous setups:

• Calendaring. Thunderbird had calendaring built in, and now I have to just pull up a web version. I did try gnome-calendar for a bit, but it doesn't allow you to see who attendees are or answer yes or no or maybe on going to the meeting. So, web interface it is for now.
• Thunderbird could display annoying html only emails well in the interface, but... I always prefer the text/plain part, and turn off remote image loading, so many of the most broken emails werent that pretty anyhow. Additionally, mutt can easily open those parts in a browser or the like.
• When you are composing an email in neomutt, you can't super easily just go and look at other emails. Thunderbird having a seperate window for this was nice. You can start another neomutt -R (read only), but that seems like a hassle. You can of course postpone your email, go look at whatever else, then resume it, but still a bit of a pain.
• My old setup used spamassassin on the mail server to detect spam, but in addition I was using thunderbirds bogofilter support as a 'second line'. Without this I have been getting more spams to my inbox (mostly to list moderator addresses).

And there's a number of TODOs left:

• I'm trying to decide if there is any point in keeping old email list posts or bug tracker emails. I was going to archive them, but then I wondered: why? I can get them from the list archives or the bug tracker application. Is there any great point in having more than say 90days of them around? Still pondering on that...
• Right now I am running 'mbsync; notmuch new' manually (I bound it to f1 in neomutt) when I want. I wonder if it's worth setting up a cron job? Right now it takes about 90 seconds to run (mostly because gmail is slow), but not sure any interval matters. I'm likely to pull before I start answering emails.
• There's still some weirdness around switching accounts. It seems like if I start neomutt and never go to an accounts folders, I never see new emails for that account until I do. Should figure out what I might be doing wrong and file a bug if it seems like it's not my fault. :)
• A full mbsync takes like 90 seconds. If I nuke a bunch of old emails (see point 1 above), I think I can reduce that time. Also, I looked at doing clever things like moving a bunch of old things to archive and not syncing them all the time, but that requires a lot of manual work, which I would like to avoid.
• Look at setting up a local bogofilter or second line spam handler to reduce spam even more.

So, all in all, I am pretty happy with this setup. Happy to share any config or help others setup the same mail workflow. We will see in a year or two how well it sticks.

Flock time is upon is! This time in lovely Budapest. As always when flock is in europe, it's a long flight for me, but otherwise travel was uneventfull: Drive 2 hours to PDX, then PDX to AMS, then a short layover for coffee and stoupwaffles and then AMS to BUD, and finally a taxi ride to the hotel.

The hotel is quite lovely. It's right next to the danube river and has a nice view. The AC is working nicely too (it's quite hot outside here right now). After getting into the hotel yesterday and a quick dinner at a very nice place down the road, I managed to sleep for 10+ hours.

A nice complementary breakfast, then flock. I went to:

• The FPLs yearly talk on the state of Fedora. This went over a lot of decisions and changes the Fedora Council made early this year around mission and how to better try and acheive and articulate our goals. Then some nice charts and graphs, and finally a bunch of things for us to ponder on how to solve.
• Next up was a coffee break. I Love that we have these scheduled, it's a great chance to visit with fellow fedorians.
• The next big talk in the main room was Cate Huston from wordpress talking about how we all fail at projects and some things to try and fail less. It was a nice talk. Lots to think about, and definitely some things that resonated.
• Final talk in the big room for the day was from the Facebook desktop support team about them rolling Fedora out as a supported desktop for their users. They went over their setup, some problems/issues they hope to work with Fedora to solve and took questions. Very interesting.
• Nice lunch in the hotel resturant, then back to flock...
• I went then to the Improving the Packaging Experience with automation. Some great ideas there, we just need people to try and drive them and spend some cycles working on them.
• Then on to the Fedora Workstation status update and progress. Lots of great stuff here, most of which I knew, but it was great seeing all the progress in one place. Keep up the great work!
• The next few timeslots I got pulled into the "hallway" track. ie, discussions with friends and co-workers and community members (and sometimes all 3 in one person!).
• Next I went to Ben Cotton's session on changing the changes process. I think the changes he wants to do will help out and make things run smoother. He had a nice overview of changes and there was a good bit of discussion with the audience.
• I sadly missed the "slideshow kareoke" session as I stopped to talk to people.
• The nights event was board games and relaxing, so we gathered up everyone from the CPE team that was there and had a very nice group dinner at a local place.
• Then, back to the hotel and time to sleep...

The next day started out with a "Fedora, Red Hat, and IBM" talk from Denise Dumas. I've heard most of the content before, but she put a nice Fedora spin on things.

• Next up I went to a panel talk lead by Brendan Conoboy about how RHEL8 was formed and where and why it diverged from Fedora. Not surprisingly, we want to try and merge things back as closely as we can to make RHEL9 smoother.
• Lubomír Sedlář's talk on making composes faster was definitely something I had to attend. In addition to the ideas he had, the audience suggested a few more and I hope everyone went away with a good understanding of how composes are made.
• Leigh Griffin had a nice talk about my team (CPE) trying to become more agile. I really like his approach to find what works for us and not just apppoint a scrum master and make everyone go through some ceremonies. What we end up with will depend on what we want and how we work, there's no one-size-fits-all.
• Laura Abbot gave a talk about the kernel tree of the future. There's a lot of will to move things to a better workflow than they are today. I hope they can get it to a nice place with some hard work.
• Finally up at the end of the day was my 25min communishift talk. The room was packed and there were tons of questions (which I hope I answered). Communishift is out community openshift cluster for Proof of concepts, development, tinkering or the like from the community. We are only responsible for the platform, users are responsible for everything above that. It seemed to get a lot of good feedback.
• The evening event was a dinner boat ride on the river. The dinner and the boat and the company was great. There was a bit too much walking to get there in the heat, but we all survived.

Saturday started off with a showcase from the outreachy and google summer of code projects. It was awesome to see some of the projects they had worked on and the progress they had made. great stuff!

• Most of the rest of saturday I spent talking to people and discussing ideas or problems. Got a lot of requests to add people to communishift, which I did later that night.

Sunday started off with a CPE hackfest. I worked it like a barcamp, so we tossed out ideas and topics of things we wanted to go over and voted on them. The top two were: "going over backlog of projects" and "improving the packager experience". We split the room into two and started working away.

• We got a ton of requirements / needs for a new monitoring / metrics setup. There was a lot of good discussion from people not on our team, but interacting with us on what they might want/need.
• There was some discussion about how to try and move forward on our next generation account system. Stay tuned here there's several good ideas being explored.
• I finally got to talk to the copr folks and come up with some plans to move copr forward from being on our old cloud. Should be some good movement in this area soon I hope.
• We got some progress on the CPE docs. Fixed some things and got them generating as we push changes to them.
• After lunch we had the "meet your FESCo" panel. There were lots of questions and I hope a good discussion. Everyone got some good mic time and it went pretty well.
• Finally the wrap up session, everyone who lead a session came up and gave a short overview. It was great to hear this for the sessions I couldn't attend. Finally there was a SUPER great video made by tatica of a bunch of people at flock. We really are a diverse bunch.
• Sunday night I had a final dinner with the team and then up at 4am to head home. Left the hotel at 4am, got home about 3pm the same day (and a ton of time zone changes in the way).

All in all another great flock in the books. This one seemed to have more energy / excited folks than I have seen in a while, and as always it was good to see old friends. Until next year!

We have been working away at getting epel8 ready (short status: we have builds and are building fedpkg and bodhi and all the other tools maintainers need to deal with packages and hope to have some composes next week), and I would like to introduce a new thing we are trying with epel8: The epel8-playground.

epel8-playground is another branch for all epel8 packages. By default when a package is setup for epel8 both branches are made, and when maintainers do builds in the epel8 branch, fedpkg will build for _both_ epel8 and epel8-playground. epel8 will use the bodhi updates system with an updates-testing and stable repo. epel8-playground will compose every night and use only one repo.

The idea here is that maintainers can (if they choose) modify the package.cfg file in the epel8 branch to only build for epel8, and then use the epel8-playground branch (and it's sperate repos/builds):

• To test out some new version of the package that might not be stable yet.
• To test out some new packaging of the package
• To test a major version change of the package that they want to land at the next epel8 minor release.
• To build a package that will never be stable enough for epel8, but still could be useful to some.

At minor RHEL releases (ie, 8.1, 8.2) people can pull in big changes from playground to the main epel8 packages. Since people will be upgrading and paying more attention than usual anyhow at those points, it's a great chance to do that change, but also you want to make sure it's panned out, so you can test before hand in playground.

We hope that this feature will be useful to some folks. Do let us know on the epel-devel list or in our next EPEL sig meeting!