Skip to main content

Pinephone pro

Once again it's been a while since I've posted, but I am getting a bit caught up with life and am going to try and resume some more regular blogging.

Lets start with a quick post about the Pinephone pro. I had gotten not one, but two of the orig pinephones. Nice fun to play with, but so many non upstream patches to get everything working made it pretty much impossible that it would ever be able to have a Fedora spin. Also, it was really quite slow at pretty much everything. ;( Enter the pinephone pro that was announced late last year.

I signed up for the very first batch of pinephone pro's and managed to get one! First, for some reason, the shipper (DHL international) decided to use USPS for the last leg of the shipping to my house. This is not ok, because I don't actually have USPS service here. :( I did finally manage to get a hold of the local postoffice to go in and claim my package from them.

The pro is very similar to the older pinephone in size, appearance and also accessories like batteries, usb dock and such. It's a LOT faster than the old pinephone. Still probibly not up to a top tier phone, but it seems pretty similar in performance to my daily driver phone (a OnePlus 3t running /e/ os). Also, much more support is already upstream or more easily upstreamable.

The pinephone pro has a SPI (apparently all generations, although I don't have a cite, this is just what I have heard). A SPI is a small flash area you can place a boot loader. There's a fork of the uboot loader called 'towboot' that many folks are flashing there. towboot has nice support for a phone device. It lights up the led based on whats happening (before the screen), it lets you use volume and select to decide if you want to boot the eMMC (internal flash) or microsd card, etc. I finally broke down and ran the towboot installer. Worked fine and it's nice to have a bit more control over the boot process. A group of folks are pushing pine64 to just ship with towboot on SPI in later batches, we will see how that goes.

Thanks to javierm, we have a Fedora kernel and uboot with pinephone pro patches on top in copr now, along with osbuild templates to use that kernel/uboot to make a raw image for the phone. There's more patches to test and then all of them to upstream, but progress is being made.

Next steps for the mobility sig are to start getting a change together, update the phosh comps group, and get kickstarts together (in case we can land before fedora uses osbuild). Progress is slower than anyone would like, but I hope later this year we will have a pinephone pro image (or two)!

Some quick framework laptop power saving tips

Some of these may apply to all laptops and some may be frame.work specific, but I thought I would throw them out there to help folks out.

  • Remove any of your modules that aren't usb-c. At least the microsd and HDMI modules pull a small amount of power and prevent the laptop from going into the lowest power saving levels. Without doing this, I never get pc10 (the lowest power saving mode). framework is working on fixing this in a upcoming bios update.
  • Turn off the camera and microphone with the switches at the top of the display. Again, these draw a small amount of power and keep things from going to lowest power settings.
  • dynamic power profiles powersave mode is much better in my testing than tlp or tuned for saving power. It's also easy to manage with workstation/gnome desktop. Others can use the powerprofilesctl command line tool.
  • powertop --auto-tune (or simply 'systemctl enable powertop' and reboot) seems to work ok on the framework laptops, with one exception: If you disable bluetooth and then try and re-enable it, it will find no controller. You will need to do a cold boot (completely power off and back on) or 'rmmod btusb; modprobe btusb enable_autosuspend=n' to get it back.
  • 'echo powersupersave > /sys/module/pcie_aspm/parameters/policy' provides some power savings and to me at least doesn't seem to cause much in the way of slowdowns.
  • Make sure you aren't hitting this bug in firefox ( https://bugzilla.redhat.com/show_bug.cgi?id=1936901 ) which basically causes it to start speech-dispatcher and then never ever let it go away. This was causing about 3-5% cpu use here, but it also causes fans to start up and no power savings to happen.
  • If you don't mind using rpmfusion-nonfree items, installing intel-media-driver from there helps video playback nicely (depending on the video).
  • bluetooth and wifi both use a fair bit of power, if you don't need them, do disable them for more savings.

Hope that helps someone out there get more battery life... :)

Frame.work laptop: The hyperdetailed Fedora review

Frame.work is a new company that appeared eariler this year. They announced that they were going to make a laptop focusing on repair-ability, with replaceable interface modules. I didn't really need another laptop, but I was very interested in the idea and wanted to support them, so I preordered one as soon as preorders started up. That laptop arrived here a few weeks ago, and I used it as my main laptop last few weeks.

Framework offered several already assembled models, as well as a "DYI" edition where you could get the base laptop and as many or as few add on parts as you liked. I of course went for the DIY edition.

The packaging was the typical these days recycled cardboard and paper, but all very nicely packed and professional looking. In the box was the body of the laptop itself (the screen/keyboard/motherboard) and then small packages with the parts: memory, wireless card, nvme storage stick, and modules. Framework wanted to ship the DYI edition completely disassembled so you had to do a lot more assembly, but it turns out there's a bunch of silly rules about laptops vs laptop parts and if they shipped the unassembed version they would run afoul of customs and shipping and taxes rules. There's no instructions or manual in the box aside from a small "user manual" that just describes the features of the latop (where the switches and ports are) and has a bunch of regulatory information. It also comes with a torx tool/screwdriver/spludger, which is a nice touch.

After unpacking and looking at everything I poked around and found https://guides.frame.work/Guide/Framework+Laptop+DIY+Edition+Quick+Start+Guide/57 that describes how to assemble things. The hardest part was getting the wireless card antenna connected and running the right way, everything else was a complete dream.The captive screws for the cover was a super nice touch. I managed to get everything installed and assembled in about 10min (and that was with some kittens wandering around distracting me :)

Some general pictures and observations on hardware:

  • In general as many have noted the laptop reminds heavily of the older macbook pro 13" models. Might be the aluminum case, or the keyboard layout or just everything. Thats not a bad thing, just an observation. The laptop seems solid and well constructed. Some reviews said the hinge was somewhat flimsy, but the one I have doesn't show that at all. It seems quite solid and workable.
  • frame.work laptop
  • The power/fingerprint reader button in the upper right has a light/glow around it when the laptop is on. Currently there isn't any way to adjust this, but apparently its hooked into the EC so it might be possible to get firmware or linux able to adjust/disable it. It doesn't bother me, but some folks don't like it in dark rooms.
  • You can charge from any of the expansion card ports if you have a usb-c card in there. On both the right and left side in between the expansion card bays is a charging light. So, if you plug power into a bay on the right, the light on the right will light up to indicate charging.
  • The expansion cards are just usb-c to whatever. You could just connect directly to the usb-c connector in the bay if you wanted to for some reason. The usb-c expansion cards are just basically passthroughs. frame.work has released 3d models for the cards and is even funding people's production of card ideas. See https://github.com/FrameworkComputer/ExpansionCards for all the details. Someone has even already come out with a 'storage' card (bascially lets you store things like microsd cards or 3 m&m's or something). See https://community.frame.work/t/the-snack-drawer-has-arrived/5757 framework is working on a ethernet card.
  • There is a headphone jack... for people who like that sort of thing. I doubt I will ever use it.
  • You can tilt the screen all way back, 180degrees (or a bit more). I don't often need that, but the dell xps 13 I have only goes about 120degrees and it's sometimes anoying.
  • The keyboard backlight has 4 settings, which you can step through with function space. low, med, high and off.
  • The hinge is kind of square and has a gap behind it when opening. Until I got used to it I found it to sometimes pinch my legs when opening it on my lap. This seemed to only happen the first few times I opened it so not really a big deal.
  • The camera and microphone have disable switches (showing red when disabled).
  • There's a light sensor next to the microphone on/off switch.
  • The firmware is a "InsydeH2O" (which I think is a 3rd party that ibm also uses on some of their laptops). Framework has expressed interest in getting coreboot working. There's a complete firmware guide at https://community.frame.work/t/bios-guide/4178 They also are going to release the EC part of their firmware
  • Framework has a 'testing' account on fwupd and plans to release firmware updates there.
  • You can boot from: nvme, usb stick, microsd in the microsd expansion, and any of the storage expansions.
  • There is a small button between the expansion card bays on both sides on the bottom of the laptop. That button is a release then you can extract a expansion card. I've found this to be somewhat difficult, which is ok as I wouldn't want cards sliding too easily out, but it could be a tad easier to extract them.
  • The screen is a '2k' screen. 3:2 aspect with 2256x1504 pixels. It's a nice bright screen, but I do miss the 4k screen on the dell xps 13 I was using (more on that below)

I choose the middle of the 3 cpu options available. There was a i5, a i7 midrange and a i7 top end. The top end i7 was quite a bit more, so the middle option seemed good to me. For storage I went with a midrange option too, a 512GB WD nvme. Finally for memory, I saw that the laptop supported 64GB, and since you can never have too much memory thats what I went with (2 32GB dimms).

There's lots of things framework plans to do moving forward, but I am sure all of those depend on how well the laptop sells. They did sell out of their first batch thought, so thats a good sign. A few things that they have in the works:

  • As mentioned above, they are working on a gigabit ethernet expansion card. I will definitely get one of those.
  • Also related to expansion cards, they are planning a 'marketplace' for people to produce and sell whatever crazy cards they make. Also in this marketplace will be replacement parts for the laptop and things like different colored display bezels, different keyboards (they are planning a transparent one ,which could be fun).
  • They have gotten tons of requests for AMD cpus or arm cpus or something else cpus. So, hopefully they will be able to ship new replacement motherboards someday with these. I would jump on a aarch64 mainboard in a second. :)
  • They have also gotten requests for different display options. I might be interested in a 4k display.
  • Finally there are folks wanting a 15" form factor. I have no idea why anyone would want that, but I suppose they might release one.

Moving on now to software. The DIY edition doesn't come with any OS, you can choose or use your own.This is of course exactly what I wanted. A quick download and boot on the Fedora Rawhide workstation image on usb stick and a install and Fedora was up and running nicely. Since I used rawhide I avoided two small gotchas for stable release users: You need a newer kernel to support wireless/bluetooth (use a Fedora respin) and a newer fprint package to get the fingerprint reader working.

Some software impressions:

  • Aside the wireless needing newer kernel and fingerprint reader needing newer fprintd, Fedora workstation just works on the laptop.
  • The screen is nice and bright. I do wish there was a 4k option, but that of course would affect performance and battery life (see below on that)
  • Everything is really fast. The Xe graphics + fast cpu + enough memory to cache everything you do makes this laptop a joy to use.
  • Switching between this laptop and my dell xps 13 9300 make me realize how little state I actually store locally. Newsflash/rss feeds are all in my tinytinyrss server, hexchat messages are all in znc on my main server, mail is synced from main server to both laptops with mbsync, and as long as a sync happens when I switch everything is on the other laptop, element, discord all store state elsewhere. Firefox does have local state, but I can share links easily so that was not a big problem. Git repos? I don't tend to have things I work on for a long time in git repos, so as long as I finish and push a pr or commit I can just pull on the other laptop and be in sync.
  • Fingerprint reader works great (apply updates if you are on f34). I'm not sure I will want to keep using it, especially when traveling, but it's kinda nice here at home.
  • kvm / virt works fine. Installing a Fedora 35 vm took just a few minutes and had no problems.
  • The default suspend level is s2idle, which uses quite a lot of power while sleeping. You can switch to 'deep' sleep mode by doing a quick grubby call to update kernel command arguments and rebooting:
grubby --update-kernel=ALL --args="mem_sleep_default=deep"

Now, lets talk about battery life. There's been some reports that battery life is really poor, and I can understand why people think so. In the early days of laptops, the laptop consumed about the same amount of power no matter what it was doing. This made is easy to show how much battery life you had left and estimates were pretty accurate. Fast forward to todays modern laptops and... it's utterly different. You have a ton of adjustments you can make: The brightness of the screen can reduce power use, what you are currently doing on the laptop matters much more, and the cpu has a million things it can do to save power or not. Sadly, it's not even as clear as that however. Consider you are on your laptop on battery power. You need to finish a spreadsheet. So, you want the cpu to run as slowly as possible, but... when you go to recalculate a bunch of things, the cpu going slow and taking 5minutes to do something it could do in 15seconds on higher power use actually net drops you power. It would be much better to be able to burst up and do something quickly and use that power for 15seconds than it would be to not and use 5minutes of lower power. You also have to take into account fans and heat and GPU and... it's complicated. So, all this complexity and knobs you can adjust result in seeing accurate estimates of how much battery life you have left, but only if you are doing _exactly_ what you are doing right now. There's a ton of linux tools related to power management now: thermald, powertop, tlp, tuned, and the newest entry: power-profiles-daemon

On the framework laptop on battery, I can see anything from 2hours (in google meet video call with bluetooth headset on, camera on, performance setting) to 14hours (low backlight, camera off, openvpn off, keyboard backlight lowest, powertop auto-tuned, powersave dynamic profile). Powertop is quite useful for tracking down things that are using power. Using it:

  • The usb camera draws a ton of power, even when not in use (this might be because it's not auto suspending correctly?) But you can fortunately just use the switch next to it to disable it when you want to save some power.
  • Folks on the framework community site are seeing a similar thing with the HDMI expansion card.
  • I was also seeing similar with the microsd card expansion card.
  • bluetooth also draws some even if you aren't connected to any bluetooth devices (sub 1w)
  • power profiles are by far the best method to get more battery life. Moving from performance mode to power-save adds about 3 hours to the estimates.
  • If you do 'powertop --auto-tune', beware that btusb doesn't handle auto suspend right and if you reboot it will load, but not work at all. You then need to rmmod btusb and 'modprobe btusb enable_autosuspend=n' to get it working again. It won't work even after reboots until you reload it without autosuspend. That one confused me for quite a while.
  • For some reason my work vpn (openvpn) uses a fair bit of power. Wireguard which I use for home seems much more energy efficent here.

In the end battery life is going to vary depending on what you want to do and how long you want it to take. I was planning on seeing how well I could do on battery during a 'normal' day, but I didn't get to it. I suspect many people never run their laptops on battery (or just when moving from place to place) or only use battery much when traveling (remember when we used to do that?). I admit to being kind of in that boat myself... the only times I am on battery is when I go outside on the deck or am needing to do something in the car while someone else drives. The battery life on my xps 13 9300 is much worse than the frame.work laptop. It makes some sense however, as it's a older generation cpu, a 4k screen (sucks much more power) and a smaller battery.

So, speaking of the two laptops, how do they compare? Well, on the pro xps 13 9300 side:

  • smaller / lighter
  • no bezel around display
  • display is 4k
  • display is a touchscreen. I admit to almost never using the touchscreen. The only exception being miro boards... somehow touchscreen helps navigating those.

And cons of the xps 13 9300:

  • battery life not good
  • only 2 usb-c ports, thats it. (well, microsd also I guess if that counts for anything)
  • graphics are kind of sluggish
  • no fingerprint reader (it has one, but doesn't work under linux)

Pros for the frame.work:

  • Better battery life
  • fast fast fast. Both cpu wise and graphics wise.
  • Screen is lower resolution, but I think it's brighter/better colors
  • camera is higher res (1080p vs 720p)
  • Repairability/Replacability is much nicer. I spent an entire day replacing the keyboard in my dell.

And Cons for the frame.work:

  • Lower resolution screen, and 3:2 is weird after using 16:10
  • larger
  • bigger bezel around display

So, which laptop will I use day to day? I'm not fully sure yet. :) I might keep switching or see how I feel after longer.

Overall the framework laptop is a pretty darn nice laptop. I really hope frame.work is around for many years and we get to see neat things like new cards, new motherboards, new screens and so on. A aarch64 motherboard would be fantastic!

Wordpress openid is somewhat fragile, so if you want to ask me anything about this review, feel free to ping me on mastodon: @nirik@fosstodon.org or drop me an email: kevin@scrye.com

References / more reading:

Nest with Fedora: 2021

Last week was the 2021 edition of Nest with Fedora. Nest is the all virtual version of Flock, the annual Fedora community conference. While sad we couldn't all get together in person again this year, the virtual version was fine. The platform again used was 'hopin' and it did a reasonable job overall. I did have some problems sharing my presentation for my talk, but otherwise I think everything went smoothly. A few of the talks I went to and some thoughts on them:

  • State of Fedora keynote: Nothing earth shattering here, but good that things all seem to be going up and to the right. :)
  • Fedora Council Town Hall: Some good questions, next time it might be good to go round robin as I think there were a few council members that didn't talk and some that talked a lot.
  • Meet CPE: Good overview of how we do things and whats going on from my teammates.
  • 6 tips working with communties: I think these were all things I knew, but I think it was a good talk for those that don't know or haven't thought about how open source communities work.
  • My Pinephone talk: I had technical problems to start with, but after that it went smoothly enough. I was worried that I would have too much time to fill, but I just had enough with some questions. (More on this talk below)
  • Exploring our bugs: Lots of interesting charts and graphs. I am still not 100% sure what they might mean. I wish Fedora had a data analysts SIG or something that tried to ask questions and answer them with data. You could spend a LOT of time in this. :)
  • CentOS and Alma: Nice talk about how things were going in both projects and how they collaborate.
  • fbrnch a year on: I knew about fbrnch, but I haven't used it. This talk made me put playing with it on my list for sure. Lots of things it can do for you/simplify workflows.
  • But it's all 'upstream', why doesn't it work on fedora? great talk from Peter about different levels of upstream and how you need to integrate things in fedora, not just assume they will all work together when updated.
  • Hassle-free Throw-away VM's with Testcloud: I missed the start of this talk, so I really want to watch the recording later and get the basic info. I was a bit lost lacking that info. :)
  • Fedora Trivia Pub Quiz: Fun as always. I never win.
  • Fedora Project Leaders of Legend and Lore: This was _AWESOME_ to see (almost all) the ex-fpl's in one screen and answering questions. I had to leave about 1/2 way through this talk, so I am looking forward to the recording to watch the rest of the fun.
  • Giving and Receiving Feedback: This was a great talk, I hadn't really though about feedback from all the angles, and changing the dynamics so you are on the same side trying to reach a goal is brilliant. I hope to use this and to see if we can make it pervasive in out communities. Also ended up reading thought a bunch of related blog posts and twitter threads... great stuff!
  • Building Initrd Images from RPMs: A plan to make initramfs from rpms instead of using dracut. I'm ok with this plan.
  • Making Your Life Easier with the Packager Dashboard: The packager dashboard is great, I hope this gets more folks using it. I have been using it myself for a while and it's really quite nice for package maintainance.
  • Cross Collaboration Panel: This was interesting, but might have been more interesting from a 'each person tell your collaboration story' instead of just trying to answer questions from the audience. Still, cross distro collaboration is a great thing and we should all strive to do it when we can.
  • Closing Remarks: Most signups and attendees ever. It seems clear even if we do in person next year, we should either _also_ do a virtual conference or some kind of hybrid. It's just too handy to not have to travel. :)

My talk on the pinephone seemed to go well, but it's hard to tell. :) I tried to put forth the idea that because smart phones are slowing down (people want longer support, don't replace their phones every year anymore, no new superfeatures, % sales lower year over year) this is a great time to move a open source community into the phone space. It's very similar to the pc market in a lot of ways. I went over the pinephone hardware (will never be super fast, but at ~150$ it's reachable to hack on from everyone) and hopefully someday may be a daily driver for some. We really need upstreaming of some connectivity so we can start making fedora images: wifi/bt/modem/usb-c, etc. Hopefully that will happen soon. Userspace is coming along fast with Phosh/plasma/other desktops and applications.

Overall a nice time, many thanks to all the organizers again this year!

A critique of google chat

I'm not usually one to say bad things about... well, anything, but more and more people I interact with lately have been using google chat at their preferred communication medium and I have been asked a few times why I dislike it, so I thought I would do a blog post on it and can then just point people here.

First a few pros before any cons:

  • It works fine in firefox as far as I have seen
  • It's included in "google workspace', so if you have that you have google chat too.
  • It looks ok. Links, emojis, reactions, links, videos all look/work fine.
  • It's easy to start a chat with multiple people or groups
  • All your chat history is there on restart (but see below)
  • It's encrypted (but no idea if google can read all the history or not)

And now all the actual current cons:

  • It's a browser only application. If I don't remember to reload that tab after rebooting, it's just not there and I get no notices. Needless to say: no 3rd party applications have a chance to do better than the web interface.
  • It's thread model is utter junk. There are 'threads' of conversation, but they are all in the same linear timeline, so its really really hard to tell or notice if someone added something to an old thread without knowing you need to scroll back up to it.
  • It's horribly "unreliable". By this I mean I could have the app loaded in my browser and _still_ not see notifications because: google is randomly asking me to login again, google chat decides to put <--- and 100 more lines here --> collapsed section, google chat randomly decides to put "and 10 more messages" button at the bottom, Someone could have started a new thread and the one you care about has any of the above happen to it and you never know.
  • Someone you are messaging could just not use google chat or be busy or away. Google chat will helpfully mail them about your message... but 12 hours later. You can also say you are 'busy' but only for 8 hours at a time. If you are gone for a week, too bad.
  • Surprisingly search is kind of bad. It finds things you search for, but there's no way to go to the 'next' match or see how many or navigate at all in it.
  • Related to search, there's no way to have logs locally to use unix tools on like grep, etc.
  • There's a indicator of how many messages are unread in a room/chat, but no indications if thats just general or if someone mentioned your name.
  • No good keyboard navigation, you have to click around to change channels, threads, conversations, etc.
  • Unless you have been invited, it's virtual impossible to find rooms. There's a 'browse rooms' dialog, but it only lets you search in the room name, not description or who is in what room or anything.
  • Depending on how things are configured, you may not be able to chat with everyone, only people in your company or workspace.
  • You can adjust notifications per room somewhat, but it's not as flexible as any IRC client. You can notify on everything, or on just mentions of your name, or mentions of your name and every new thread or nothing. No way to look for keywords or anything.

And of course there's potential future cons:

  • google changes chat products more than a bakery changes it's day old bread. Is this iteration of google chat going to stick around? Who knows? Could be a completely rebranded and different one next week. ;(

So, all in all, I will use google chat if I have to, but it's going to be my least favorite chat method and I wish everyone would switch to something else. Matrix perhaps?

Ansible and Fedora/EPEL packaging status

Just thought I would post a current status on ansible packaging in Fedora/EPEL

ansible 2.9.x (aka, "ansible classic") continues to be available in EPEL7/EPEL8 and all supported Fedora releases. Odds are most people are just still using this. It does still get security and some small bugfixes, but no big changes or fixes.

ansible 3.x (aka, "ansible-base 2.10.x + community collections"). I had packaged ansible-base in rawhide/f34, but due to the naming changing and lack of time, I have dropped it. ansible-base is retired now in Fedora and likely never will land there.

ansible 4.x (aka, "ansible-core 2.11 + community collections"). I have renamed ansible-base to ansible-core in rawhide. Unfortunately, a dep was added on python-packaging, so there's 6 or so packages to finish packaging up and getting reviewed. The collections are a bit all over the place as people have been submitting them and getting them in. You can find the ansible collections via 'dnf list ansible-collection\*'. After I get ansible-core in shape, I am going to look at packaging up at least the rest of the collections for 4.x. At that point we could look at dropping ansible-classic (or moving it to 'ansible-classic' and shipping ansible-core + community collections as 'ansible' 4.x. Note that collections work with both ansible-classic and ansible 4.x.

ansible-core 2.12 (I don't know if this will be in ansible 5.x, but I guess so?) will REQUIRE python-3.8 or larger. So, EPEL7 will probibly never move to this. EPEL8 may, but it might be tricky to use the python3.8 module for this.

So, progress is being made, all be it slowly. :) I'll post again soon hopefully to announce that ansible-core is usable/testable in rawhide.

Pinephone and Fedora

Greetings everyone. I thought I would just write up a quick post on some current status and tips and information around running Fedora on pine64 pinephones.

First, let me talk about scaling. One of the problems putting a desktop OS into a small screen on a phone is scaling. Phosh (a librem started gnome-shell replacement for small screens) and Phoc (a mutter/window manager replacement that works with Phosh do there best with this issue. There's a setting to try and resize all windows from all applications, and a way to do it on a case by case basis, however many applications are just not friendly to small screens. They refuse to shrink below a point, or they cut off valuable parts. I guess this might be something thats best solved upstream at the toolkit level, but it's a hard problem. By default Phosh sets 200% scaling on the pinephone as well. It all depends on how small a screen/type you can handle, but lowering that gets more applications usable. You can do so via: 'wlr-randr --output DSI-1 --scale 1.25' for 125% for example. This also makes it harder to press buttons, so beware. :)

Next some great news. The latest round of uboot updates in Fedora Rawhide get my 3GB pinephone booting from a stock rawhide workstation image. Things boot, the display comes up, gnome comes up (and is really difficult to use, see above scaling issue). There's no net, no modem, no camera, etc, but the display and the base boot chain all works great. Many thanks to Peter Robinson for getting those patches in!

A short note about flatpaks: I installed some flatpaks from flathub, and they work just dandy. The Fedora flatpak registry should soon also be advertising all it's flatpaks for aarch64 as well. This is a great step for us down the road when we start working on a ostree version.

On the remix front, more things are getting reviewed and added to Fedora repos. chatty (sms/mms/matrix/jabber) client just finished review. There's only a few more things on the list, but then the big one: The kernel patches. Hopefully we can find some folks to help us upstream things so we can get a vanilla fedora kernel usable. Even just getting in the patches for wifi or modem would be a great help (allowing you to ssh in and apply updates).

I'm still not using mine as a daily driver yet, there's stil some important things not there that I need:

  • MMS handling (in progress via a mmsd and chatty, but not there yet). For those in EU, in the US all group chats and anything with media uses MMS (at least on my carrier)
  • The camera now works, but it's still really not good at all. I'm hoping for improvements.
  • There's so far no good otp app packaged up. Will have to look at them and package up the best one.
  • Lots of small stuff thats nice, but not urgent: the torch mode, auto rotation, docking detection

But a number of things are looking better:

  • Sound works fine along with bluetooth and gpodder, so podcasts are good. (Side note: if you set the dip switch for serial console, it messes up the audio, you need to reset it back to get working audio. ;)
  • epiphany works pretty well here as a web browser. Resizes nicely and is pretty fast.
  • neochat seems the best of the matrix clients here (but chatty is adding matrix support, so will be interesting to see that)
  • newsflash works pretty great for rss reading (although you have to actually read each article to mark it as read. The android tt-rss reader I am using on my android phone lets you just scroll by things to mark them read.
  • gnome-maps seems to work fine, except no GPS. There's some AGPS upload song and dance I haven't looked into yet. Hopefully that can get automated so it works out of the box.
  • tootle works fine for mastodon.
  • cawbird works ok if I don't give up twitter entirely.

Finally a few hardware related notes:

  • In case you missed out on any of the community edition pinephones, the "Beta edition" should be open for preorders on March 24th: https://www.pine64.org/2021/03/19/beta-edition-pre-orders/ This is the same hardware as the later community editions (1.2) at the same price. :)
  • There should be in a few months a keyboard for the pinephone. It will be a big honking battery case, so in addition to a hardware keyboard you will get a large battery too. Even though battery life has been pretty good having that extra battery will be very nice. I definitely plan to get one.
  • Finally, and I can't stress how much I still find this cool and amusing: The modem on the phonephone is a armv7 SOC. You can even run linux on it (The postmarketos folks are making a distribution for it). So wild to see a armv7 processor to just run the modem. How far we have come.

If you're interested in the pinephone (or any other linux mobility projects), come join us on #fedora-phone on freenode or matrix or telegram.

dumpster fire^W^W2020 year in review

Since today is new years eve and 2021 begins here tonight, I thought I would take a look back on 2020.

The big item for me in the first and middle part of the year was our Datacenter move. Planning for that started last year and spilled into early 2020. Then, setup of new machines, migration of services for a minimum Fedora, then moving all the rest of the machines, then getting them online. It was a ton of work, not only in planning, but setting up things, migrating and communicating with everyone involved. Overall I think it went pretty well. Downtime was pretty low for the most part and we managed to get things up pretty fast. It wasn't perfect of course, we _still_ have things we moved to one datacenter that are not back online yet, but hopefully soon in 2021. I really think how well it worked was a testament to everyone involved: Folks on our team, RHIT planning and networking, and even all the Fedora community members (who were super understanding of the outages and issues!). Here's to never ever moving from the new datacenter ever again. :)

I actually did manage to get one trip in in the early part of the year before covid-19 destroyed everything: I made it to devconf.cz. A excellent time as always!

You may think that the pandemic didn't affect me much. After all, I am an introvert that lives in the forest 20 miles from the nearest town. However, it really did: When everyone started working from home there was a big flood of video meetings and such, I think because everyone who normally talks to lots of people at the office wanted contact more, and that caused a lot more meetings, making things hard to get done. That died down some, but all the stress about whats locked down and what precautions you need to take to go to the store and all the US politics doom mixed in made it pretty stressful, even when avoiding people.

We managed to get Fedora 32 and 33 out the door, and on time again for the most part!

Flock was of course canceled, and I feel sad not being able to see all my Fedora friends in person. However, nest with fedora was pretty great! Not the same, but still good.

I got a new 2020 dell xps 13 laptop. It's alright, but I am not amazed by it. I might need to upgrade sooner than the normal 3 years if something amazing comes out and finances permit.

I picked up not one, but two pinephones. I have had a bit of time to play with them over the holidays, but need some more. They are still not to the level of what I would need for daily use, but they are rapidly advancing. Here's to a Fedora pinephone spin in 2021. :)

I decided I should try more microblogging, so I setup a mastodon account: https://fosstodon.org/@nirik Come and join me there and dump twitter. :) No idea if it will stick, but I am going to give it a try. I really didn't blog much in 2020, given all the datacenter work and doom, perhaps I will try more in 2021.

On the work front, we had some great folks migrate out to other places and some new people come in. If there's one thing that always stays the same, it's change. I am pretty happy about how we have moved to actually planning larger projects (initatives) that our team works on now. I think it's much better for everyone to know what we are working on, what the priority is for everyone involved and clear ideas what done is for those things. Hopefully we keep it up and refine it in 2021. Another thing that I think has been a smashing success is our daily ops standups. Just getting a few folks together every day for 30min to triage tickets, process quick things and discuss things has been great! We have really killed the infrastructure ticket queue down before the holidays. In 2021 we hope to finish that and work on the releng ones.

Finally here we gained a kitten (she was a feral cat that decided she loves people), lost the last of our dogs (he was nearly 15) and just had a bunch of stuff done in our back yard (retaining wall, a bunch of paver stones around our deck, it looks really nice!). I expect in 2021 after we get the back yard grass grown we will look at getting some more dogs, it's... strange without them around. We have also started (before Christmas even) to eat more healthy and excersize more (I'm down 9lbs so far... at least it's a dent in the pandemic weight gain.).

Hope 2021 is a great year for everyone! Good riddance to 2020!

Matrix and Fedora

Recently the Fedora Council has been floating the idea of modernizing the Fedora community real-time chat platform (currently IRC hosted at freenode.net). The front runner is matrix. I last looked at matrix 4 or so years ago, so I thought it would be a good time to revisit it and see how it looks today.

TLDR: I suspect we will have IRC and Matrix bridged together for a long time to come, if you are new user, use Matrix, if not keep using IRC.

First a few words about IRC (Internet Relay Chat). IRC is a 30+ year old chat protocol. There's tons of clients. There's tons of bots and add-ons. There's tons of gateways and aggregators. So, whats not to like? Well, everything is a add-on mish mash that can be very confusing and frustrating for new users. For example, IRC has no persistance, you only see messages while your client is connected. So, folks invented irc "bouncers" to connect for you to the IRC networks you care about and when you reconnect play back all the messages you missed. Authentication is via messaging various services bots. Encryption is via plugins or other add ons (and often not setup). So, most old timers have a client they have carefully tuned, a bouncer and a bunch of custom bots, which is fine, but new users (not surprisingly) find this all a hassle and frustrating. IRC also has it's own culture and rituals, some of which still make sense to me, but others that don't.

Matrix on the other hand is pretty new (6 years). You can interact with it as a guest or have an account on a particular homeserver. If you have an account all your history is saved, and can be synced to your client on login. You can send pictures and moves and fancy gifs. You can (somewhat) have end to end encryption (see clients below) with encrypted rooms where the server can't know what was said in the room. You can have 'reactions' to things people say. You can redact something you said in the past. You can have a nice avatar and a Real Name (if you like). You can join rooms/conversations with other matrix servers (for example the kde, mozilla and others are running servers). You can get read receipts to see who read your message and notifications when someone is typing (also client dependent see below).

Finally there's matrix <--> IRC bridges. These allow conversations to be relayed from one side to another. Of course some things don't translate over at all (reactions, receipts, typing notification, avatars) and some look different (if someone in matrix posts a picture, the people on IRC accross the bridge will see a link to the picture). The main text of the conversation is properly relayed to both sides.

I managed to re-setup my matrix homeserver (matrix.scrye.com) from 4 years ago. Ran into a little problem with the version of matrix-synapse in Fedora (It's too old to federate right, so I built the new one and the other update thats blocking it. Hopefully that blockage will be fixed soon). It's still a memory hog, but it runs well enough. They are working on the next generation server written in go (dentrite), but it still lacks a lot of features.

After that it was on to testing clients. There's a bunch more available than there used to be, which is great. The documentation on them is pretty much missing for most of the clients (exception: element has docs).

The premier client is element (used to be riot). It's available as a web app, an android app and a native linux app (which isn't available except direct from them that I could find). The web app and android app are likely the most full featured of the clients. They support setting up client encryption and cross signing your connections so all of them can read the same encrypted rooms. For chat clients, I really prefer stand alone, as web apps have a lot of issues (not restarting on browser restart, notifications not working right, poor integration into the desktop, etc).

Fractal is the gnome native client, available as a flatpak. My impression: full screen is horrible as the chat text is centered in a small col in the middle. No way to adjust the text size or font, making it really small and non readable by me. On the plus side, it does have a 'take me to next room with unread messages' key, which is really nice.

nheko is a QT client with a mix of features implemented, it's available as a rpm packaged in Fedora. My impression: Looks pretty nice, nice to be able to tag rooms into groups. Looks good full screen. There's only really a "this room has unread messages in it", not any indicator of _how many_ and no easy way to go to the next room with unread in it (that I could figure out). No docs at all anywhere that I could find. :(

Quaternion is another QT client with a mix of features implemented. No end to end encryption, but lots of the other features. It's also available in Fedora as a rpm. My impression: looks pretty nice, lets you tag rooms and seperates them easily. Doesn't seem to have a 'go to next unread room' function. ;( No docs.

spectral is a c++ client. Its packaged in Fedora, but it seems to crash on launch for me, so I didn't get much chance to look it over. ;(

FluffyChat is a port of a native android matrix client. It's pretty full featured and available as a flatpak. Does the chat sort of 'sms' style, which is cute and all, and fine for small rooms, but bad for larger discussions and such. Otherwise looks outstanding and is really fast!

Neochat is another QT client available as a rpm in Fedora. I had to tinker with my server setup and get /.well-known/matrix/client and server working before I could get neochat connecting. After that it connected fine, it was really fast, but

All of the clients seemed to handle basic chatting and history fine. Other features are all over the map. Element was the only one I saw with a search feature (to search the history). None of them had logging, which I guess could be mooted at least partly by a good search of the history/backlog. Element was the only one that seemed to have the url previews working (where the server fetches them for you and shows a preview in the client). I am not sure why so many of the clients are using QT, perhaps because kde is running their own matrix server?

So, as far as clients, I'm really missing easy ways to go to the next room with unread messages in most of them (I use this ALLLLLLLLLLLLL the time in hexchat). Logging/searching is really important to me too. I often have to look up what happened back on day X or see the exact command I used to solve something a year ago.

If you're a new user/contributor these days I think it completely makes sense to just use a matrix client. You get history without having to deal with a bouncer and some nice other features and you can bridge to all the old fogeys on IRC. If Fedora gets it's own home server this will be even easier (as I assume you will just be able to use your fedora account to login and create an account for you).

The real question is how long should we keep the current situation with Matrix and IRC bridged? What advantages would be dropping the irc bridges bring? Right now, not too much. End to end encryption isn't that interesting for an open source project. Reactions are interesting (think about using them to vote up or down proposals in meetings?), but we have done without them so far. I think migration from IRC is going to be a long process, nor is there great advantage to pushing things to go faster. I hope that over coming years matrix clients continue to get better and implement more features. Someday (probably years down the road) more Fedora users will be on Matrix than IRC, then sometime after that things will have shifted enough that the community will start assuming you are on Matrix.

I have also a few other things I use my existing IRC client with: a bitlbee server to pull in other IM/twitter/etc, and a few old IRC servers that I still hang out in, so it probably doesn't make sense for me to move over to matrix full time yet.

One additional thought: we have several IRC bots that do various things on IRC. Matrix has a handful of bots, but nothing like IRC. It's practically a programming rite of passage to make a IRC bot. :) I think we could safely look at starting design on bots for our needs for matrix and switch to them when ready (but again, no hurry at all here).

ansible 2.10.x and Fedora/EPEL

As some of you all may know, there were some big changes around how ansible upstream is distributed and maintained with the 2.10.x release(es). I thought I would recap for everyone who was not aware of these changes, then share my plans for the Fedora/EPEL ansible packages. Everyone is going to end up in a better place after this settles out, so there's no need to panic. :)

ansible has some good problems with their community: They are very popular, very easy to work on and very widely used. This means there's a flood of people always submitting bug reports, pull requests, fixes, enhancements, and all manner of things. In the old ansible setup before 2.10, this resulted in bottlenecks. The core ansible maintainers couldn't review, merge and handle all the incoming flow of things. There were a bunch of things that were tried to help this (ansibot marking up issues and PR's), making modules git submodules of ansible, making more people 'community maintainers' with merge and other powers over some modules, etc. Even with these measures however, that hits another problem: ansible only releases every N months. If you have fixed some big bug in your module, users aren't going to get that fix very quickly at all.

The 2.10 changes are another stab at fixing or at least mitigating things so that ansible can surge forward with all the community energy behind it. Basically: The ansible "engine" has been split off into 'ansible-base'. This allows it to move as quickly (or slowly!) as it needs to to be stable and improve things. ansible-base does have some modules included, but only the very base ansible modules like copy or the like. Most of the rest of the modules that used to be in older ansible versions are now shipped as 'ansible-community' in a group of collections that have all agreed to ship at the same time as a bundle. Each of those collections are seperate and able to be managed by people who care about that collection. Finally there's all the other collections out there that can self manage, release when they like, etc. the 'ansible' collection contents can be seen in: https://github.com/ansible-community/ansible-build-data/blob/2.10.1/2.10/ansible-2.10.1.deps (for example for 2.10.1).

So now we have ansible-base (the engine and just a few base modules), ansible-community (a collection of collections that is most of the modules in old ansible +- some) and all the collections in ansible galaxy. Each can now be maintained by people who care about just that collection/thing and update on their schedule. This removes the central bottleneck and hopefully gets things cranking along at a good pace.

So, what does this mean for Fedora/EPEL? Well, for one thing, upstream is no longer going to ship rpms or tar.gz releases of ansible-base ("ansible") or ansible-community. They have decided distributions can do a fine job there and it's just duplicating effort (releases of ansible/ansible-base and ansible-community will be on pypi just like many other python upstreams). Next, of course, ansible-2.9.x is not going away anytime soon. It should continue to get updates for a while. I'm planning on submitting 'ansible-base' package to Fedora review today (it's all ready except that you cannot build full web docs currently as it needs a stack of new packages and a bugfix/update to python-pydandic, so I will be disabling full web docs to start with). Then I am going to start on a ansible-community package. Once they are both done in rawhide and the packaging is all sorted out, I'm planning on encouraging collections in the ansible-community meta collection to packaging and maintaining their own collections along with any collections that are not part of that meta bundle that want to package up. Once thats all working, we will look at retiring the old 'ansible' (2.9.x) package in favor of ansible-base + collections and push that out to other Fedora branches and EPEL. Some versions of EPEL (I'm looking at you 7) may stick with 2.9.x for as long as possible as 2.10.x may start to use newer python features that won't work on EPEL7. EPEL8 may need to switch to newer python module, but time will tell.

I'm looking forward to this new setup and the added velocity it gives ansible!

Any feedback/comments/better ideas are welcome in the tracking bug for ansible-2.10.x: https://bugzilla.redhat.com/show_bug.cgi?id=1848739