As many of you know, we had a long string of difficulties with getting rawhide to compose at all recently. This was mostly a combination of poor timing for things landing, people landing things at the last minute for Fedora 28, and various bugs. Last week, we hit something that seemed to be pretty special: An actual finished compose. To explain, pungi (the tool that does the composing in Fedora land) has several states that a compose can be in: It can be "STARTED" which means that it started and is composing, or died in such a way it couldn't properly note that to the compose. It can be "DOOMED" which means that one (or more) required deliverable items failed to compose. In the past this might still have meant part of the compose was shipped out anyhow, but not these days. That prevents the compose from going out at all, and releng folks have to look and make sure the issue that caused the problem is fixed. A compose can be "FINISHED_INCOMPLETE": This means all the required deliverables are there, but optional/non blocking ones may or may not be. This is the status most of our composes have gotten in the past. Finally, a compose can be 'FINISHED' which means all parts of it successfully composed. We had our first "FINISHED" rawhide compose last week. So, break out the harmonica and start dancing right? well, not so fast. There was a bug in pungi where it would have some failed optional deliverables, but still report FINISHED. So, the FINISHED we had wasn't really a true FINISHED (some ppc images were still failing). Finally late last week we got in a workaround for those last few images, and I am happy to report today that the FINISHED we got was a true, full, real, everything composed FINISHED. I think some of the credit for this goes to the simple compose tracker that Dusty Mabe setup. This allowed us to track down and fix things that in the past we would just have written off that we don't have time to fix. Thanks for the issue tracking Dusty! There was of course lots of hard work from many others as well to get to this point: Thanks Mohan, Patrick, Adam, Dennis, Sinny and others I am sure I am forgetting. Congrats on a FINISHED compose, and lets see how long we can keep it that way.

Greetings everyone! Lots of news from the rawhide trail this time: Astute observers will note that from 2018-02-04 until 2018-02-17 there were no (success-full) rawhide composes. This was due to a long line of issues, problems, new packages, changes and just bad luck, including, but not limited to:

• New pypkickstart that broke livemedia-composer
• New pykickstart that broke appliance-creator
• New libevent that bumped SO, requiring everything using it to be rebuilt but...
• GCC8 issues with chromium (and also qt5-qtwebengine) that prevented them from being rebuit for the libevent bump.
• Ongoing issue with armv7 composers getting stuck in mkfs.ext4 on composes, requiring manual intervention to continue the compose.
• General broken deps after the mass rebuild that broke some (release blocking) live composes.
• system-config-printer pulling in a PackageKit subpackge, which broke the Xfce live due to it excluding PackageKit*
• rng-tools, which was broken in the past and untagged, but the mass rebuild rebuilt it again due to the maintainer not cleaning it up yet.
• Probably other things I am forgetting.

In order to help out all the folks working on fixing things we are trying out a ticket system to track failed composes. See the proof of concept version at: https://pagure.io/dusty/failed-composes/issues I think this already is helping coordinate and communicate breakage. Hopefully we will get it polished up and made "official" soon. Of course some of the above compose issues were due to the mass rebuild, which happened a few weeks ago. Things seemed to take longer than in the past, I think due to src.fedoraproject.org not committing as quickly as it had in the past. Hopefully we can fix that. Also, today we are at another milestone: Fedora 28 is branching off rawhide. Rawhide will continue on it's trail to Fedora 29. Do make sure you know what path you want to follow and that your machine(s) are taking that path. If you do nothing you should end up on Fedora 28 branched and follow it to release. Gnome was briefly broken in rawhide, but quickly fixed by a new gnome-session build. Thanks gnome maintainers for quickly fixing things up! The kernel has moved on to 4.16 pre releases in rawhide. Early (before rc1) releases here had no networking with the ath10k card I have in my laptop (it would just never see any wireless networks). After rc1 however things are mostly back to normal. The ath10k is a bit flaky, but mostly works. Finally, on a personal note, one of my dogs has had major surgery and a long slow road to recovery. I've started a gofundme to help defray his massive medical bills. If you would like to donate, pass along or just look at some cute pictures of him, you can go here:  https://www.gofundme.com/medical-bills-for-nash

Last week (and part of this), I traveled out to lovely Brno, CZ for some Fedora Engineering meetings and then devconf.cz. Overall it was a great trip! On the way out (last sunday) I drove up to the portland arirport (about an hour), then plane from PDX to AMS (amsterdam) (about 10 hours), where I met up with Patrick and we both were on a flight from there to PRG (prague)(about 2hours). Then it was a train from Prague to Brno (about 2.5hours) and a deep sleep at the hotel. Since many of the folks in Fedora Engineering were going to be coming out to devconf.cz, we decided to do some meeting up and discussing things to work on this coming year and to plan out a hackfest we are hoping to do in a few months. Look for more details on all of that, but in a nutshell:

• We planned out things we could work on a a hackfest: https://fedoraproject.org/wiki/Infrastructure_Hackathon_2018 which looks to be in early april.
• We talked over ownership of some apps that are needing love and/or having team members move to other things.
• We talked about plans for account system and where we would like to be.
• We talked about gating packages in rawhide. We came up with somewhat of a plan, but needs some work/discussion for sure, look for more details soon.
• We talked about some apps we want to just drop or move to another service, etc.

Look for more news on all of those before we do anything about any of them. Next was devconf.cz. I was told how big and busy it was, but seeing it all was different. I went to a number of talks, some were pretty good, others just things I already knew. The short talks (23min) were kind of difficult, because it's really hard to give someone much information in just 23min. You do a intro, discuss something and then it's over. I was going to give a talk about using Fedora Modular server 27 in production, but since that didn't ever happen to be released I gave my talk slot over to Stephen Gallagher to talk about the new modularity approach. The hallway track was very nice at devconf.cz... lots of Fedora contributors around to meet again and talk with. Had a bunch of production discssions with too many people to count. It will be interesting to see how the devconf.us works out, it it's as busy and vibrant. Then all too soon it was time to pack back up and head home. I managed to get in tuesday afternoon, but promptly crashed for 14 hours. All in all a great trip!

Happy holidays everyone! A few notes for those riding the rawhide trail or just joining those of us who are.

• NetworkManager has finally dropped libnm-glib (deprecated 3 years ago now). There's proposed gnome-shell changes being worked on to switch to libnm, but they have not yet landed. This means if you connect to VPN's via gnome-shell, things aren't going to work. As a workaround for now use nmcli: mcli c up 'your vpn name here' --ask. More info at https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/4YJIWJXDMND2VL7KGG2C6UNE7RJMHJEI/ and https://bugzilla.gnome.org/show_bug.cgi?id=789811
• better SATA power management is being enabled. See https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/T2STBDPXKZ7DHC7GS6VLM3ESYI6RHVDM/ There is a small chance of disk corruption, so this is a great time to check your backups. If you have a laptop with a SATA SSD you should expect some nice power savings from the change.
• Additionally bluetooth power saving mode has been enabled a while back. If you find your bluetooth no longer working you can boot with btusb.enable_autosuspend=n. (My bluetooth here on my yoga 920 needs to revert this change currently. Tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1514836 )
• I reinstalled the armv7 compose builders a few days ago with Fedora 27 and since then they have done great for rawhide composes. With Fedora 26 they were sometimes hanging, causing rawhide composes to hang until manually rebooted. If they keep working well we can hopefully close https://bugzilla.redhat.com/show_bug.cgi?id=1504264 soon, but I would like to see it work for a few more days.

Here's to a happy rawhide holiday for everyone!

Once again, things have been real busy and I haven't kept up with sending my rawhide notes, but I will try and do better! Astute rawhide trail travelers would have noted that there was a distinct lack of updates to rawhide recently. There were composes up to December 4th, and then nothing until late yesterday (December 11th). Here's a list of the reasons all those composes failed: * As part of a big infrastructure move (Moving every server in our main datacenter to a new area) we also applied updates all around. The December 5th rawhide failed because the new koji package had a bug with python3 and 'runroot' plugin we use for koji. This has since been fixed upstream and a fixed version applied to infrastructure machines. * Another few runs failed because we were moving the signing infrastructure and didn't have everything back up and working. Recent changes to the rawhide compose require signing because it signs ostrees commits that are made as part of the compose (unlikely packages where they are signed before the compose). * A firefox update came out that did not build on armv7. This was ok until a new hunspell update came out, a bunch of things built against it and the existing firefox had a broken dependency on the old version. Ideally we could have untagged the hunspell update and all the things that had already built against it, but that was a pile of packages to look for. In addition, the firefox update was a security one (of course). So, I filed a bug on it, excluded armv7 (just until the compile is fixed) and did a new build. This was only part of the rawhide fix however, as the Xfce armv7 image is release blocking and it no longer had a firefox. So, I dropped firefox from it for now (until the compile is fixed). * While dealing with the firefox thing, I merged some more kickstart PR's folks had submitted. Sadly, one of them had a syntax error in KDE (which is release blocking), causing another compose to fail. So, we are back on track for now. Hopefully firefox will get fixed soon as we can revert hacks. I'll keep watching things over the holidays.

Having just purchased a Lenovo Yoga 920, I thought I would offer the following (probibly too detailed) review for any interested parties. History / Background:

This is now the third yoga laptop I have owned. First a yoga 2 pro in 2013, then a yoga 900 in 2015 and now the 920 here in 2017. Lenovo does come out with new models every year, but for me at least they don't become compelling to jump to until another model, so I have skipped the yoga 3 pro and the 910 models (and all the other side models they have now like the yoga 700). This cycle I seriously considered moving over to a dell xps 13 developer edition, but in the end a few things drove me to the yoga 920: 8th gen cpu (which tuns out to be a pretty big deal, see below), higher screen resolution, and no "nostil cam" (webcam at the bottom of the screen looking up). I use a laptop as my primary machine, so I am sitting at it typing away for many many hours a day, which makes it well worth it to me to get something nice. The dell xps 13 developer still definitely has some advantages, like firmware updates via fwupd seamlessly in Linux instead of needing to keep windows 10 around just to do that.

stack of yoga Hardware / Specs:

As I usually do, I went for the top of the line model. The 920 is supposed to come in 3 colors and 3 'glass' editions. The colors are "platinum", "bronze" and "copper" and the glass ones have a gorilla glass overlay over the cover with a design on it: A fancy wave pattern "vibes", a star wars rebel logo and a star wars empire logo. The copper and star wars models aren't out yet, the vibes model didn't seem worth an extra ~$300, and the platinum seemed kind of boring, so I went with the bronze model. It's a pretty dark coppery bronze and it looks pretty nice.

Specs dump:

• 16GB memory.
• 8th gen intel i7 cpu (i7-8550U)
• 1TB NVMe storage
• 13.9" display (3840x2160) with digitizer / touchscreen
• Lenovo active pen 2 (4096 levels of pressure), included in the bundle
• 2 usb-c thunderbolt 3 ports. You can charge via either of them. They are both on the left side next to one another
• 1 headphones/headset jack. I predict I will never ever use this.
• 1 old USB (on the right side)
• 1 very very tiny "novo" button to get into the bios or recovery or the like.
• Fingerprint reader.
• Touchpad (happily without the sort of chrome border the yoga 900 had)
• Keyboard with 3 levels of backlight (low, high, off).
• 720p webcam (at the top of the screen)

The laptop feels very solid and sturdy. It's a slight bit lighter/thiner/smaller than the 900. The top lid has a slight overhang over the bottom, which actually makes this laptop much easier than the yoga 900 to open. The screen is very lovely... and side bezels are very very small. The top bezel is a little bit wider (to accoodate the webcam I guess) and the bezel on the botton is about an inch or more. The keyboard layout has changed a good deal: They removed the home/end/pageup/pagedown dedicated buttons on the right and lumped them into being Fn arrow keys. They also shrunk the 2 dedicated up/down arrow keys to 1/2 height. The missing pageup/pagedown and small up/down arrows has been anoying to get used to, but otherwise the keyboard actually "feels" much better. It's just more relaxing to type on than the 900 or the yoga 2 pro. The touchpad is large and responsive. If you press it with CPU intensive tasks you can get the fans to fire up, but they are not any worse than the ones in the 900, and it fires them up seldom in day to day use. The speakers seem a bit better than the 900's, but if you really want to listen to something, put on headphones, as all laptop speakers are likely to disappoint.

Fedora Install:

With my previous yogas I had just wiped the drive entirely and installed Fedora off the bat. This time I wanted to keep a windows partition around so I could upgrade firmware and the like. This turned out to be a large time sink. :) First I got sidetracked because I assumed I would have to tweak the drive settings in the firmware and change from Intel raid to AHCI. Turns out thats not possible (no options for it) or needed (linux sees the NVMe just fine). Next I ran into windows saying it would only shrink itself by about 50%. I did not want to give it 1/2 of my disk for a glorified firmware updater, so I had to mess around with disabling page files and hibernation and all sorts of things until I could get rid of the "unmoveable" files in the middle of the disk. Finally I managed to do so and shrunk windows down to about 50GiB (still too much, but better than it was).

I grabbed the latest (at the time) Fedora 27 nightly Workstation iso and copied it to a usb drive, then booted from that. Things looked pretty good in some casual testing, so I went ahead and installed Fedora into the free space there and booted up the install to customize/add things and upgrade to Rawhide. On reboot it seems I had no bluetooth or wireless. As was the case for the previous yoga's I have had, it needed to be added to the ideapad_laptop module. Someone already sent in a patch to do this upstream. A simple blacklisting of the ideapad_laptop module for now gets things working.

What doesn't fully work:

As noted, for the most part things all just work fine with Fedora. There's 2 exceptions however:

First, the fingerprint reader. It seems this is a synaptics / validity reader, which is a bit of an odd bird. There is a reverse engineered driver underway, but it seems stalled: https://github.com/nmikhailov/Validity90 It seems it uses encryption talking to the device and synaptics is not being very open about specs either. Oddly, on my 920 I can't really see the device at all when booted under Fedora. Looking at it when booted under Windows shows the device name.

Second, I had a fair bit of trouble with the wireless card. It's a "Qualcomm Atheros QCA6174 802.11ac" card and it's recognized and works, but it has had various problems with stalling out and needing to reconnect, very very slow transfer speeds, hanging and then bursting out again. It seems like it's a firmware issue: When I removed the -6 firmware (forcing it to use the older -4 one) it seemed a bit better, at least when it disconnected it would (usually) recover, but then I upgraded to the very latest firmware upstream (that is not yet in the linux-firmware package and now it seems pretty solid. Hopefully this will all get sorted out soon by them pushing that new firmware out to linux-firmware.

What does work well:

The new 8th gen CPU(s) really shine here over the yoga 900. You end up with Linux seeing 8 cpus (with HT) and they really make short work of most day to day tasks. I don't usually build things or compile kernels, so the only time I so far have managed to make it spin up the fans was with thunderbird sorting and filtering a ton of emails or a large dnf update transaction.

Battery life is great. I am going to have to do some real life testing on it, but it's saying its going to get 10-12 hours or so. My yoga 900 is down to 5-6 now (but the battery is a bit old at this point).

Unknowns:

The Lenovo active pen 2 seems to pair ok with bluetooth, but then it disconnects a short time after. I am not sure if this is a Linux kernel bug or something else going on. Since I don't care too much about the pen, I figure I can play around with it some more when I get time.

yoga 920 (front, left), yoga 900 (back, rear) So, overall, I think I will be quite happy with it for a few years. If anyone out there has questions or would like some data about the yoga 920 and Fedora linux that I didn't include, feel free to drop me an email or pingback and I will update this article over time.

Greetings. Just a note for anyone looking for ansible in epel7. It's been retired there because with the release of RHEL 7.4 it's now int the rhel-extras channel. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/7.4_Release_Notes/index.html#technology_previews_red_hat_enterprise_linux_system_roles_powered_by_ansible Accordingly, you can get ansible now from rhel extras channel, or CentOS extras repo. You can also get ansible rpms now from http://releases.ansible.com/ansible/rpm/ Note that ansible continues to be available from epel6.

The last day of Flock started even earlier with a team breakfast. The Fedora Engineering team (which I am on) really only gets together once a year at flock, so we gathered up for a team breakfast to catch up on things and see each other in person. Nice breakfast at the Egg and I. Then back to the conference where folks were showing off what they did or telling people how their sessions went. I think a number of folks had to depart before this, so I am not sure how well it worked, but it was nice to hear how sessions I was unable to attend went. Next up was the weekly FESCo meeting. Since most of us were at flock, we gathered in the hotel lobby with laptops and had a IRC meeting while all together. The meeting went pretty quickly and then it was time to catch a ride to the airport. The ride was smooth and trouble free until we got close to the airport. Then an accident made the last few miles take an hour or so. I had time to catch a sandwitch and a beer before my flight (did you know Sam Adams makes a nitro vanilla porter?). Flight was long, but uneventfull and then just the hour or so drive home. Finally got home around midnight local time. Some final thoughts:

• I'm not sure the recap works on the last day. Might be better the end of the previous day so everyone is still around?
• I'm not sure I feel like I got much "do"in done. I definitely got a lot of nice discussions and learned some very nice things, but not sure how much I really did while there. Perhaps it would be good if we want to do that focus again to have workshops/sessions decide up front what they want to do and then report on if they did it or more than that or less than that. Have a concrete goal to work on. It's so easy to get distracted.
• I met a number of new folks or people I only knew via IRC, but perhaps we could do something to mix things up more and get people to meet up with new people instead of talking only to old friends. Perhaps some game/badge you can only get by scanning people's badges or a event where random teams are made for something? Or assign everyone a conference buddy that is the oldest fas account holder with the newest and so on.

Anyhow, I had a great time as always. Many kudos to the flock team and all the attendees! +1 would flock again. ;)

Day 3 started again at 8:30, so nice and early. First up was Mike McGrath giving the "What does Red Hat want?" talk. He had a very interesting (and likely right on the money) concept: Fedora lives in the space right after the very early adopters of something and before the bulk of people start adopting something. In other words, if we aren't adopting new things, stablizing them a bit and then moving on, we are not in the right place. We don't want something too soon, it has to be starting to gain adoption, and we also don't want something thats too stable and going on to the masses because it's boring. Excellent food for thought. Next was my workshop (luckily nearby Mikes talk). The room was pretty packed, which was great. I first went over plans for the rest of this year, next year and beyond as I see them. I'd like to see us get more apps in openshift. A lot of the simple ones shouldn't be hard at all. Then, I'd like to look at leveraging some cloud providers some more. Since we have openshift and apps in it we could look at migrating some of them to cloud providers and reduce our dependency on our hardware in our main datacenter. This would also allow us to spread out more and handle more traffic and load more easily. I'd like to also work more with other open source community infrastructures: in particular CentOS folks, who we already talked to about running our jenkins instance and setting up better integration between pagure and centosci. We share a lot of needs and there's no reason we can't also share resources to meet those needs. I'd also like to get more metrics than we have these days (about bwandwith used, cpu consumed, etc... which moving to a cloud provider will give us). There will of course be lots of things where that doesn't make sense (like the buildsystem/releng machines), but it should make us a lot more flexable. After going over that I opened things up to questions and ideas. I was hoping for some more crazy off the wall ones, but we did get a number of good ones. We talked jenkins and CI, we talked about metrics and things that do that, monitoring (if we wanted to move away from our current setup), how we might best make use of modules (perhaps have infra specific ones we use for our apps, which makes us not be tied to a specific Fedora release), making docs or other artifacts via openshift, or even reworking how packages are built to use it. Then we broke up to hack on various things and let other folks have some time back if there were things they needed to do. Then after a bit of hallway/fixing compose issues time, it was lunch again. After lunch was a general talk on modularity. I think I had a handle on things before this, but it was good to see the current state and there were some great questions from the audience. After that it was back to the hallway to discuss things with folks and work on composes and other fires. There was no planned dinner this evening, so went out and had a relaxing dinner with some folks at a local pasta place.

Day two of flock started too early for non morning people (8:30am), so I was up around 7am to shower, get some breakfast (at the hotel resturant, it was ok) and get over to the conference rooms. There were a ton of interesting talks in the morning, but we were also having some rawhide and branched compose issues, so I opted to go to the hallway track and work on that and discuss all kinds of things with all kinds of people. It was great to be able to have full bandwith discussions with someone about something. Had lunch outside on the deck. It was very nice there, but a bit hot and bright for a bunch of nerds. Lunch discussion was all over the place, including new arm laptop (pinebook) and Fedora builders and how various things were made to laptops and support from various vendors. Then it was time to rush to the EPEL for the future talk from Smooge. The room was standing room only, so perhaps people are realizing how popular and important EPEL is. Smooge presented some proposals for dealing with various problems in EPEL, but I had to go and get a server that crashed back up and running, so I missed the discussion part of this. Hopefully we will see more discussion in EPEL meetings and on the epel-devel list. Next up was Ricky's talk about our recent openshift deployment. He went over how we are doing things and the various caveats and issues around any new tech. The slot was only 30min, but he covered things very well. Next (in the same room) was Jeremy's talk about the future of fedmsg. I was a bit confused here I admit (sorry Jeremy!) about the scope and nature of changes. Really it was a bunch of things we could (or decide not to) change. I somehow thought of it as all tied together. I am sure this will also have a bunch more discussion soon and we will see what we all want to do here. I then went back to the hallway to work on compose issues and some urgent tickets. Then it was time for the evening event: Wackenhammers arcade. It was a pretty amusing old time arcade with beer and food from a food truck. Got some nice visiting in here with the Fedora kernel team and tons of other folks.