I decided to take a slightly different way home. In part because I couldn't find a flight from DRS -> AMS that would get me there in time for the AMS->PDX leg and partly just because it would be fun: I took the train from dresden to frankfort and then on to Utrecht. It was about a 10 hour journey, but it was nice. I went with Patrick and Randy and we had a bunch of discussions and watched the landscape go by. In Utrecht I got a hotel for overnight and we went to a great little resturant along the canal. Great place. My only regret it that it got too late for me to see Patrick's place, so I just went to bed and got up the next morning for my flight. On checking in I saw that I had a lovely SSSS on my pass, so I got to get extra searched at the gate, but it wasn't too bad, and it let me board before I would have normally been able to. Then the flight back and the drive home were mostly uneventfull and I managed to get in around 2pm. Overall this was a great flock, I had a lovely time and all the organizers did a awesome job!

The last day came nice and early right into sessions. However, I instead gathered with a few folks to distribute some keys for various things so we were sure that if someone won the lottery and left we would be able to have access. The next session I went to was Randy Barlow and Patrick Uiterwijk's Openshift session. Not much I didn't already know, but some good discussion and questions from the audience. After that was group photo (many people already suggested we do this on the first or second day so some people haven't left yet), then council wrap up. After things wrapped up, some of us stayed at the hotel visiting and hanging out and saying goodbye. Until next year flock!

Day 4 jumped right into sessions: Protecting the batcave against the Joker: Fedora infra hardening workshop Michael Scherer. Some interesting vulnerabilities that misc found (but did tell us about in advance), and how they were found and some ideas to help mitigate such things moving forward. The big one was a predictable git checkout in tmp, so we could add noexec to tmp and/or do seperate tmp for all users. Running ansible-lint also is something we should really look at doing. After that I again jumped to hallway track, and then a team lunch, which I was late getting back from. After running back late from lunch I got to the Fedora Infrastructure hacking and planning and hacking workshop session. I wrote up a bunch of areas we wanted to talk about or work on in the coming year. There really wasn't too much discussion on most of them, they were all things we knew we wanted to do it's just a matter of doing them. Neal Gompa did a live demo of the current pagure 5.0 themes and appearence. It looks nice! The list was:

• new cloud - this is just very very close, so we decided to finish it asap.
• project tracking - I will setup a test for this once pagure 5.0 is out.
• replace nagios - many people suggested we look at zabbix again.
• CAIAPI/noggin - asked folks to file requests for features
• Repospanner - very close to existing, just need to finish it off.
• fedora messaging - abompard gave us a overview of where we are and we discussed corner cases like FMN and datanommer.
• sigul 1.0 - pretty much done we just need to have a flag day to update everything in step (since it's not compatible with 0.9)
• copr - we discussed plans to move parts of copr into main infra. Possiblly frontend in openshift and dist git and keysign replaced by sigul.

We had a number more we didn't get to because we ran out of time:

• python3 all the things
• more openshift
• more ci stuff
• javascript frameworks

Next up was my Infrastructure Fix Fest. I was hoping we would get folks to come to us with all the small bugs they hit that we just haven't had time to fix up and we could fix them right then in the session. Unfortunately, I think because it was the last session of the day and later in the week, many people went back to their rooms to rest for the evening activity or went to the hallway track to talk to people. We did get a few things fixed... I managed to close about 5 infra tickets, track down and fix a issue with kerneltest application that Justin wanted me to fix and discuss a few others. I think next year if we do this again I would ask people for items before hand and we can use it as a chance to show how to fix things. The evening entertainment was a 'roller coaster' restaurant. You order from a tablet and they send your beer or food down on a set of rails. It was pretty amusing. There was a lot of sci-fi clash (the servers were wearing star trek the next generation outfits, but there were star wars characters all around too). At one point a bottle of beer broke, but no one was hurt and they got it cleaned up pretty fast. Overall a lot of good company and amusement.

Day two started out with The power of one: For the good of the community by Rebecca Fernandez. She had a lot of great information there on how to handle various community interaction and we had a number of questions from the audience. Some of the things were how to approach busy folks with your ideas, or how to suggest things in communities you are new to as well as how to handle 'heated' issues on mailing lists. A number of great points to ponder there. Next up was my talk about keeping rawhide on the trail. I had added a few points to my slides early in the morning before coffee and that bit me at the end of the talk (my summary/questions slides got deleted somehow). Other than that I hope it went well. I talked about history and then a bunch of interesting compose issues we hit in the last while and then some things we could try and do to make it better. Next I wanted to go to Brian Stinsons CI talk, but it was already standing room only by the time I got my laptop all packed up from my talk. So, I did a bit more hallway track. I had to go to Christoph Wickerts talk "From Zero to Hero and Back again – My Decade in Fedora Land". A lot of great old photos and discussion. Ah memories. Since I am in the server working group, I decided it might be a good idea for me to next go to Stephen Gallagher's "State of the Fedora Server edition" session. There were some interesting ideas here, many around leveraging cockpit as our deployment/management application. Several people promised to mail the server list their ideas, so I look forward to hearing those in the coming weeks. Next I went to the SIGs roundtable and talked about EPEL with a number of folks. Some great ideas, we will see how they play out in the next year or so. After that I went into hallway track mode and talked to various folks.

After a nice sleep and breakfast things started out with announcements then the state of Fedora talk from the FPL, Matthew Miller. Summary: doing pretty well, but we should try and get the numbers of users estimates more accurate, and keep doing the good work on all the things we have in the pipeline for the next few years. Fedora Objective leads then gave updates on their objectives.

• Matthias Clasen talked about all the good stuff in Fedora Workstation over the last few releases
• Langdon White updated us on Modularity
• Stephen Gallagher talked about Fedora Server and directions for upcoming work
• Dusty Mabe talked about Feodora CoreOS and Atomic host. We should see a combined setup for f30.
• Peter Robinson talked about Internet of Things and plans for it coming up.

Overall a great bit of info on where we are and what we had done this last year. There seemed to be more excitement in the air than last year. In the middle of the above reporting we took a coffee break. I won't mention them again, but I'd like to note here that they worked really really well. There was an area setup with coffee and pastries and some tables with no chairs and everyone could mingle around and talk. It was a really great way to have a "hall way" track without missing any talks. Since it was all in the same area you could often find whoever you wanted to talk with, or wait for others to come talk to you. Kudos to the organizers! Lunch was in the hotel resturant, buffet style. Was pretty good overall and was nice to avoid travel time, etc. After lunch, I went to the "RHEL, Fedora and CentOS: Solving the penrose triangle" talk by Josh Boyer and Brenden Conaboy. It's very true that there is not a normal direct upstream -> downstream flow to the relationship between these three, and they had some interesting ideas on how we could make the feedback better and understand it more. Next was the Fedora Infrastructure onboarding talk by Ricky Elrod. This is a topic that is near and dear to me, so it was great to see the ideas and also to see the interaction from the audience. There were a bunch of questions and ideas. I hope we can implement some of them soon and see if they help folks out or not. After that, I went to the session on improving compose times with Mohan Boddu and Lubomir Seldar. This one had a bunch of good ideas on how we can get our compose processes down in time from the current 8-9hours to much less. There was a lot of audience participation here. Great to see! The next slot there were several things I wanted to see, but I ended up in the hallway track talking to lots of people, so that was all good too. I think the talks this year were all recorded so they should appear soon I hope and folks can see them for themselves.

This year flock ( https://flocktofedora.org ), our yearly Fedora mega get-together is in Dresden Germany. Since I am near Salem, OR in the USA, it's a fair bit of travel:

• 9am monday (2018-08-06) - drive to Portland airport
• 10am get to the long term parking and take a shuttle to the terminal.
• 11am have a quick lunch before boarding my flight
• 12:30pm, off from Portland to Amsterdam (10.5 hour flight)
• 8:30am (2018-08-07 now) - arrive at AMS.
• 12:30pm flight from AMS to Dresden
• 1:30pm arrive Dresden, head to hotel.
• 2:30pm Hotel checkin.

Then I just had to manage to stay awake until after dinner. Flock is always a great conference! So many old and new friends, great talks and energy for the coming year. Should be a great one again!

I thought I would share today some history, recent happenings and plans for OpenShift in Fedora Infrastructure. We currently have two OpenShift clusters running in Fedora Infra (not counting two more dedicated to osbs, the container build system). One of these is our staging instance and the other our production instance. Astute readers who know OpenShift may ask why we even bother with a staging cluster, since a lot of workflows with OpenShift let you test and validate things and deploy them as you like. Well, there's a few reasons we set things up this way: First we don't have all that many apps moved into OpenShift yet, so they have to interact with all the rest of our staging deployments on vm's. Also until recently it wasn't easy to seperate routes in Openshift to make sure you were only sending staging messages to the staging network and not ever mixing in the production network (this has since been vastly improved, but haven't taken advantage of the new functionality yet). Finally we wanted to make sure we had a test bed for deploying and upgrading the clusters without just having to do all that work in production. Or production cluster has 3 apps running in it currently: bodhi's web frontend, waiverdb and greenwave. Or staging instance has those three, as well as a number of not quite completed apps: modernpaste, rats, release-monitoring, librariesio2fedmsg, and transstats. Last week I went ahead and reinstalled both our staging cluster (on wed) and then our production cluster (thursday). I had actually upgraded our staging cluster from 3.7 to 3.9, but I wanted to make sure we could redeploy from the ground up. Production had been on version 3.6 (upgraded from 3.5). The nodes we were using were actually sized when we installed 3.5 and the requirements changed after that making them too small. I had re-installed the compute nodes ok, but the masters didn't go as well, so I figured it would be a good time to just reinstall completely with the right sizes and 3.9. Additionally, support was added in 3.9 to use cri-o instead of docker containers and we wanted to take advantage of that, as well as adding persistent storage for our registery so we didn't need to rebuild images all the time. There were various hiccups and issues in our config, but after staging we didn't hit too many in production. All our apps are now using cri-o containers! Our plans moving forward include:

• Moving more apps into OpenShift. It's still not as smooth as we would like to do initial config, but we hope to improve that and after an app is setup, it's much nicer for everyone to maintain and deploy from there.
• We want to improve our ansible setup for deploying / managing apps. Right now our roles and setup are very flexable, but if we change them to be more opinionated we can hopefully make it easier for people to run apps there.
• As soon as our new cloud is up and running we plan to deploy a OpenShift there. I have been calling it a 'dev' env but I am not sure I think thats the right name now. It might be better to think of it as 'manage your own'. We hope to give people who need it wide access there. The downside is of course if we do a reinstall everyone would have to redeploy their apps.
• Someday we should just have our staging cluster to use for upgrades/re-installs and have all our apps in one prod cluster.
• Someday it would be great to pull from upstream git a prod or a stg branch and build and deploy from it. Most of our apps currently are just using rpms, so it doesn't save as much work on deployments.
• We definitely want to move as many apps as make sense over, just will need some elbow grease.

It's exciting to see all the improvements to OpenShift over the last year. It just gets better and better!

In case you didn't hear it all the other places you would normally hear it: Fedora 28 is out. Get it! https://getfedora.org/ I updated my servers here at home last friday, as usual 0 problems at all with the upgrade. Upgrades really are painless these days.

Last week I had the pleasure of attending the 2018 Infrastructure Hackfest in Fredricksberg, VA. It was a very productive week and very nice to meet up face to face with a lot of folks I work with mostly over IRC and email. Travel went pretty well for me (direct flights, 4-5 hours each way) and the hotel worked out nicely. I liked that the hotel had a big table (with power!) in the corner of the lobby for us to use in evenings for more hacking. Our day workspace was a classroom at a nearby grad college. Aside from some firewall issues monday morning (They were blocking everything but 80/443) it worked pretty well too. Lots of tables we could move around, and whiteboards/projector. Monday we went over current package maintainer workflows and looked for improvements we could land. Pagure 4.0 (out very soon now) should let us fix the silly 'get a pagure token for this and put it in a config file' workflow. We also got bodhi to have a 'waive' button to waive test results from the web ui, along with showing what exact tests were missing or failing (so we can get rid of a hacky shell script on the wiki for this purpose). After workflows, we started on package maintainer docs rework. We wanted to move them out of the wiki and clean them up and reorg them. First we tried to identify all wiki pages that are related here and make sure we got all the popular ones, then we brainstormed some personas (new to packaging, upstream maintainer wanting to just package their project, someone who doesn't have time to follow process and just wants the steps to do an update, someone who wants to know how it all works, etc), then we wrote up a bunch of things these people would want to do and tried to organize new docs for it. It's not really yet in a state to get wide feedback, but hopefully soon it will be. Tuesday we talked about reassigning some apps that were maintained by someone who moved to other work, then retired a few apps that have been limping along that we no longer want to spend time on: darkserver (which actually has been broken the last year or so), and summershum (which got checksums of source file uploads), which we never quite figured a use for. Next up was rawhide gating. We went over the proposal send to the devel list and got more detailed about what exact things we would need to do for it. It turns out that just adding some things to the update object we could handle the case of side tags and the work won't be as big as we thought it would. Wed was setting up a AWX instance. This is the upstream version of ansible tower. We hoped that using it would give us some good advantages. Unfortunately we hit a number of issues. If we tried to install with a external db the install failed, so we had to move back to a db in container next to the rest, then we got pretty far on configuring authentication and hit a bug preventing SAML from working, then finally we hit an issue where we couldn't get our private repo mounted in the right container to allow us to have access to our secrets. There will be more discussion on this in the coming weeks and we can decide if the effort is worth it or if we should just keep on going the way we had been. As a side note we looked more at the loopabull setup we have. It has only one demo job in it, but it seems ready to add more into as soon as we would like. Look for us to use that more soon too. Thursday was more work on rawhide gating and some discussion around openshift. Patrick took the projector and he and Randy got bodhi's web frontend completely moved over to our openshift (but prod is still not pointing at these new instances until we are ready). We also did some more work on release-monitoring.org (anitya) and got it much further along. There were a number of tweaks and improvements to our openshift playbooks too. Friday we just met at the hotel as everyone was traveling away, so we just worked on finishing some things up. A few things that got done also over multiple days or between other issues:

• jenkins.fedorainfracloud.org is now redirecting to centos-ci. This is the jenkins used by various pagure projects to test their code. We had been working on moving it, but hadn't gotten to it. Now it's all done except for a more perm redirect.
• Various internal hardware/budgeting/ordering things were sorted out
• Tracked down and fixed our old cloud when updates caused it to stop processing copr builders right.
• Initial work was started on CAIAPI and the small flask based web frontend in front of it (This will replace fas2).
• Priorities were added to the infrastructure issue tracker to help set expectations on what issues are waiting on what.
• A new template was added to the infrastructure issue tracker to help us know what issues need to be done by and what they impact.
• Some thoughts for making the weekly meetings more interesting or more accessable to the developers on the team were brainstormed.
• We got RHEL 7.5 and openshift 3.9 all set to upgrade to after the upcoming freeze is over.

All in all, a very pproductive week of hacking on things. Many thanks to Paul Frields (stickster) for organizing everything and the Council/OSAS/Fedora Engineering for sponsoring things.

It's continued to be a bit of a rocky trail of late for rawhide. The last 6 composes have failed due to:

• 2018-04-03: New xorg-x11-server landed (1.20). However, tigervnc and xorg-x11-drv-vesa were not rebuilt for it yet, the broken deps caused the failure.
• 2018-04-04: Same as 03, but the right folks fixed things this day.
• 2018-04-05: pykickstart dropped python2-kickstart subpackage, which is needed by python2-img which is needed by appliance-creator. Untagged pykickstart for now.
• 2018-04-06: The kde live media ran out of space.
• 2018-04-07: The kde live media ran out of space. Bumped up the space slightly to allow it to (hopefully) complete.

I am looking forward to next week when we hope to get things setup for some gating in rawhide. I know it couldn't handle all these issues, but it's a start and we can add things as we know how to detect them in advance.