Infrastructure Fedora 44 Beta freeze

We started the beta freeze in infrastructure. This is to make sure that we don't cause any problems for the release building and distributing pipeline. We require some acks for any changes to things that might impact those things until the day after the Beta is released.

I think this has served us fine over the years. Every once in a while I wonder if we could just stop doing it as we are usually pretty good about not breaking things day to day, but having the extra eyes on changes and slowing down a bit is a good thing I think.

Forge migrations

We have been busy working on migrating things from pagure.io to forge.fedoraproject.org. On tuesday just before the freeze we finally got our ansible repo moved over. I've really been looking forward to this as the review interface in forgejo is a good deal nicer than the pagure one. I've already used it to great effect.

We do still have a few more things to migrate, but overall it's moving along nicely.

Last bits of rdu2-cc move

We finally finished off the last things (at least that I am aware of) for things we moved in last december from rdu2-cc to rdu3.

There was a very strange and difficut to figure out problem for copr builders on ipv6 that I wasn't able to track down, but luckily Pavel worked with networking and finally did so! It seems to have been a odd caching bug in the switches. Hopefully it's now gone once and for all.

There was some hardware issues to sort out: some bad network cards that had to be replaced, a machine that didn't actually move when it was supposed to, etc.

Anyhow I hope all that work is all finally done.

signing work

Finally got back to deploying / testing the new signing path for secure boot signing. I got it all deployed, just need to get things tested now and hopefully we can switch over after the freeze.

This should hopefully allow us to sign aarch64 kernels for secure boot as well as removing reliance on an old smart card for signing.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/116110354434738317

Strange long httpd reload times on proxy11

I spent a fair bit of time looking at one of our proxies. We have them all to a reload (aka 'graceful restart') every hour when we update a ticketkey on them. For the vast majority of them, thats fine and works as expected. However, proxy11 decided to start taking a while (like 12-15seconds) to reload, causing our monitoring to alert that it was down... then back up.

In the end, it seemed the problem was somehow related to some old tls certificates that were present, but not used anywhere. All I can think of is that it's doing some kind of parsing of all certs and somehow those old ones cause it undue processing time. I removed those old certs and reload times went way back down again.

I'm tempted to try and figure out what it's doing exactly here, but I already spent a fair bit of time on it and it's working again now, so I guess I will just shrug and move on.

Anubis and download servers

A while back I had to hurredly deploy anubis in front of our download servers. This was due to the scrapers deciding to just download every rpm / iso from every fedora release since the dawn of time at a massive concurrency. This was saturating one of our 10G links completely, and making another somewhat full. So, I deployed anubis and it dropped things back to 'normal' again.

Fast forward to this last week, and my rush in deploying anubis came back to bite me. We have a cloudfront distribution that uses our download servers as it's 'origin'. Then we point all aws network blocks to use that for any fedora instances in aws. This is a win for us as then everything for them is cached on the aws side saving bandwith, and a win for aws users as that traffic is 'local' to them so faster and doesn't cause them to need to be billed for ingress either.

Last week, anubis started blocking CloudFront, so uses in aws would get a anubis challenge page instead of the actual content they were expecting. But why did it this just happen now? well, as near as I could determine, someone/scrapers were hitting the CloudFront endpoints and crawling our download server (fine, no problem there), but then they hit a directory that they handled poorly.

The directory was used/last updated about 11 years ago with a readme file explaining that the content was moved and no longer there. Great. However, also it had previous subdirectories as links to '.' (ie, the current directory). Since scrapers don't use any of the 20 years of crawling code, and instead just brute force things, this resulted in a bunch of requests like:

GET /foo/ GET /foo/foo/ GET /foo/foo/foo/

and so on. These are all really small (just a directory listing), so that meant it could make requests really really fast. So, after some point anubis started challenging those CloudFront connections and boom.

So, the problem with the hurred deployment I had made there was that The policy file I had deployed was not actually being used. I had allowed CloudFront, but it didn't seem to help any, and it took me far too long to figure out that anubis was starting up, printing one error about not being able to read the policy file and just running with the default configuration. ;( It turned out be a podman/selinux interaction and is now fixed.

I also removed those . links and set that directory tree to just 403 all requests to it.

Anubis and forge

Also this week, folks were reporting problems with our new forgejo forge. Anubis was doing challenges when people were trying to submit comments and it was messing them up.

In the end here, I just needed to adjust the config to allow POSTs through. At least right now scrapers aren't doing any POSTS and just allowing those seems to fix the issues people were having.

Some more scrapers

Friday we had them hitting release-monitoring.org. This time it was what I am calling a 'type 0' scraper. It was all coming from one cloud ip and I could just block them.

This morning a bit ago, we had a group hit/find the 'search' button on koji.fedoraproject.org, taking it offline. I was able to block the endpoint for a few hours and they went away, but no telling if they will be back. These were the 'type 2' kind (botnet using users ip's/browsers from 100's of thousands of different ips).

I am sad that the end game here sounds like there's not going to be so much of a open internet anymore. ie, for self defense sites will all have to go to requiring registration of some kind before working. I can only hope business models change before it comes to that.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/116070476999694239

Fedora 44 Branching

The big news this week was the Fedora 44 branching off rawhide. This is by far the most complicated part of the release. There's updates that have to happen in a ton of places all in the right order and with the right content.

Things didn't start when they were supposed to (tuesday morning), because we had some last minute mass rebuilds (golang and ghc). Then, they didn't start wed morning because we were trying to get the gnome 50 update to pass gating. Finally on thursday we just ended up unpushing that update and starting the process.

This time the releng side was run by Patrik. It's the first time he's done this process, but he did a great job! He asked questions at each step and we were able to clarify and reorder the documetation so I hope things will be even more clear and easy next cycle.

You can see the current SOP on it (before changes from this cycle): https://docs.fedoraproject.org/en-US/infra/release_guide/sop_mass_branching/ Look at all those steps!

This was also a bit of a long week because I am in PST and patrik is in CET, so I had to get up early and he had to stay late. Timezones are anoying. :)

Anyhow, I think things went quite smoothly. We got rawhide and branched composes right away, and only a few minor items to clean up and figure out how to do better.

Sprint planning meeting again monday

We had our last sprint planning meeting almost two weeks ago, so on monday it's time for another one. We did manage to run it in matrix, and although we did run over time I think it went not too badly.

I'll probibly do some prep work on things this weekend for it.

But if anyone wants to join in/read back it will be in #meeeting-3:fedoraproject.org at 15UTC on matrix.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/116030844840004998

rdu2cc to rdu3 datacenter move cleanup

In december, just before the holidays almost all of our hardware from the old rdu2 community cage was moved to our new rdu3 datacenter. We got everything that was end user visible moved and working before the break, but that still left a number of things to clean up and fully bring back up. So, this last week I tried to focus on that.

• There were 2 copr builder hypervisors that were moved fine, but their 10GB network cards just didn't work. We tried all kinds of things, but in the end just asked for replacement ones. Those quickly arrived this week and were installed. One of them just worked fine, the other one I had to tweak with settings, but finally got it working too, so both of those are back online and reinstalled with RHEL10.

• We had a bunch of problems getting into the storinator device that was moved, and in the end the reason why was simple: It was not our storinator at all, but a centos one that was decomissioned. They are moving the right one in a few weeks.

• There were a few firewall rules to get updated and ansible config to get things all green in that new vlan. That should be all in place now.

• There is still one puzzling ipv6 routing issue for the copr power9's. Still trying to figure that out. https://forge.fedoraproject.org/infra/tickets/issues/13085

mass update/reboot cycle

This week we also did a mass update/reboot cycle over all our machines. Due to the holidays and various scheduling stuff we hadn't done one for almost 2 months, so it was overdue.

There were a number of minor issues, many of which we knew about and a few we didn't:

• On RHEL10 hosts, you have to update redhat-release first then the rest of the updates, because the post quantium crypto on new packages needs the keys in redhat-release. ;(

• docker-distribution 3.0.0 is really really slow in our infra, and also switches to using a unpriv user instead of root. We downgraded back for now.

• anubis didn't start right on our download servers. Fixed that.

• A few things that got 'stuck' trying to listen to amqp messages when the rabbitmq cluster was rebooting.

This time also we applied all the pending firmware updates to all the x86 servers at least. That caused reboots to take ~20min or so on those servers as they applied, causing the outage to be longer and more disruptive than we would like, but it's nice to be fully up to date on firmware again.

Overall it went pretty smoothly. Thanks to James Anthill for planning and running most all the updates.

Some homeassistant fun

I'm a bit behind on posting some reviews of new devices added to my home assistant setup and will try and write those up soon, but as a preview:

• I got a https://shop.hydrificwater.com/pages/buy-droplet installed in our pumphouse. Pretty nice to see exact flow/usage of all our house water. There's some anoyances tho.

• I got a continous glucose monitor and set it up with juggluco (open source android app), which writes to health connect on my phone, and the android home assistant app reads it and exposes it as a sensor. So, now I have pretty graphs, and also figured out some nice ways to track related things.

• I've got a solar install coming in the next few months, will share how managing all that looks in home assistant. Should be pretty nice.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115991151489074594

Infra tickets migrated to new forejo forge

On tuesday I migrated our https://pagure.io/fedora-infrastructure (pagure) repo over to https://forge.fedoraproject.org/infra/tickets/ (forgejo).

Things went mostly smoothly, the migration tool is pretty slick and I borrowed a bunch from the checklist that the quality folks put together ( https://forge.fedoraproject.org/quality/tickets/issues/836 ) Thanks Adam and Kamil!

There are still a few outstanding things I need to do:

• We need to update our docs everywhere it mentions the old url, I am working on a pull request for that.

• I cannot seem to get the fedora-messaging hook working right It might well be something I did wrong, but it is just not working

• Of course no private issues migrated, hopefully someday (soon!) we will be able to just migrate them over once there's support in forgejo.

• We could likely tweak the templates a bit more.

Once I sort out the fedora-messaging hook I should be able to look at moving our ansible repo over, which will be nice. forgejo's pull request reviews are much nicer, and we may be able to leverage lots of other fun features there.

Mass rebuild finished

Even thought it started late (was supposed to start last wed, but didn't end up starting really until friday morning) it finished over the weekend pretty easily. There was some cleanup and such and then it was tagged in.

I updated my laptop and everything just kept working. I would like to shout out that openqa caught a mozjs bug landing (again) that would have broken gdm, so that got untagged and sorted and I never hit it here.

Scrapers redux

Wed night I noticed that one of our two network links in the datacenter was topping out (10GB). I looked a bit, but marked it down to the mass rebuild landing and causing everyone to sync all of rawhide.

Thursday morning there were more reports of issues with the master mirrors being very slow. Network was still saturated on that link (the other 10G link was only doing about 2-3GB/sec).

On investigation, it turned out that scrapers were now scraping our master mirrors. This was bad because all the BW used downloading every package ever over http and was saturating the link. These seemed to mostly be what I am calling "type 1" scrapers.

"type 1" are scrapers coming from clouds or known network blocks. These are mostly known in anubis'es list and it can just DENY them without too much trouble. These could also manually be blocked, but you would have to maintain the list(s).

"type 2" are the worse kind. Those are the browser botnets, where the connections are coming from a vast diverse set of consumer ip's and also since they are just using someone elses computer/browser they don't care too much if they have to do a proof of work challenge. These are much harder to deal with, but if they are hitting specific areas, upping the amount of challenge anubis gives those areas helps if only to slow them down.

First order of business was to setup anubis in front of them. There's no epel9 package for anubis, so I went with the method we used for pagure (el8) and just set it up using a container. There was a bit of tweaking around to get everything set, but I got it in place by mid morning and it definitely cut the load a great deal there.

Also, at the same time it seems we had some config on download servers for prefork apache. Which, we have not used in a while. So, I cleaned all that up and updated things so their apache setup could handle lots more connections.

The BW used was still high though, and a bit later I figured out why. The websites had been updated to point downloads of CHECKSUM files to the master mirrors. This was to make sure they were all coming from a known location, etc. However, accidentially _all_ artifact download links were pointing to the master mirrors. Luckly we could handle the load and also luckily there wasn't a release so it was less people downloading. Switching that back to point to mirrors got things happier.

So, hopefully scrapers handled again... for now.

Infra Sprint planning meeting

So, as many folks may know, our Red Hat teams are all trying to use agile and scrum these days. We have various things in case anyone is interested:

• We have daily standup notes from each team member in matrix. They submit with a bot and it posts to a team room. You can find them all in #cle-standups:fedora.im space on matrix. This daily is just a quick 'what did you do', 'what do you plan to do' any notes or blockers.

• We have been doing retro/planning meetings, but those have been in video calls. However, there's no reason they need to be there, so I suggested and we are going to try and just meet on matrix for anyone interested. The first of these will be monday in the #meeting-3:fedoraproject.org room at 15UTC. We will talk about the last 2 weeks and plan for what planned things we want to try and get in the next 2.

The forge projects boards are much nicer than the pagure boards were, and we can use them more effectively. Here's how it will work:

Right now the current sprint is in: https://forge.fedoraproject.org/infra/tickets/projects/325 and the next one is in: https://forge.fedoraproject.org/infra/tickets/projects/326

On monday we will review the first, move everything that wasn't completed over to the second, add/tweak the second one then close the first one, rename the 'next' to 'current' and add a new current one. This will allow us to track what was done in which sprint and be able to populate things for the next one.

Additionally, we are going to label tickets that come in and are just 'day-to-day' requests that we need to do and add those to the current sprint to track. That should help us get an idea of things that we are doing that we cannot plan for.

Mass update/reboot outage =========================o

Next week we are also going to be doing a mass update/reboot cycle with outage on thrusday. This is pretty overdue as we haven't done such since before the holidays.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115951447954013009

Holidays

So, how did that go? It was pretty good. I did manage to finish my december of docs with only missing one day. Closed about 40 tickets, made about 30 pr's for infra docs. Good progress. I might try and resume doing them, it is kind of nice to chip away at them over time.

I got a number of home projects done that I kept not having time for:

• Migrated all my bespoke firewall rules to nftables

• Fixed up my email setup to block more junk

• setup fail2ban finally when the ssh and dovecot and postfix brute forcers were making my logs hard to read. I wasn't worried about any of them getting in, but no reason not to just block them all.

Also did a bunch of learning about solar and home backups and batteries. I'm likely to pull the trigger on a solar system before too long.

Sadly, the world in 2026 is pretty horrible, but I will do my best to do what I can to help others and hope we can all collectively return the world to some sanity.

FESCo

fesco elections completed and I was elected again. Thanks so much everyone who voted for me. I will try and do my best (as always). If you have some concern or issue for fesco, feel free to let me know.

I think the slate of candidates were all excellent and I look forward to working with the new members to make fedora better.

Catchup

I tried to keep an eye out on things during the holidays, so catching up wasn't as bad as if I ignored everything, but still a lot to process through. I have now I think mostly done so, so if you asked me something or sent me a message looking for a reply and haven't seen it yet, please do check with me again.

mass rebuild

The f44 mass rebuild started, all be it a few days late. There was a large boost update and then some last minute tools builds that had to get in, but it's finally going now since yesterday. Typically these take a few days to rebuild most everything, then there's a few stragglers. I am not sure if it will be done by monday, but it shouldn't be too far off that. Then, we will merge things in and start getting ready for branching in a few weeks.

forge move

Next tuesday ( 2026-01-20 at 21UTC) I plan to migrate the fedora infra ticket tracker ( https://pagure.io/fedora-infrastructure ) to the new forge ( https://forge.fedoraproject.org/infra/tickets ). I sent out an announcement and will try and make things as smooth as possible.

If you do interact with infra you may want to login to forge and set your notifications as you like. Right now email notifications are not on by default, so if you want emails you need to enable them in your prefs.

I'm really looking forward to having this moved. We will likely look at moving our ansible repo after this and a few others. Our docs repo was already moved before the holidays.

Upcoming

I have a few things I really want to get to soon:

• I want to FINALLY land the pesign changes to sign secure boot via sigul. I have a pr I made last year, need to fix it up some, deploy and test.

• I want to look into the 502's people have been seeing and try and figure out where in the proxy stack thats happening and hopefully fix it up.

• We got over 100 tickets pending last week. I processed/fixed a number, but we are still up to 92 as of today. I'd like to fight those down again.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115911910269440220

Datacenter moves

There was one gigantic Datacenter move, and one smaller one. Overall I am glad we moved and we are in a much better situation for doing so, but I really hope we can avoid moving more in 2026. It takes so much planning and energy and work and there is so much else to do that has to be put on hold.

As a reminder, some of those advantages:

• We have lots of new machines. They are newer/faster/better in almost every way.

• dual 25G links on all the new machines (and 10G on old ones)

• all nvme storage in new machines

• space to expand for things like riscv builders and such.

• ipv6 support finally

So much of my year was datacenter moves. :(

Scrapers and outages

This year was sadly also the year of the scrapers. They are hammering pretty much everyone these days and it's quite sad. We did deploy anubis and it really helped a lot for most of the scrapers, but the's another group of them it wasn't able to. For those before the holidays I just tossed enough resources at our stuff that they can scrape and we can just not care. I'm not sure what next year will look like for this however, so we will just keep doing the best we can. I did adjust caching some that also really helped (all the src static files are cached now).

There were also a number of outages toward the end of the year, which I really am not happy about. There were a number of reasons for them:

• A tcp_timeout issue which turned out to be a firewall bug that was super hard to track down.

• The scrapers causing outages.

• I myself caused a few outages with a switching loop of power10 lpars. ;(

• Various smaller outages.

We have dicusssed a bunch of things to improve outages and preventing them, so hopefully next year will be happier on that front.

Power10

Speaking of power10, that was quite the saga. We got the machines, but the way we wanted to configure them didn't end up working so we had to move to a much more complex setup using a virtual hardware management console appliance and lpars and sr-iov and more. It's pretty complex, but we did get everything working in the end.

Fedora releases

We got Fedora 42 and 43 released this year, and pretty much on time too. 43 seems to be a release with a lot of small issues sadly, not sure why. From the postgresql upgrades, dovecot changing config format completely, nftables not enabled and matrix-synapse not being available, my home upgrades were not as smooth as usual.

Home Assistant

This was defintely a fun year of poking at home assistant and adding sensors and tweaking around with it. It's a nice fun hobby and does give you real data to solve real problems around your house. Also, all your data is your own and stored locally. This has really turned my perception of iot things all around. Before I woulde deliberately not connect things, now I connect them if they can be made only to talk to my home assistant.

I added a weather station, a rain guage, a new zigbee controller, a bunch of smart power plugs and temp sensors, and much more. I expect more on the way in 2026. Just when I think I have automated or instermented everything, there's a new thing coming along.

AI

I'm still in the 'There are in fact use cases for LLM's' group, but I am pretty weary of all the people wedging them in where they are not in fact a good use case, or insisting you find _some_ case no matter what.

I've found some of them useful for some things. I think this will continue to grow over time, but I think we need to be measured.

On the other side I don't get the outrage for things like calibre adding some support for LLM's. Its there, but it does exactly nothing by default. You have to set it up with your desired provider before it will do anything. It really doesn't affect you if you don't want to use it.

laptop

I have stuck with using my Lenovo slim 7x as my main laptop for most of this year. The main thing I still miss is webcam working (but I have an external one so it's not the end of the world). I'm looking forward to the X2 laptops out in the next few months. I really hope qualcomm has learned from the X1 ones and X2 will go better, but time will tell.

Fedmsg finally retired

We finally managed to turn off our old message bus. It took far too long, but I think it went pretty smoothly overall in the end. Thanks to much to Michal Konečný for doing everything around this.

nagios (soon to be) retired

Thanks to a bunch of work from Greg Sutcliffe, we have our zabbix setup pretty much done for a phase one and have officially announced that nagios is going to be retired.

iptables finally retired

We moved all our iptables setup over to nftables. There were a few hiccups, but overall it went pretty smoothly. Thanks to James Antill for all the work on this.

Blogs

I wrote a bunch more blogs this year, mostly for weekly recaps, but also a few home assistant review blogs. I find it enjoyable to do the recaps, although I don't really get much in the way of comments on them, so no idea if anyone else cares about them. I'll probibly continue to do them in 2026, but I might change it to do them sunday night or friday so I don't have to think about doing them saturday morning.

The world

The world was very depressing in 2025 in general, and thats speaking as someone living life on the easiest difficulty level ( https://whatever.scalzi.com/2012/05/15/straight-white-male-the-lowest-difficulty-setting-there-is/ ) I really hope sanity, science and kindness can make some recovery in 2026.

I'll probibly see about doing a 'looking forward to 2026' post soon.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115794282914919436

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115754591222815006

RUD2-CC to RDU3 datacenter move

This last monday was the physical datacenter move. It had been pushed back for various reasons, but I am glad we could get it done and over with this year.

Things didn't go as smoothly as planned unfortunately.

• There was bad weather in the area of the datacenters (snow and ice). The truck transporting things took a bit longer to arrive, the folks doing the move had to head home before things became impassible and also took longer to get back in to finish cabling. :(

• There was a miscommunication between planning folks and datacenter folks on the ground: we thought that everything was moving to dual 10G network (so networking can upgrade/reboot switches and we are fine). The folks doing the actual cabling thought that we were just reconnecting things the way the old datacenter was setup (1 1G connection). So, it took a while to get 10G all connected and configured.

• Of course there were some casualties too: One machine (our retrace server) had a broken rail. DC folks got it setup anyhow, but new rails are going to need to be installed soon. And another of our servers for some reason refuses to accept any mgmt passwords. That will need to be reset locally.

• There's one cursed machine that has a 10G network card in it, and lspci on the server shows it, but it has no interfaces for it, and the mgmt interface doesn't show it at all. Probibly the card is dead or otherwise needs vendor intervention.

Otherwise important things are back up with some reinstalling and cleanup to be done. Here's hoping for a datacenter moveless 2026!

Scraper news

I did a bunch of tweaking since last week to try and get things in a state where we could not need manual intervention for scraper handling. This included some httpd changes, some proxy changes and a bit of hardware tuning. So far, we are doing good. I haven't had to manually look at it much this week. We have still been under scraper load, but blocking the blame endpoint really helped along with the other tuning. I hope it will be a quiet holidays.

Decemeber of docs

So far we are just under half of december gone by, and so far I have kept up working on at least one docs pr/ticket every day.

We have moved infra docs over to forge now also!

You can see activity here:

https://forge.fedoraproject.org/infra/docs/activity/monthly

Hopefully I can keep it up. We are down about 21 tickets now. Perhaps I can even do a bit more now that I am on holidays.

Happy holidays everyone

Happy holidays everyone!

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115713891385384001

scrapers

I started a discussion thread about the current scrapers we are dealing with. To summarize, anubis has cut out a bunch of them and really helped out quite a lot. It has caused some issues with clients as well, but we have been working thought those as we hear about them. The remaining scrapers are large botnets of browsers, probibly running on end user machines. Those are more troublesome to deal with.

The discussion thread is at: https://discussion.fedoraproject.org/t/scrapers-and-ideas-for-how-to-deal-with-them/175760 if anyone would like to read or contribute.

We had another run in with them eariler this morning. A great way to spend saturday morning, but I did look more carefully this time. The main cause of issues was them hitting src.fedoraproject.org and it's /history/ and /blame/ endpoints. This was causing the backend to have to do a somewhat expensive git blame/history call to the local repos and since it took a bit to come back requests piled up and latency went way up. I have for now blocked those endpoints in the src.fedoraproject.org web interface. This brought everything back to normal. If you need to do those things, you can easily clone the git repo locally and do them.

rdu2-cc to rdu3 datacenter move

This last week, I moved pagure.io (virtually) to the new datacenter. Unfortunately it didn't go as smoothly as I had hoped. All the data synced over in about 15minutes or so, but then I tried to test it before switching it live and it just wasn't allowing me to authenticate on git pushes. Finally the light bulb went on and I realized that pagure was checking for auth, but it wasn't 'pagure.io' yet because I hadn't updated dns. ;( It's always DNS. :) After that everything went fine. There were a few loose I had to fix up the next day: mirroring out was not working because we didn't have ssh outgoing listed as allowed. Uploading releases wasn't working due to a selinux labeling issue, and finally our s390x builders couldn't reach it because I forgot they needed to do that. Hopefully pagure.io is all happy now and I even gave it more resources in the new dc.

Monday the actual physical move happens. See: https://pagure.io/fedora-infrastructure/issue/12955 for more details. Mostly, folks shouldn't notice these machines moving. abrt submissions will be down, and download-cc-rdu01 will be down, but otherwise it should be a big nothing burger for most folks. Machines will move monday and we will work tuesday to reinstall/reconfigure things and bring it all back up.

Matrix outage on dec 10th

There is going to be a short outage of our fedora.im and fedoraproject.org matrix servers. We are migrating to the new MAS setup (Matrix Authentication Server). This will allow clients to use things like element-x and also is a important step we wanted to complete before moving forward on deploying our own matrix servers.

forge migration

A number of groups have already moved over to forge.fedoraproject.org from pagure.io. I really was hoping to move infrastructre, but haven't had the cycles yet. We do have the orgs created now and are planning on moving our docs over very soon. I don't know if we will move tickets before the end of the year or not, but we will see.

December of docs

So, I committed myself to doing a docs pr/issue/something every day in December, and so far I am doing so! 6 days and 6 PR's and more tickets updated. Hopefully I can keep it up.

comments? additions? reactions?

As always, comment on mastodon: https://fosstodon.org/@nirik/115674367344830186