Mastodon Instance Operators Report on the Impact of the #TwitterMigration

This report covers the San Francisco Bay regional Mastodon instance sfba.social, including general statistics, financial details including income (donations) and expenses (hosting costs), moderation efforts, and changes made and considered. It has been wonderful to welcome all of our new friends and neighbors. We have expanded our server capacity and refined our moderation process, including a new version of the code of conduct and updated the server rules to match. This has helped to improve expectations and free our users to be nice and have fun!

Since April of this year I‘ve been running my own Mastodon server and 3 days ago we hit 100 users which was a huge milestone for my tiny little server... and then all of a sudden something happened, the other Mastodon servers started to get full and new users were looking for homes. Less than 72 hours after being excited for hitting 100 users we hit 10,000 users.

The past month has changed the Fediverse, and, by extension, our instance. We‘ve continued as normal (apart from limiting sign-ups) to give ourselves time to figure out which changes were only temporary, what seems to be changed for good, and how to react. A month seems ample time, and here we are with a set of changes in how chaos.social will work in the future.

I was going to write an article for a while now, but there was too much work to do with the latest influx. Together with my co-admin @rixx we run the chaos.social instance. As of writing this, we are one of the biggest and most active instances on the fediverse and one of the oldest mastodon instances, starting mid of April 2017. For the last 5 years everything was simple, one VM with 10 cores, 500GB NVMe SSD Storage and 32GB RAM. This VM did everything from the database to the webserving. Then Musk happened.

This is honestly a very hastily written selection of various snippets, with text extracted, and notes. No real editing thought was put into this, so I hope it‘s not too confusing. This blog post will be kept up to date as I find out more information and publish my findings. It‘s currently organized in no particular order with a bunch of micro fragment thoughts split out in a row.

Mastodon has recently gained popularity amongst tech-savvy users after Elon Musk has bought Twitter. With November‘s wave of new Mastodon users, many servers experienced mayor problems with their performance - so did metalhead.club, the Mastodon instance that I‘m hosting myself. Here‘s how I tackled performance issues on metalhead.club.

The steep line itself is when I realised that doing it that way actually gave us less control over new user numbers, so we went back to ‘anyone can request an account but accounts need to be approved by a mod’. This was about the time that big news sites had started reporting people were leaving Twitter for Mastodon, and they were all linking to joinmastodon.org, where we were one of three art instances listed and the biggest one by far... We had 2000 new user accounts placed in the approval queue that day. These weren‘t approved accounts, so they weren‘t logged on and using .art - so they weren‘t contributing much (aside from the verification emails being sent out) to server activity. Still, server activity skyrocketed.

If you haven‘t heard, Elon Musk recently agreed a deal to buy Twitter for a whopping $44bn. That cause a little turmoil over in Twitter land, which resulted in expats coming over to Mastodon in their droves. Because we‘re one of the biggest technology focussed instances on the Fediverse, a lot of people requested an account on Fosstodon. As you can see from the graphs below, the spike in users from Twitter made our usual activity almost flat line. This effectively resulted in a sustained DDoS that lasted for around 36 hours. As you can imagine, that was a lot of fun for myself and the team. Here‘s the details of what happened during those 36 hours…

Woooooh! It‘s been a crazy few weeks at Fosstodon HQ. If you‘ve been living under a rock (or are just not interested) Elon Musk, or better known as Melon Tusk on the fediverse, has taken over Twitter and looks to be on a mission to ruin the platform even more. Because of this, people have been flocking to Mastodon in their droves, and since Fosstodon is one of the biggest tech-focussed instances on the fediverse, we‘ve seen A LOT of that traffic.

The instance grew from less than 60 users to what is now 24 500 users in the span of just two weeks. It went from an experiment to a service demanding significant infrastructure, moderation, and governance. I want to offer a warm welcome to everyone and thanks for their patience as I bolstered Mastodon Canada with the necessary server capacity to handle the load.

Hachyderm has reached 30,000 users. A ‘small sized’ service in regard to scale. However, in the process we have hit very familiar ‘medium sized’ scale problems which caused us to migrate our services out of my basement. This is the outage report, post mortem, and high level overview of the process of migrating to Hetzner in Germany. From observation to production fixes. This is the story.

Thank you to everyone who has been patient with Hachyderm as we have had to make some adjustments to how we do things. Finding ourselves launched into scale has impacted our people more than it has impacted our systems. I wanted to provide some visibility into our intentions with Hachyderm, our priorities, and immediate initiatives.

November was a heck of a time for Cloud Island, and the Fediverse as a whole. With the Twitter purchase, the Fediverse as a whole saw a massive increase in use and traffic. Cloud Island is seeing a 15x sustained increase in traffic on Sidekiq, which is the behind-the-scenes processing layer that interacts with the rest of the Fediverse. Cloud Island is seeing 1700% more interactions through November than in October. And, finally a 12.5x increase in the amount of bandwidth being used from October to November. In October, we used about 100GB of bandwidth. In November, we used 1.25TB of bandwidth.