In the coming weeks, we are going to apply major software upgrades across our whole hosting infrastructure.

After many months of preparation, we are finally getting into the rollout phase of one of our longest-running engineering projects. The reason why it took us more than a whole year to get to this point is what our industry likes to describe as “technical debt”.

Why this was a long time coming

Since the inception of our business, we have been practising “Infrastructure as Code”, i.e. automating our hosting platform using technologies like Chef, Consul and Terraform. Maintaining this growing code base (currently about 45k lines of code) is a big part of our engineering work. And we have to be honest, it’s become a challenge in recent years. Starting out, it helped us significantly that we fought the urge of the “Not Invented Here” syndrome and relied on open source code for basic infrastructure setup. It allowed us to iterate quickly on what made freistilbox special. Over time, however, more and more of these external dependencies stopped being maintained and updated. This became apparent when we started preparations for this operating system upgrade. Instead of just making sure that our code was working on the new OS version and then rolling it out, we first had to untangle our whole dependency tree and replace all the dead parts with alternatives we either found in open source or had to rewrite ourselves.

It took months and months of work to arrive at this point, and we had to put many other projects on hold for it. The upside is that, aside from our code, we were also able to modernise our infrastructure tooling. It will allow us to roll out this and future upgrades more efficiently than in the past.

What we are going to do

In the coming weeks, we’re going to upgrade operating systems and infrastructure code throughout our hosting infrastructure. For most services, we’ll be able to make the process transparent to the outside. Since our web hosting architecture is designed for high availability, we can make use of its built-in redundancies to upgrade the majority of its services without downtime. Instead of upgrading existing servers in place, we’ll add new boxes to your running freistilbox clusters. After verifying that the new boxes have taken over website operation properly, we’ll remove the old ones.

There is one exception where you are going to notice changes, and that’s the shell boxes to which you have secure login access. Your cluster’s new shell box will have a different IP address and SSH host key. We’ll update our DNS records, therefore you can keep using the familiar hostname. However, your SSH client will notice that this is not the old familiar box and issue a warning message. Don’t get worried.

What about migrating existing website and user data? We’re happy to assure you that we’re taking care of most of it for you. The new web and shell boxes will have your current application code deployed and they’ll be attached to the same central network storage system that stores your websites’ asset files. Additionally, we’ll transfer the cron jobs you’ve scheduled on your old shell box to your new one.

What we are not going to transfer are the contents of user home directories on shell boxes. Since in our experience, a big portion of this data is just leftovers from past maintenance work, transferring it would unnecessarily prolong the system migration. This means that users who need certain files in their login home directory will have to download them from the old shell box while it’s still available. You can then upload them to the new shell box later.

How we’re going to do it

Starting end of January, we’re going to do the upgrades on a cluster by cluster basis. A few days before we upgrade a cluster, we’ll make a maintenance announcement on the freistilbox status page. If you haven’t made the preparations mentioned above yet, this is the time to do so. We’ll let you know as soon as your new cluster has been set up, and you can verify yourself that the transition went without flaws. After a brief grace period, we’ll remove the old boxes.

Any questions? We’re here.

If you have any questions or want to make special arrangements for your cluster upgrade, feel free to get in touch with us via the freistilbox dashboard. We’re going to make this transition as smooth as possible for you.