Statement from Fosshost's Founder

Dear community,
I want to make a statement regarding the recent announcements on the Fosshost website and social media.

After deep discussions, the company directors have come to the conclusion that it is untenable to continue to run Fosshost in its current form. In simple terms, Fosshost will withdraw from offering hosting services directly to the free software community.

I want to explain how we arrived at this situation, as that detail is vital in understanding our intentions of how we intend to put things right from now on.

The Beginnings

We made a mistake by expanding too far and too fast, and Fosshost could not handle the burdens of developing. We struggled to address both the needs of our tenants and our own needs as our footprint grew beyond our volunteer time and comfort levels.

Very quickly, during our early stages, we had infrastructure deployed in multiple continents and data centres, each operating independently. These early design choices came with limitations, such as making our ability to automate and scale the infrastructure difficult, resulting in lots of manual work to maintain the x86 architecture.

I needed to take time out quickly after the project expanded because I was mentally and physically exhausted. During the first six months of Fosshost, I barely slept whilst covering support and processing new applications across multiple timezones (Not to mention that this all happened in the midst of the COVID-19 pandemic). This is not anyone's fault, but it's essential to understand the failures as part of a lessons learnt approach.

While I was away, new volunteers joined and I became less involved with the day-to-day technical operations of the project. My involvement in the project being less active was not a surprise to those close to the project and was shared openly. We continued to receive massive surges of applications, increased support tickets and security incidents. An ever-growing list of items went on the backlog. I accept full responsibility for making myself a single point of failure and not producing adequate documentation where x86 was concerned.

AArch64

I continued talking to new prospective sponsors and the open-source community at large, and we started a project called AArch64, which was successful thanks to hardware donated by Ampere Computing. The AArch64 project was designed to make access to ARM computing more accessible. We had a great team onboard helping us build out the platform, but Nate and Hammy notably committed significant amounts of their time. To accelerate plans, we used donations to pay for colocation services and equipment. To accelerate plans, we used donations to pay for colocation services and equipment.

The board agreed to sign new colocation agreements; however, we needed physical engineering resources to active these locations, resulting in vacant colocation racks. We used the funds to grow even faster, and before we knew it — the technical team was growing, new services were being deployed and systems were being launched. We had no change control procedures, and suddenly, AArch64 was taking over x86.

We entered into discussions with a well-known chip designer to help with our AArch64 expansion plans. These discussions did not transpire into reality because the vendor expperienced a significant restructuring process, and Fosshost was not a priority for them. Ultimately, we made commitments to suppliers based on these assurances.

The Peak

No sooner, we were operating multiple technologies across lots of different hardware globally as an ISP. Fosshost was supported exclusively by a team of part-time volunteers which resulted in a fragmented approach whilst trying to live up to our expectations of providing a world-class hosting experience. Expectations by the community were that we were almost like a business. Division within the camp created some tension with volunteers, and as the project leader, I failed to lead the project through some difficult times. Corwin and I had discussions around this time about working it out and trying to make the best out of the situation, but morale was low and our busy lives outside Fosshost took priority. Fosshost again lacked leadership.

Today

There has been a lot of confusion and misunderstandings due to that lack of leadership. This has resulted in a catalogue of errors and mistakes.

As it stands: Fosshost is still live and operational.

We will publish plans of how we intend to wind down our operations and continue to fulfill all of our supplier contracts. Critical hosting projects remain on our radar, and of course, we want any migration of hosting services to new homes for your projects to be smooth.

Moving Forward

Fosshost services will be sunset, and Fosshost will cease to exist as a direct-to-tenant hosting solution. However, ending our tenant solutions means we still have to continue our mission to help the free software world.

Through our triumps and failures, our Fosshost team members have better understood what to do and what not to do in our field.

We've experienced working with datacentres, hardware manufacturers and cloud providers to find partnerships where we have the resources to provide others with what they need.

We've learned of the challenges involved in crafting a robust governance model and know what can happen when the organisational model itself is not molded with longevity in mind.

We've discovered the challenges in widening our scope and experienced first-hand the problems that arise when our user base goes from a few dozen to a few hundred and more.

Lastly, we've proven that being the only organisation trying to tackle this problem means everyone gains when we don't do it right.

Thus moving forward, Fosshost will shut down, and we'll be applying our combined knowledge to help new organisations prosper.

We will do what we can to help new players in the space find agreements with providers to host their ecosystem, find hardware to power it, and help providers to host their ecosystem, and help recommend solutions to help keep them afloat.

We'll also be divesting Fosshost's existing resources, including our hardware assets and any vendor agreements we can transfer, and providing them as grants to organisations trying to solve the same goals we were.

We could have done better. We did a lot wrong. And we're sorry to both our partners, volunteers, and tenants. We sincerely hope that by helping those hosting solutions prosper, we can start to make things right for all of you.

Thomas Markey,
Fosshost Founder