Get in touch
04 Juni 2020
In this post, Eryk Zalejski, DevOps Consultant at Russmedia Consultant, will show you how to share files across Kubernetes cluster – the easy way.
This post was also published on Medium.
Bit of Erento’s background
My journey with Erento started back in 2015. I joined a company that was transforming from a self-hosted monolith to a cloud-based microservices architecture. We also decided to go on with Kubernetes. We went with one of the very first stable releases 1.0.5, we had no resources limits control, no affinity, even simple deployment resource was far distant. It was a very interesting journey and we learned a lot.
Journey to the clouds
In short: First, we landed on AWS with self-managed Kubernetes Cluster, separate etcd cluster and CoreOS as a system backbone. Everything was running smoothly but the maintenance of such setup absorbed quite a lot of time and our ops resources were quite limited (it was me only). We slowly observed growing google cloud ecosystem, and when google launched managed Kubernetes we decided it is time to go. It turned out to be a great decision. Managed infrastructure saved us a lot of time and money.
There is always an issue that follows you
In our case, one of the issues was always managing customer-uploaded files. On our old datacenter, we had NFS with all of its downsides. It was running on one server and was pretty static, but the whole infrastructure was. When we were doing a redesign we wanted to pick something modern. For us the crucial points were:
We did a lot of tests with managed filesystems like AWS EFS and Google Cloud Filestore but they were lacking a snapshotting feature. We needed an option to quickly restore/recreate the whole cluster and refresh our dev environments.
GlusterFS to the Rescue
Finally, we found the tool that ticked most of our boxes — GlusterFS backed by cool Google Cloud-based GlusterFS setup made by rimiusz. We were only lacking some terraform management — so we decided to write our own terraform module.
The base idea is very simple. We run 3 nodes in a separate subnetwork. Each has a static disk attached. We run 3 copies of each file, so we have a very high redundancy. We chose such setup to be able to restore the whole cluster using just a snapshot of one disk.
Before creating the Terraform module for GlusterFS solution, we have also investigated different approach with hosting GlusterFS on Kubernetes itself (as described here).
We have decided not to host on Kubernetes for a few reasons:
– with our solution, multiple Kubernetes clusters can access GlusterFS cluster, which brings us high availability and failover
– separation of applications and static hosting layer allows us to maintain and upgrade Kubernetes clusters without any risk of impacting GlusterFS
– solution with separated GlusterFS is simpler to maintain
Open sourcing the terraform module
After the period of testing we realised that we are really happy with the module, it is maintenance-free. Other companies within Russmedia Equity Partners also had similar needs. So together with erento’s CTO Konrad Cerny, we decided to go all the way and share it back to the community: https://github.com/erento/terraform-google-glusterfs
FYI, we are very passionate about OpenSource here 🙂
The module can be configured to:
So… Now to create the cluster all you need to do is define:
And after the terraform run, you will get a ready to apply YAML with endpoints and service.
All you need to do is to apply them to your Kubernetes cluster:
To consume endpoints in your app all you need to configure, is to add natively supported GlusterFS volume and mount in your deployment:
We would like to run GlusterFS similarly to Elasticsearch clusters — as a build-in Kubernetes setup (via. stateful sets). We also plan to create the helm chart for GlusterFS.
It took us a while to find the solution that would suit our needs almost perfectly. As usual, there is never a one size fits all option, but if this article can help you and users within the company we will be very pleased. Erento is part of Russmedia Equity Partners. We are a group of companies that embrace knowledge and tech sharing. We regularly meet at internal conferences and organize calls to make it happen. It is seriously pushing us forward and we are hoping to spread this kind of spirit all over the world. Please make sure to follow Russmedia Equity Partners on Medium. Enjoy!
04 Februar 2021
You’re next Let’s go