Back to the data center in 2026 (just a little bit)
Just a few years ago, moving everything to the cloud was a no-brainer. The cloud was cheaper and more flexible than running your own data center. But the cloud is not always the right answer. This post is about one specific case where we decided to move back to our data center, and the reasons behind it.
Our setup #
My team owns a service that runs in the Allegro data center, on a Kubernetes (k8s) cluster. The service depends on a pipeline running in GCP that stores the results in Google Cloud Storage (GCS). The service downloads the latest files from GCS on startup.
This seemed like a good idea, and we already used this pattern before. What was different this time was the amount of data — the files together total over 10 GB.
The incident #
Our architecture was vibe-checked one day, when a bug in our pipeline caused the files to double in size, causing out-of-memory errors during service startup. The service was restarted by the supervisor, but failed again and again. It was stuck in this loop of starting and failing for almost two hours.
This caused repeated downloads of these files, which resulted in a large bill for egress traffic. During these two hours it burned around $1,000.
Before this incident, we were not aware that this failure mode would result in such large costs. The engineer on-call (that’s me) thought the lone “container restarts per hour” alert was a harmless false positive, caused by a recent deployment.
Once we correctly diagnosed the core issue, we were able to fix it and the service operated normally afterward. However, we wanted to avoid such incidents in the future.
What we considered #
We considered multiple approaches before coming to a solution. Some of them were:
- Limiting the number of restarts — unfortunately, k8s does not support setting a maximum number of restarts for Deployment pods
- Private interconnect between GCP and our data center — we needed a short burst of high bandwidth, and the interconnect we had was not set up for this traffic pattern
- Init containers — our platform team uses them for infrastructure services, and we were not able to use them for application code
- Persistent volumes — doesn’t work well with autoscaling
- Leader-follower setup with a custom cache — too complex to implement, many corner cases, and high maintenance cost
The solution #
We decided to use Ceph: an S3-compatible storage already hosted in our data center. Our pipeline pushes the prepared files to the storage after they are built. When the service starts, it downloads the latest files without incurring GCP egress costs.
So far we have not had any new problems with this approach, and the service starts up quickly and reliably.
Conclusion #
While the cost of cloud services can be negligible when everything works as planned, it can balloon unexpectedly when things go wrong. Identifying and proactively addressing these architectural traps is crucial to avoid unnecessary costs. Sometimes that means moving back to the data center — even in 2026.