Just a few years ago, moving everything to the cloud seemed like a no-brainer. For many companies, the cloud offered greater flexibility than operating their own data centers. But the cloud is not always the right answer. This post is about one specific case where we decided to move back to our data center, and the reasons behind it.
Many companies face the challenge of efficiently processing large datasets for analytics. Using an operational database for such purposes can lead to performance issues or, in extreme cases, system failures. This highlights the need to transfer data from operational databases to data warehouses. This approach allows heavy analytical queries without overburdening transactional systems and supports shorter retention periods in production databases.
Hardware is always hard. The amount of operations, maintenance and planning that goes into supporting a data center is a daunting challenge for any enterprise. Though often unseen, without hardware there is no software.
Some time ago, our team faced the issue of moving an existing Apache Spark job from an on-premise Hadoop cluster to public cloud. While working on the transition we came across another way to process data that is Apache Beam. We were curious whether this tool had more advantages in comparison to traditional Apache Spark. We wanted to find the answer relatively quickly with minimal effort. Hence, we built two projects to process the same data using these technologies. Below you can get to know the architecture of the jobs written in Apache Spark and Apache Beam.
We are excited to announce that we have just released BigFlow 1.0 as open source. It’s a Python framework for big data processing on the Google Cloud Platform.
Configuration management is one of the key challenges you have to face when you decide to build an application as a distributed system based on microservices deployed to the Cloud. There are multiple ways of addressing different aspects of this problem, using several tools such as Spring Cloud Config Server or Hashicorp Consul. However, this article will focus on the tools that Google Cloud Platform offers out of the box. The approaches mentioned should be seen as complementary rather than mutually exclusive.
