Apache Mesos is an open-source project to manage computer clusters. In this article we present one of its components called Executor and more specifically the Custom Executor. We also tell you why you should consider writing your own executor by giving examples of features that you can benefit from by taking more control over how tasks are executed. Our executor implementation is available at github.com/allegro/mesos-executor
Some of our engineers run their own tech blogs. We encourage them to move here, but for various reasons, they prefer to publish on private blogs. We respect their decisions. What we can do is to gather all the blog posts published by allegro.tech engineers around the web in the one place.
In this post I present a story of a bug that hit us recently. Everything was caused by unexpected (although documented) behavior of Go built-in function append. This bug has lived silently for nearly a year in allegro/marathon-consul. Ensure you run the latest version.
I bet you have found this article after googling some of the issues you encounter when working with a Hadoop cluster. You probably deal with Hive queries used for exploratory data analysis that are processed way too long. Moreover, you cannot adapt Spark in your organization for every use case because of the fact that writing jobs requires quite strong programming skills. Clogged Yarn queues might be your nightmare and waiting for the launch of the container when you run even a small query drives you mad. Before we deployed Presto — a Fast SQL engine provided by Facebook — our analysts struggled with these problems on a regular basis.
Last year Agile Testing Days Conference was held between 6th and 8th December in Potsdam, Germany. According to statistics provided by organizers there were around 600 attendants and about 30 speakers in keynotes / workshops. During the whole event around 22 different workshops took place, 9 keynotes and around 30 other speeches / presentations divided into 7 parallel tracks. I had the opportunity to attend the first two days of the conference and in this article I’d like to share with you a short review of the most interesting sessions and my impressions about the event.
In our services ecosystem it’s usually the case that services can handle a limited amount of requests per second. We show how we introduced a new algorithm for our publish-subscribe queue system. The road to production deployment highlights some key distributed systems’ takeaways we’d like to discuss.
Running Mesosphere Marathon is like running… a marathon. When you are preparing for a long distance run, you’ll often hear about Hitting the wall. This effect is described mostly in running and cycling but affects all endurance sports. It happens when your body does not have enough glycogen to produce power and this results in a sudden “power loss” so you can’t run anymore. At Allegro we have experienced a similar thing with Mesosphere Marathon. This is our story on using Marathon in a growing microservice ecosystem, from tens of tasks and a couple applications, to thousands of tasks and over a hundred applications. If there is no mention of Marathon version, it is 1.3.10 and below; we need some time to test and deploy the latest 1.4 release. If you are interested in how our ecosystem is built, take a look at below MesosCon presentation.
Caching is a good and well-known technique used to increase application performance and decrease overall system load. Usually small or medium data sets, which are often read and rarely changed, are considered as a good candidate for caching. In this article we focus on determining optimal cache size based on big data techniques.
Understanding data model is sufficient to design good database schema in RDBMS (relational database management system). Having this knowledge you are able to construct normalized tables, add appropriate constraints and finally create indexes to speed up queries. In the world of NoSQL there are no simple solutions, rules and answers. That’s why we can only talk about patterns, tips and hints. MongoDB is not an exception. Besides the comprehension of stored data, deep understanding of an access pattern, how data is searched, inserted and updated by an application is needed.
At Allegro we are faced with a technical challenge: how to recognize whether a given image (a product thumbnail) shows just a product itself. One of the things that we would like to detect is when the product is surrounded by a frame. In this post we would like to present our approach for detecting a frame in the image.
Are you developing a service and planning to make it publicly available? Do you want developers of mobile applications to get integrated with your API fast and painless? Do you care how stable and predictable mobile applications using your service are?
We’re all familiar with TDD, or at least write unit tests for our software, but unit tests won’t check application state after complex UI interactions. If you want to make sure that an application behaves correctly when users interact with it, then you need to write UI tests.
In July we attended the scientific conference SIGIR 2016 held in Pisa, Italy. SIGIR is an annual conference related to new advances in Information Retrieval. The shortcut is for Special Interest Group on Information Retrieval. This is an annual conference with the highest ranking at Microsoft Academic Research in Information Retrieval field field and the 16th in the ranking in the whole Computer Science field.
In everyday work programmers are facing various problems. We would like to focus on two of them: big systems with non-blocking API and specific business needs that can be solved using Expression Language. Hold tight!
High performance and availability are always hard to achieve, and microservice architecture is no exception. In the microservices world, every user request sent to a frontend application triggers a cascade of remote calls. The frontend application calls API facades, API facades ask backend services, backend services communicate with databases and even more backend services. Statistics work against you. Latencies add up but failure probabilities combine by multiplication. The more services are engaged in a flow, the more threats for performance and availability.
This article is a part of CQK Top 10 series.
This article is a part of CQK Top 10 series.
This article is a part of CQK Top 10 series.
Code reviews play an important role in how we develop software at Allegro. All code we developers write is reviewed by our peers. If you apply for a job with us, we may ask you to review a sample piece of code during your interview. A code review done right carries a lot of value, but if done wrong it can become a waste of time. In this article I will describe what I think makes a good code review, how reviews have evolved over time at the teams I worked with and what you can do in order to make code reviews worthwhile.
At Allegro we want to be sure that our software works as designed. That’s why tests are so important to us. In several projects we are using Elasticsearch. In order to make writing integration tests that uses Elasticsearch easier, we’ve created a little tool called embedded-elasticsearch. It sets up Elasticsearch instance that you need for your tests (including installation of plugins) and gives you full control of it.
This article tells a story of chasing an iOS bug – a bug hidden so deep that it required many different skills and debugging on different levels to identify it. I think every native mobile app developer (not only an iOS developer) will find this text interesting. Non-mobile developers may find it an intriguing read as well.
Kotlin may seem like a new kid on the block — itʼs been officially released only in February. Its history however dates a few years back and itʼs mature and stable enough to be used for developing solid reliable applications. Therefore at Allegro we decided to give it a chance — we built our new shiny server-side system using Kotlin as its primary language and we do not regret it.
In this post I will introduce to you a recently released version of a well known library for consuming RESTful services — Retrofit2. Even though it is mainly targeted at Android platform it works very well on the “server” Java. Its lightness and low garbage generation overhead make it an interesting option if one does not like existing solutions (like Jersey Client, or Spring’s RestTemplate). I will also show how to configure it properly and fully utilize its great RxJava compatibility.
Recently our team has been tasked to write a very fast cache service. The goal was pretty clear but possible to achieve in many ways. Finally we decided to try something new and implement the service in Go. We have described how we did it and what values come from that.
In the beginning I would like to stress that this is not yet another article about Minimum Viable Product (MVP) mechanics. For theory and examples of its use in practice, please refer to the great article by Andrzej Winnicki. What made me share my thoughts on MVP is what I consider a prerequisite to start working with this method. Moreover, I also recognised this prerequisite as the biggest obstacle which is stopping many enthusiasts from fully understanding it.
