In this article, we want to share our journey of searching for optimizations in one of Allegro’s main microservices: opbox-web. You’ll read about the issues we had to deal with and how we managed to overcome them — together with a few surprises along the way and even one golden rule broken.
When we think about the Circuit Breaker pattern, we instantly associate it with the HTTP client. Just make some annotation or wrapper and proceed with coding. In this article, I will encourage you to use this pattern to resolve business problems. Based on a live example from Allegro I will show you how to use the implementation of CircuitBreaker from Resilience4j library for cases other than HTTP calls.
Did you know that in October this year, DRY principle will celebrate its 25th anniversary? It was proposed by Andrew Hunt and David Thomas in The Pragmatic Programmer book in 1999. 25th birthday is quite a good reason to celebrate, isn’t it? At least, it’s a good opportunity to bring this principle back into the spotlight and to discuss how to use it properly.
Hi, I am Magda and I will tell you a story about coming back to work after a break of 21 months and 2 days. Everything here will be a subjective perspective about my experience.
Are you, as a test automation engineer, tired of Selenium’s flakiness? Are you seeking a better tool to automate your end-to-end tests? Have you heard of Playwright? Perhaps you’ve encountered opinions that it is only worth using within a Node.js environment. I have. And as a tester, I decided to verify if this is true. If you’re interested in the results, I encourage you to read the following article.
This article is a case study of how we improved stability in our critical application. It’s mostly a technical analysis of what happens in fresh Java based instance, how JIT Compiler toyed with us at application start and how we learned to control it.
If you have experience with Event Storming and have ever found yourself wishing there was a way to document the insights gathered during a session, or wanting to communicate the process to other team members, then I have a solution for you. This idea can be expressed in a famous saying: One picture is worth more than a thousand words.
Site performance is very important, first of all, from the perspective of users, who expect a good experience when visiting the site. The user should not wait too long for the page to load. We all know how annoying it can be when we want to press an element and it jumps to another place on the page or when we click on a button and then nothing happens for a very long time. The state of a site’s performance in these aspects is measured by Web Vitals performance metrics and most importantly by a set of three major Core Web Vitals metrics (LCP — Largest Contentful Paint, CLS — Cumulative Layout Shift, INP — Interaction to Next Paint). They are responsible for measuring the 3 things: loading time, visual stability and interactivity. These metrics are also important for the websites themselves because, in addition to the user experience, they are also taken into account in terms of the website’s positioning in search engines (SEO), which is crucial for most websites on the Internet, Allegro included.
In this article we’ll present methods for efficiently optimizing physical resources and fine-tuning the configuration of a Google Cloud Platform (GCP) Dataflow pipeline in order to achieve cost reductions. Optimization will be presented as a real-life scenario, which will be performed in stages.
One tech blog/newsletter gained traction and popularity for a couple of years now: Pragmatic Engineer.
This story shows our journey in addressing a platform stability issue related to autoscaling, which, paradoxically, added some additional overhead instead of reducing the load. A pivotal part of this narrative is how we used Couchbase — a distributed NoSQL database. If you find yourself intrigued by another enigmatic story involving Couchbase, don’t miss my blog post on tuning expired doc settings.
In early 2024, I hit ten years at Allegro, which also happens to be how long I’ve been working with microservices. This timespan also roughly corresponds to how long the company as a whole has been using them, so I think it’s a good time to outline the story of project Rubicon: a very ambitious gamble which completely changed how we work and what our software is like. The idea probably seemed rather extreme at the time, yet I am certain that without this change, Allegro would not be where it is today, or perhaps would not be there at all.
At Allegro, we use Kafka as a backbone for asynchronous communication between microservices. With up to 300k messages published and 1M messages consumed every second, it is a key part of our infrastructure. A few months ago, in our main Kafka cluster, we noticed the following discrepancy: while median response times for produce requests were in single-digit milliseconds, the tail latency was much worse. Namely, the p99 latency was up to 1 second, and the p999 latency was up to 3 seconds. This was unacceptable for a new project that we were about to start, so we decided to look into this issue. In this blog post, we would like to describe our journey — how we used Kafka protocol sniffing and eBPF to identify and remove the performance bottleneck.
Have you ever thought about ways of reducing repetitive, monotonous tasks? Maybe you would like to try to automate your own tasks? I will show you what technology we use at Allegro, what processes we have automated, and how to do it on your own.
This story shows how we strive to fix issues reported by our customers regarding inconsistent listing views on our e-commerce platform. We will use a top-down manner to guide you through our story. At the beginning, we highlight the challenges faced by our customers, followed by presenting basic information on how views are personalized on our web application. We then delve deeper into our internal architecture, aiming to clarify how it supports High Availability (HA) by using two data centers. Finally, we advertise a little Couchbase, distributed NoSQL database, and explain why it is an excellent storage solution for such an architecture.
Ready to turn web accessibility from a headache into a breeze? Join us as we demystify WCAG, explore its latest 2.2 version, and gaze into the future of digital inclusivity. Get ready for a journey that’s as enlightening as it is entertaining!
Icons are an integral part of most modern UIs. What is the best way to embed icons nowadays?
This article is a form of a public postmortem in which we would like to share our bumpy way of revealing the cause of a mysterious performance problem. Besides unveiling part of our technical stack based on open-source solutions, we also show how some false assumptions made such a bug triage process much harder. Besides all NOT TO DOs, you can find some exciting information about performance hunting and reproducing performance issues on a small scale. As a perk, we prepared a repository where you can reproduce the problem and make yourself familiar with tools that allowed us to confirm the cause. The last part (lessons learned) is the most valuable if you prefer to learn from the mistakes of others.
B-tree is a structure that helps to search through great amounts of data. It was invented over 40 years ago, yet it is still employed by the majority of modern databases. Although there are newer index structures, like LSM trees, B-tree is unbeaten when handling most of the database queries.
The idea for this article arose during a meeting where we learned that our supervisor would be leaving the company to pursue new opportunities. In response, a colleague lamented that what we would miss most is the knowledge departing with the leader. Unfortunately, that’s how it goes. Not only do we lose a colleague, but we also lose valuable knowledge and experience. However, this isn’t a story about my supervisor; it’s a story about all those individuals who are experts in their fields, who understand the paths to success and paths that lead to catastrophic failures. When they leave, they take with them knowledge that you won’t find in any book, note, or Jira ticket. And this leads to a fundamental question: What can be done to avoid this “black hole” of knowledge? How can we ensure it doesn’t vanish along with them? That’s what this article is all about.
MongoDB is the most popular database used at Allegro. We have hundreds of MongoDB databases running on our on—premise servers. In 2022 we decided that we need to migrate all our MongoDB databases from existing shared clusters to new MongoDB clusters hosted on Kubernetes pods with separated resources. To perform the migration of all databases we needed a tool for transfering all the data and keeping consistency between old and new databases. That’s how mongo-migration-stream project was born.
After six years as a Team Leader, I went back to hands-on engineering work, and I’m very happy about taking this step. While it may appear surprising at first, it was a well-thought-out decision, and actually I’ve already performed such a maneuver once before.
In the era of ubiquitous cloud services and an increasingly growing PaaS and serverless-oriented approach, performance and resources seem to be becoming less and less important. After all, we can scale horizontally and vertically at any time, without worrying about potential performance challenges that the business may introduce.
As a part of a broader initiative of refreshing Allegro platform, we are upgrading our internal libraries to Spring Boot 3.0 and Java 17. The task is daunting and filled with challenges, however overall progress is steady and thanks to the modular nature of our code it should end in finite time. Everyone who has performed such an upgrade knows that you need to expect the unexpected and at the end of the day prepare for lots of debugging. No amount of migration guide would prepare you for what’s coming in the field. In the words of Donald Rumsfeld there are unknown unknowns and we need to be equipped with the tools to uncover these unknowns and patch them up. In this blog post I’d like to walk you through a process that should show where the application hangs, although there seems to be nothing wrong with it. I will also show that you don’t always know what code you have – problem known as dependecy hell, place we got quite cosy in during this upgrade.
Label noise is ever-present in machine learning practice. Allegro datasets are no exception. We compared 7 methods for training classifiers robust to label noise. All of them improved the model’s performance on noisy datasets. Some of the methods decreased the model’s performance in the absence of label noise.
Hermes is a distributed publish-subscribe message broker that we use at Allegro to facilitate asynchronous communication between our microservices. As our usage of Hermes has grown over time, we faced a challenge in effectively distributing the load it handles to optimize resource utilization. In this blog post, we will present the implementation of a dynamic workload balancing algorithm that we developed to address this challenge. We will describe the approach we took, the lessons we learned along the way, and the results we achieved.
Many of us, software engineers, have experienced those days when nothing really sparks joy in coding, debugging, preparing spikes or refining tasks for the next sprints. Obviously, we would like to have as few of such days as possible and go on with our work effectively. A solution to this definitely is not tormenting our brains with guilt and forced labour. There are other ways, and I would like to invite you to explore them with me and learn a little about our nervous systems in the process. We’ll find out where the motivation comes from on a biological and psychological level. We’ll also take a look at the changes you can introduce into your day to take advantage of certain mechanisms working on a neural level and boost your motivation and productivity.
Software Architecture is an elusive thing which, if neglected, can lead to a hard-to-develop and maintain codebase, and in more drastic circumstances to the failure of a product. This article discusses one of the backend application architecture styles which proved to be successful in providing a good foundation for building and maintaining an application in the long run: Onion Architecture.
Sometimes great results in code performance come with a small amount of work. We’d like to tell you a story about how we changed the Allegro mobile homepage and reduced usage of Allegro service infrastructure with only a few lines of code.
Let’s look at what transactions in MongoDB are and how they differ from SQL transactions.
Building a complex web platform can be a real challenge, especially when parts of it are delivered by independent teams. Picking out the correct architecture is crucial, but maintaining it can be even more challenging. Frontend microservices, aka microfrontends, is an architecture that gives a lot of flexibility, but can cause performance issues in the future, if not managed well. This article presents an approach to the microfrontends architecture to keep the frontend technology stack efficient based on the complexity of user interface.
What would you say if we stored 1 000 records in a database, and the database claimed that there were only 998 of them? Or, if we created a database storing sets of values and in some cases the database would claim that some element was in that set, while in fact it was not? It definitely must be a bug, right? It turns out such behavior is not necessarily an error, as long as we use a database that implements probabilistic algorithms and data structures. In this post we will learn about two probability-based techniques, perform some experiments and consider when it is worth using a database that lies to us a bit.
Currently, in the Android world, the topic of modularization is very popular. Many bloggers describe their experiences with it and analyze what Google recommends. Our team started the modularization process before it was hot. I will describe our reasons, decisions, problems and give you some advice. We will see if modularization makes sense and what it brings to the table. I will also post some statistics showing what it looked like before and after the modularization process.
Cloud providers like to brag about high availability and unlimited scaling of their services – and they are correct, as these features are indeed significant advantages of cloud solutions. Their computational power is so high that for most use cases, it’s almost unlimited. In this blog post, I would like to tell you about our experiences with Azure Cosmos DB and batch processing.
In this article, we want to share our approach to using server-driven UI in native mobile apps. In 2019 we created the first version of the in-house server-driven rendering tool called MBox and used it to render the homepage in the Allegro app on Android and iOS. We have come a long way since then, and now we use this tool to render more and more screens in the Allegro apps. After almost three years of working on MBox, we want to share how it works and the key advantages and challenges of using this approach.
With this article, I would like to introduce you to EventStorming and explain to you how to get started. I am not discovering anything new, just gathering available knowledge in one place. What I will show you is a few tips on how to conduct and facilitate EventStorming workshops.
Each of us has probably experienced a time in our career when we wanted to get rid of the Garbage Collector from our application because it was running too long, too often, and perhaps even led to temporary system freezes. What if we could still benefit from the GC, but in special cases, also be able to store data beyond its control? We could still take advantage of its convenience and, at the same time, be able to easily get rid of long GC pauses. It turns out that it is possible. In this article, we will look at whether and when it is worth storing data beyond the reach of the Garbage Collector’s greedy hands.
At Allegro we decided to introduce GraphQL as our API Gateway for building several internal client systems. By building such a solution we’ve learnt a lot about this technology and we would like to share it with you in this article.
Nowadays, technical debt can be considered the bread and butter of most IT-powered enterprises around the world. Almost every company that survived the startup phase and managed to deliver its first products to customers will face at some point technical challenges related to past architectural decisions. Although code engineering gets better every year, we cannot argue with the obvious fact of life: the market will always force many of us to deliver tech products faster than we wish. Time To Market has always been a key success factor for many product companies and it puts a lot of pressure on Engineering to keep up with challenging deadlines.
This blogpost is the result of a research collaboration between the Allegro Machine Learning Research team and the Institute of Mathematics of the Polish Academy of Sciences (IMPAN), Warsaw.
A picture is worth a thousand words — see for yourself how this unique Agile team learns BASIC from their Product Owner.
We use written (source code) language to express our intentions in a machine-readable form. We use spoken language to communicate with other people. We pride ourselves as ones choosing a programming language optimized to the task at hand. Do we use the optimal way to express our ideas?
Will automatic UI tests be able to replace manual testers as artificial intelligence will try to replace programmers? I’ll show you how we write automatic UI tests on Android in Allegro Pay.
The attack on Ukraine has shocked the entire world, but in Poland it is making especially big waves. Not just due to geographical proximity - Ukrainians are one of the largest minorities in Poland and simply put it is affecting people we know personally.
Have you ever worked on a project where after downloading the code from the repository you start to wonder what business requirements are hidden under the layer of unreadable tests? Or maybe you are currently wondering how to test a new feature that you have been entrusted to implement?
Everyone repeats like a mantra that tests are an indispensable element of development work. Is there anything to it? Well, I need to admit that as a developer, I rather often want to skip the test writing stage. I assume I’m not the only one. I’m aware that it’s a mistake, even with testers on board. Effective and efficient testing of your own code can help with catching bugs in new functionalities, as well as in changes to already existing ones. It cannot be questioned. Sometimes tests also help to understand how some long-unused functionality or component works. And that’s a small bonus too. Can a large project cope without testing? Probably so. But the number of errors and their severity will probably be much higher. That’s why in Allegro Ads we pay attention to writing tests.
Most of applications we create are basically loops. An average program waits for an event, then processes it following some business logic. Afterwards it begins waiting for another event to arrive. Java Servlets work this way too. Popular frameworks such as Spring allow us to only care about the business logic, while the framework takes care of the application main loop.
The source code of Allegro iOS app for buyers used to be divided into separate modules hosted in multiple repositories (polyrepo). The source code was migrated to a monorepo a few years back along with the history of all repos that constituted the app. Updating source code of a module on one repository could affect another module hosted on a separate repository. Versioning modules and propagation of dependency update led to long release process of the entire application. Our main repository for the iOS application thus became our monorepo. After 9 years of development of the app the repo size has grown enormously and the git clone command became a nightmare taking too much time. We had a possibility to shrink the project size during the migration from an on-premise to an external git repo hosting provider.
A new version of MongoDB, 5.0, has been recently launched. The list of changes included one that I found particularly interesting: the time series collections. It is a method of effective storing and processing of time-ordered value series. In this article we will verify whether the processing of time series is really as fast as promised by the authors.
The Clean Architecture concept has been around for some time and keeps surfacing in one place or another, yet it is not widely adopted. In this post I would like to introduce this topic in a less conventional way: starting with customer’s needs and going through various stages to present a solution that is clean enough to satisfy concepts from the aforementioned blog (or the book with the same name).
At some point in your career, you realize that it’s time to try to advance through the hierarchy. You think you are doing a good job. You are constantly developing and learning something new. But at the same time, someone you know, with much less experience and knowledge than you, has long been higher up the hierarchy than you. Then you ask yourself: what is wrong with me? In my case, the answer turned out to be properly gathering the expectations concerning my skills and work.
Micronaut is one of the new application frameworks that have recently sprung up. It promises low memory usage and faster application startup. At Allegro we decided to give it a try. In this article we’ll learn what came out of it and if it’s worth considering when creating microservices-based systems.
Every e-commerce platform needs some kind of central authorization system. At Allegro we use OAuth and have our own implementation that stays true to the RFC. Allegro has millions of users. There are also a lot of requests that go through OAuth services. At some point there comes a need to have better control over how much traffic we want to allow in a certain time window, while maintaining full performance of the platform. Here is where the idea of rate-limiting comes in handy.
This article describes a classic case of refactoring a search form UI component, a critical part of every e-commerce platform. In it I’ll explain the precursor of change, analysis process, as well as aspects to pay attention to and principles to apply while designing a new solution. If you are planning to conduct refactoring of a codebase or just curious to learn more about frontend internals at Allegro, you might learn a thing or two from this article. Sounds interesting? Hop on!
One of the key elements ensuring efficient operation of the services we work on every day at Allegro is fast responses from the database. We spend a lot of time to properly model the data so that storing and querying take as little time as possible. You can read more about why good schema design is important in one of my earlier posts. It’s also equally important to make sure that all queries are covered with indexes of the correct type whenever possible. Indexes are used to quickly search the database and under certain conditions even allow results to be returned directly from the index, without the need to access the data itself. However, indexes are not all the same and it’s important to learn more about their different types in order to make the right choices later on.
It’s easy to find resources about improving Elasticsearch performance, but what if you wanted to reduce it? In Part I of this two-part series we looked under the hood in order to learn how ES works internally. Now, in Part II, is the time to apply this knowledge in practice and ruin our ES performance. Most tips should also be applicable to Solr, raw Lucene, or, for that matter, to any other full-text search engine as well.
It’s easy to find resources about improving Elasticsearch performance, but what if you wanted to reduce it? This is Part I of a two-post series, and will present some ES internals. In Part II we’ll deduce from them a collection of select tips which can help you ruin your ES performance in no time. Most should also be applicable to Solr, raw Lucene, or, for that matter, to any other full-text search engine as well.
The main goal of boosting website performance is to improve the user experience. In theory, a satisfied customer is more likely to use a particular company’s services, which is then reflected in business results. However, from my own experience I can say that not every change can be easily converted into money. I would like to tell you how to reconcile these two worlds, how to convince the business that the benefits of better performance are a long-term investment, and how to streamline the development process during the design or code writing process.
The following article is an excerpt from Software Mistakes and Trade-offs book. In real-world big data applications, the amount of data that we need to store and process can be often counted in the hundreds of terabytes or petabytes. It is not feasible to store such an amount of data on one physical node. We need a way to split that data into N data nodes.
It’s been over 5 years since the introduction of the article describing the ongoing transformation of Allegro’s frontend architecture — an approach that was later formalized by the industry under the name of Micro Frontends. I think that after all this time we can safely say that this direction was correct and remained almost entirely unchanged in relation to the original idea. Still, some of the challenges foreseen in the publication soon became the reality. In this article I would like to focus on the CSS part of the whole adventure to tell you about how we manage consistency and frontend performance across over half a thousand components, and what it took us to get to where we stand today.
Some time ago, our team faced the issue of moving an existing Apache Spark job from an on-premise Hadoop cluster to public cloud. While working on the transition we came across another way to process data that is Apache Beam. We were curious whether this tool had more advantages in comparison to traditional Apache Spark. We wanted to find the answer relatively quickly with minimal effort. Hence, we built two projects to process the same data using these technologies. Below you can get to know the architecture of the jobs written in Apache Spark and Apache Beam.
I’m a big fan of logging as much extra data as possible. I’m also a DRY approach believer. I feel strong anxiety when I see repetition in code. Combining all these “passions” is not always easy. It’s hard to log everything without repeating things. Even if it’s possible, it usually leads to inelegant code.
Some time ago we announced that Allegro passes Core Web Vitals assessment and thanks to that we were awarded in “Core Web Vitals Hall of Fame”. It means that Allegro is in the group of the 27% fastest websites in Polish Internet.
Accurate forecasting is key for any successful business. It allows one to set realistic financial goals for the next quarters, evaluate impact of business decisions, and prepare adequate resources for what is coming.
The beginnings in a new job can be really tough, especially in such uncertain times as the pandemic. Remote onboarding, Zoom meetings, inability to talk face-to-face — it’s a big test, especially if you’re switching industries. My story is quite similar: in July 2020 I started a three-month internship at Allegro to train for product management. When the internship ended, I was offered to stay permanently as Junior Product Manager. I’d like to describe my beginnings at Allegro. But first — a few words about me.
According to Wikipedia there are approximately 700 computer languages available. Seven hundred. This is an unbelievable number and it’s the reason why programmers face the problem of choosing a programming language to work with, which frameworks to use and which tech stack to learn. All of them have pros and cons, but when looking for an all-purpose language you should take Kotlin into consideration and ask yourself the question: “Do I really need another programming language?”.
Maybe it’s another “How I’ve changed my worklife” story, but I hope it is an interesting one. It seems quite important to tell: before the beginning of my Allegro adventure, I was working as… an archaeologist. It was both physical work on excavations and research work at my PhD studies. And the only thing that connects my past work and the current one is: attention to detail.
Coordinating complex processes, both business and technical, can be a challenging issue in a distributed system. Especially when the complications associated with them, such as concurrency, idempotency, scalability and hindered testability, come into play — possibly all at once. This is definitely something that can keep many programmers awake at night.
So you’re new to Allegro, have just finished your tech onboarding and are stunned with information overflow? Or perhaps you are planning to join Allegro and don’t know what it looks like in here? I am about to try and describe how I felt just a few months ago and what startled or dismayed me. I hope this short article will answer all your concerns.
Many of us spend most of their software development careers improving and extending applications protected by pre-existing security mechanisms. That’s why we rarely address problems related directly to authentication and authorization unless we build apps from scratch. Regardless of your experience I still hope you will find this article interesting. It’s not meant to be a tutorial. I would like to focus on clarifying basic concepts and highlighting common misconceptions.
Surprisingly, this is the first time we share with you a report from Allegro Tech Meeting (ATM) — even though it’s already the 12th edition of this special event.
In this last instalment of our series about team tourism at Allegro, our two engineers describe their eventful visits in two rather technical teams, one dealing with our message broker - Hermes, the other with web performance.
Following up on our previous post about team tourism at Allegro, as promised we present you with three case studies describing experiences of our engineers during their visits in other teams. The teams visited worked on content automation, machine learning and search engine optimisation.
Are you dreaming about getting a job in IT, but you feel stuck? You may be constantly working your way through books and tutorials about the technology that excites you (be it JavaScript, PHP or whatever else), but it still feels like you will never be able to get that Junior Developer position you want.
We often hear about the importance of exchanging knowledge and practices between different teams. However, less often do we hear concrete suggestions for how to do it. In this article, we discuss one of our practices to address the problem.
Developing our own A/B testing platform from scratch came with many lessons during the process. Our platform was built for people who differ vastly in terms of statistical knowledge, also for those who are not familiar with statistics at all. So the main challenge wasn’t technical implementation. The hardest thing was (and still is) developing good practices and spreading them among our users. We want to share some of our current knowledge that we think is crucial for A/B testing at scale. From this article, you will get to know about calculating sample size, what it means and what benefits it provides for you and your organization.
Application release process, or in fact software development process, as a release is the final stage of application development, is not an easy thing. Books and IT websites discuss many approaches and each has its supporters and opponents. On the one hand, you have product owners, project managers and customers who want a ready-to-use application as soon as possible. On the other hand, we developers and testers, would like to release an application of the highest quality, which may affect the delivery time. Balancing these needs is a hard nut to crack. Usually, both sides need to make some compromises to establish a common way of working. For developers and testers, it involves answering several questions concerning software development methods, skills, use of manual or automated testing, and storage of test cases and test logs. In this article I describe best practices and tips for starting a new project. I think that by following them, you will make the software development process as effective as possible and adjusted to conditions of your project.
If you always search for ways to increase code quality and would like to encourage everyone in your project to keep high test code coverage all the time, then Codecov may be another step in your journey.
Moving Allegro’s services and infrastructure from its main Data Center (DC) was considered a huge endeavour fraught with high risk from the very start. This particular Data Center, known as DC2, was situated in Poznań, Poland. It was our first genuine Data Center and back in 2007, when we moved in there, it was the state—of—the—art DC in Poland. For a couple of years our infrastructure was growing along with the company and the number of devices grew up to 2,000 filling 90 rack cabinets in 2014. DC was used by Allegro core sites such as allegro.pl, aukro.ua and aukro.cz as well as other Naspers-owned Polish operations like: payu.pl, olx.pl, ceneo.pl, otomoto.pl, otodom.pl. To make the story simpler, Allegro synonym is used to name them all.
Microservices architecture has been given a lot of attention in recent years. You can find many articles defining what it is and explaining the difference between microservices and monolithic applications. Unfortunately, the dark side of the solution is rarely mentioned — such as its level of complexity and how difficult it is to design it correctly. Nevertheless, I would like to talk about my experience with microservices architecture. I hope that this article will help you avoid the mistakes I made and save time.
