This article is a case study of how we improved stability in our critical application. It’s mostly a technical analysis of what happens in fresh Java based instance, how JIT Compiler toyed with us at application start and how we learned to control it.
Site performance is very important, first of all, from the perspective of users, who expect a good experience when visiting the site. The user should not wait too long for the page to load. We all know how annoying it can be when we want to press an element and it jumps to another place on the page or when we click on a button and then nothing happens for a very long time. The state of a site’s performance in these aspects is measured by Web Vitals performance metrics and most importantly by a set of three major Core Web Vitals metrics (LCP — Largest Contentful Paint, CLS — Cumulative Layout Shift, INP — Interaction to Next Paint). They are responsible for measuring the 3 things: loading time, visual stability and interactivity. These metrics are also important for the websites themselves because, in addition to the user experience, they are also taken into account in terms of the website’s positioning in search engines (SEO), which is crucial for most websites on the Internet, Allegro included.
This story shows our journey in addressing a platform stability issue related to autoscaling, which, paradoxically, added some additional overhead instead of reducing the load. A pivotal part of this narrative is how we used Couchbase — a distributed NoSQL database. If you find yourself intrigued by another enigmatic story involving Couchbase, don’t miss my blog post on tuning expired doc settings.
At Allegro, we use Kafka as a backbone for asynchronous communication between microservices. With up to 300k messages published and 1M messages consumed every second, it is a key part of our infrastructure. A few months ago, in our main Kafka cluster, we noticed the following discrepancy: while median response times for produce requests were in single-digit milliseconds, the tail latency was much worse. Namely, the p99 latency was up to 1 second, and the p999 latency was up to 3 seconds. This was unacceptable for a new project that we were about to start, so we decided to look into this issue. In this blog post, we would like to describe our journey — how we used Kafka protocol sniffing and eBPF to identify and remove the performance bottleneck.
Sometimes great results in code performance come with a small amount of work. We’d like to tell you a story about how we changed the Allegro mobile homepage and reduced usage of Allegro service infrastructure with only a few lines of code.
What would you say if we stored 1 000 records in a database, and the database claimed that there were only 998 of them? Or, if we created a database storing sets of values and in some cases the database would claim that some element was in that set, while in fact it was not? It definitely must be a bug, right? It turns out such behavior is not necessarily an error, as long as we use a database that implements probabilistic algorithms and data structures. In this post we will learn about two probability-based techniques, perform some experiments and consider when it is worth using a database that lies to us a bit.
Each of us has probably experienced a time in our career when we wanted to get rid of the Garbage Collector from our application because it was running too long, too often, and perhaps even led to temporary system freezes. What if we could still benefit from the GC, but in special cases, also be able to store data beyond its control? We could still take advantage of its convenience and, at the same time, be able to easily get rid of long GC pauses. It turns out that it is possible. In this article, we will look at whether and when it is worth storing data beyond the reach of the Garbage Collector’s greedy hands.
At Allegro we decided to introduce GraphQL as our API Gateway for building several internal client systems. By building such a solution we’ve learnt a lot about this technology and we would like to share it with you in this article.
A new version of MongoDB, 5.0, has been recently launched. The list of changes included one that I found particularly interesting: the time series collections. It is a method of effective storing and processing of time-ordered value series. In this article we will verify whether the processing of time series is really as fast as promised by the authors.
Micronaut is one of the new application frameworks that have recently sprung up. It promises low memory usage and faster application startup. At Allegro we decided to give it a try. In this article we’ll learn what came out of it and if it’s worth considering when creating microservices-based systems.
One of the key elements ensuring efficient operation of the services we work on every day at Allegro is fast responses from the database. We spend a lot of time to properly model the data so that storing and querying take as little time as possible. You can read more about why good schema design is important in one of my earlier posts. It’s also equally important to make sure that all queries are covered with indexes of the correct type whenever possible. Indexes are used to quickly search the database and under certain conditions even allow results to be returned directly from the index, without the need to access the data itself. However, indexes are not all the same and it’s important to learn more about their different types in order to make the right choices later on.
It’s easy to find resources about improving Elasticsearch performance, but what if you wanted to reduce it? In Part I of this two-part series we looked under the hood in order to learn how ES works internally. Now, in Part II, is the time to apply this knowledge in practice and ruin our ES performance. Most tips should also be applicable to Solr, raw Lucene, or, for that matter, to any other full-text search engine as well.
It’s easy to find resources about improving Elasticsearch performance, but what if you wanted to reduce it? This is Part I of a two-post series, and will present some ES internals. In Part II we’ll deduce from them a collection of select tips which can help you ruin your ES performance in no time. Most should also be applicable to Solr, raw Lucene, or, for that matter, to any other full-text search engine as well.
The main goal of boosting website performance is to improve the user experience. In theory, a satisfied customer is more likely to use a particular company’s services, which is then reflected in business results. However, from my own experience I can say that not every change can be easily converted into money. I would like to tell you how to reconcile these two worlds, how to convince the business that the benefits of better performance are a long-term investment, and how to streamline the development process during the design or code writing process.
The following article is an excerpt from Software Mistakes and Trade-offs book. In real-world big data applications, the amount of data that we need to store and process can be often counted in the hundreds of terabytes or petabytes. It is not feasible to store such an amount of data on one physical node. We need a way to split that data into N data nodes.
It’s been over 5 years since the introduction of the article describing the ongoing transformation of Allegro’s frontend architecture — an approach that was later formalized by the industry under the name of Micro Frontends. I think that after all this time we can safely say that this direction was correct and remained almost entirely unchanged in relation to the original idea. Still, some of the challenges foreseen in the publication soon became the reality. In this article I would like to focus on the CSS part of the whole adventure to tell you about how we manage consistency and frontend performance across over half a thousand components, and what it took us to get to where we stand today.
Some time ago we announced that Allegro passes Core Web Vitals assessment and thanks to that we were awarded in “Core Web Vitals Hall of Fame”. It means that Allegro is in the group of the 27% fastest websites in Polish Internet.
Let me start with a story. Once upon a time I stumbled upon an excellent article by Philip Walton where he describes how expensive script evaluations could (and should!) be deferred until the browser is idle or they are actually needed. One of the examples that awakened my interest was creating an instance of the Intl.DateTimeFormat object, as I was using this great API quite often but never thought it can cause real performance problems. Turns out it can, especially if used inside loops. Apart from the technique described in Philip’s article, another solution is to simply reuse Intl.DateTimeFormat instances instead of creating them every time.
As a developer interested in both web technologies and game development I always found myself disagreeing with a large part of articles about using a particular technology to solve some problems. While such articles are often true, they often skip some important details that make given solution unacceptable in some other cases. And in this article I will try to look at immutability in a negative way from game development perspective and how it can affect web services too. It is always more fun to look in a negative way at something everyone loves ;)
One of the coolest features added in just announced TypeScript 3.7 is optional chaining syntax. It promises a much shorter and more readable code for dealing with deeply nested data structures. How may this nice new feature affect the performance of your project?
When we measure the page loading speed from the user’s perspective, we pay attention to the appearance of subsequent elements on the screen. Metrics such as First Contentful Paint, First Meaningful Paint and Visually Complete directly reflect what the user sees and when. But what if the page is invisible, when it loads in the background, for example in a different tab? Should we consider such views interesting for us? Don’t the collected metrics distort the results?
Lately my colleague Michał described how he tracked a Java memory leak. Although that problem was completely solved, new complications suddenly appeared on the horizon. As it usually happens, everything started with an alert from our monitoring system which woke me up in the middle of the night.
Kotlin may seem like a new kid on the block — itʼs been officially released only in February. Its history however dates a few years back and itʼs mature and stable enough to be used for developing solid reliable applications. Therefore at Allegro we decided to give it a chance — we built our new shiny server-side system using Kotlin as its primary language and we do not regret it.
You might have read a recent post by our developers concerning performance analysis tools and its follow up concerning sysdig. In the database world these tools come handy almost everyday. In this blog post I will show you a case where I have put tools to action diagnosing a MongoDB issue.
This post is an introduction to sysdig — an “open source, system-level exploration” tool that ease the task of performance troubleshooting in Linux operating system.
