Back
Automatically translated version. May contain inaccuracies compared to the original.
IT engineer Nikita Kuznetsov on the invisible infrastructure that the stability of a digital product depends on.
In development it is customary to discuss what can be shown: a new interface, the speed of the application, a new feature, an algorithm, a model of artificial intelligence. Much less often they talk about DNS, certificates, service accounts, timeouts, retry attempts, and logs. For the user, these things do not exist at all—until one of them breaks.
IT engineer Nikita Kuznetsov considers this invisible part of the product to be one of the most underappreciated areas of modern development. You can write high-quality code, run tests, configure servers, and months on end service users without problems. And then one old DNS record, an expired certificate, or a forgotten technical account can erase everything else. The application continues to run, servers are healthy, the database responds, but the user can no longer get in.
Kuznetsov calls such elements the “critical boredom”: things nobody particularly wants to deal with because, in a normal state, they do not produce noticeable results.
“People fix the storefront and forget that the building stands on pipes. While water flows, no one cares about the pipes. Then, on a Saturday evening, the pipe bursts, and it turns out the storefront was nothing to do with it.”
This metaphor fairly accurately describes the difference between how the product looks externally and how it exists technically. The user sees a site or an app. The engineer sees dozens of dependencies through which a single ordinary request must pass successfully. And as the system becomes more complex, the reason for a major outage more often lies not in its most visible part.
Nikita Kuznetsov on why functional code does not yet mean a functioning product.
For the user, the product exists as a single whole. He does not know and is not obliged to know that behind a single button there may be an API gateway, several microservices, a database, a message queue, a third-party payment provider, and a separate authentication system. If he presses the button and nothing happens, the product does not work for him.
Engineers see the situation differently. Each component can be checked separately and yield a fairly encouraging picture: the app is running, the server responds, the processor is not overloaded, the database is available, a new version has not been deployed yet. But the entire chain can still be nonfunctional.
This is one of the unpleasant features of distributed systems: local correctness of components does not guarantee the correctness of a user scenario.
According to Nikita Kuznetsov, it is useful here to abandon the habit of treating infrastructure as a secondary technical layer:
“Critical boredom works like a foundation: you cannot see it in a picture, but everything you later call the product stands on it.”
A new feature almost always has an obvious owner. There is a task, a performer, a deadline, testing, and a release. With infrastructural dependencies things are often much less defined. Who is responsible for the domain? Who controls certificates? Who knows all the service accounts? Who checks whether the contractor who finished work six months ago still has rights?
While everything works, the lack of answers to these questions is almost imperceptible. A failure immediately makes them central.
DNS can disable a product without changing a single line of code.
Nikita Kuznetsov explains why DNS and other external dependencies can make a reliably working service unavailable.
One of the phrases Nikita Kuznetsov considers particularly characteristic of emergency situations is this: “But we didn’t deploy anything.”
And this can indeed be true.
The app has not changed. Servers are running. The database responds. The last release was a few days ago. Inside the infrastructure, the service may even open via a direct address. But a ordinary user types the familiar domain name and does not get a working site.
“DNS is a house number on a building that no one reads until they lose entry. Then the whole company stands in the entrance and argues about who is to blame in the code.”
Technically, DNS links a human-understandable domain name to information needed to locate the desired resource. The chain involves recursive resolvers, caching, and authoritative DNS servers. Therefore, an incorrect record or a failed update may behave not as obviously as a typical application error. Different users may receive different results for a period due to caches. The TTL parameter determines how long a DNS record can be cached: the larger it is, the longer the old value may continue to be used after a change.
From this arises an unpleasant scenario. For one engineer the site already opens, for another — not yet. From one region the request goes correctly, from another region it continues to use the cached value. The team sees conflicting symptoms and begins to look for instability in the application, even though the problem lies one level up.
Nikita Kuznetsov therefore speaks of “out-of-repo outages.” Something that did not change along with the source code can break: a DNS record, a network rule, a certificate, a secret, an external account, or a provider configuration.
Moreover, such settings often outlast several generations of the product itself. The app has long been rewritten, the infrastructure migrated, the staff changed, but some critical record still exists only because five years ago it was created once by hand.
The main problem here is not so much a particular technology. It arises when a critical object has no owner, change history, or a clear way to verify.
Expired certificate — a failure whose date could have been known in advance.
Kuznetsov cites certificates as a particularly illustrative example. Unlike many software bugs, a certificate's validity period is not a surprise. The date is known beforehand.
“A certificate does not break the system abruptly. It breaks it precisely on the day that everyone saw and no one put on the calendar.”
Today, a large portion of TLS certificate work can be automated. Automatic renewal systems can renew certificates before expiration, and separate monitoring can proactively alert if this process fails. For short-term certificates this is especially important: automation alone should not be the sole point of trust; its operation also needs to be monitored.
Therefore an outage due to an expired certificate is not interesting for its technical complexity. On the contrary — it reveals an organizational problem.
Someone was sure that renewal was automated. Notifications were sent to an old address. The automatic renewal system once stopped working, but there was no separate monitoring of its status. The responsible employee changed positions, and the duty formally was not assigned to anyone.
As a result, the browser begins to warn the user about a problem with a secure connection, some clients refuse to continue, support receives inquiries, and the technical team suddenly deals with an object whose existence almost no one had previously mentioned.
Kuznetsov notes an important detail: for the user there is no excuse of “the service works, it's just the certificate.” If a person cannot safely open the product, then the product does not work for them.
An infrastructure nuisance becomes a business problem.
Temporary access is dangerous precisely because it stops seeming temporary.
An even more serious situation arises with access rights. Here the consequences can go well beyond ordinary downtime.
“The scariest account isn’t a hacker’s. It’s the one created “for a few days” and forgotten to turn off.”
Nikita Kuznetsov describes a common lifecycle of such access. The team urgently needs to connect a new service or allow a contractor to set something up. Time is short, so permissions are made broader than needed. After the launch they plan to limit them.
The release goes through. New tasks appear. A few weeks later the temporary solution is no longer mentioned. After a year the technical key still exists.
The problem becomes especially serious when it is not possible to quickly determine who owns the account, what process uses it, and what happens after it is disabled. The team sees potentially dangerous access but fears deleting it: suddenly at night some old process will stop working.
A paradoxical situation arises. Unnecessary access is kept precisely because the system is not well documented.
The same applies to people.
“An indispensable person with full access is not a team achievement. It is a hole called competence.”
An experienced engineer can indeed know the infrastructure better than others. The problem begins when the knowledge and rights of this person become the only way to support a critical process. Then a vacation, illness, dismissal, or simply unavailability of the employee becomes a technical risk.
Therefore Kuznetsov suggests looking at access management not only as a matter of information security. It is also a matter of business resilience. The company should understand who has access to critical systems, why they need it, when it was last used, and whether there is an alternative way to restore the system without one specific specialist.
Why microservices break not only inside services.
Nikita Kuznetsov on the links between services, timeouts, and retries that can turn a local failure into a system outage.
Architectural diagrams often look very neat: a rectangle for users, a rectangle for payments, authentication, catalog, notifications, analytics. Each service is isolated from its neighbor and has a clear function.
Nikita Kuznetsov suggests looking primarily at the connections between them.
“Everyone draws squares. Almost no one draws roads. And the road collapses.”
The point here is the behavior of services during problems in neighbors. How long will one service wait for another’s response? What happens after a timeout? Will the request be retried? How many times? Over what interval? Is it safe to repeat this operation at all?
Suppose that the payment service usually responds in 200 milliseconds, but due to load began to respond in several seconds. The client service treats this as an error and sends the request again. Then again. The same thing happens simultaneously with thousands of other requests.
As a result, the system itself increases pressure on the component that already has problems.
This is a real failure class known as a retry storm. Uncontrolled retries under high load can further overwhelm the network and the service, thereby reducing availability. Standard protection methods include limiting retries, exponential backoff, random jitter, and a circuit breaker that temporarily stops calls to a clearly problematic dependency.
Retries are dangerous for another reason. Not every operation can be done again blindly. If the first request effectively went through but the client did not receive a confirmation, retrying can lead to duplicate actions—for example, creating a second operation. That is why when designing retry mechanisms one must consider idempotency.
For this reason the phrase “if there’s an error—we’ll try again” for Kuznetsov is not a complete reliability strategy. The engineer must understand which errors are truly temporary, how many retries are permissible, and what happens if thousands of clients start retrying simultaneously.
Thus a small problem in one service either remains local or becomes the outage of the entire system.
Nikita Kuznetsov on logs that exist but cannot be investigated.
Nikita Kuznetsov on observability: logs and telemetry should help quickly reconstruct the chain of events during a failure.
The next problem arises after a failure. A company may store a huge amount of logs and still know almost nothing about what happened.
“If after an outage you spend half an hour looking for the needed line, you do not have observability. You have a diary that no one read.”
A typical log records that at a certain time a particular component logged a specific event. But in a distributed system this is not enough. A single user request may pass through an API gateway, authentication, several internal services, and a database. If each of them maintains a separate log without a shared link, engineers must manually assemble the history by time and indirect cues.
Modern observability is therefore built not only around logs. Usually several types of telemetry are used — traces, metrics, and logs. A distributed trace allows tracking a path of one request through different components, and individual spans show specific stages of that path. This makes it possible to see not just an error message, but the place and moment where a user operation began to break.
Particularly useful is the correlation of logs with a trace ID and span ID: entries from different services can be linked to a single operation. Without this context, logs from distributed components often remain a disjointed set of events.
For Nikita Kuznetsov, this is a fundamental difference between “we collect data” and “we understand the system.”
Terabytes of logs by themselves do not guarantee anything. During a failure, the engineer needs answers to concrete questions: when did the problem start, which user scenario was affected first, through which services did the request pass, where did the initial delay appear, what errors followed it, and how widely did the outage spread.
If answers require several hours of manual correlation, observability exists formally but does not perform its main function.
The system must alert about the problem earlier than the client.
From this, Nikita Kuznetsov derives another maturity criterion for a product: who is the first to know about the outage.
If the technical team receives the first report from the client, the system has already lost time. The user should not have to perform the role of free monitoring.
But good monitoring is not just checking whether the server answers to a request. A service can be technically available and still not fulfill its function. Reliability must be evaluated by real user outcomes: a system may have almost continuous uptime, but remain unreliable if a key user action regularly ends incorrectly.
Therefore Kuznetsov proposes looking at the product on several levels at once. You need to know not only CPU load and memory usage, but also the percentage of errors, response latency, queue status, times of key user operations, the operation of external dependencies, and anomalous changes in normal system behavior.
And here again appears the “critical boredom.” Monitoring rarely brings new user functionality. A good alert should ideally never be shown to the client. But it is precisely what can give the team ten or twenty minutes that separate a small degradation from a full outage.
Why do companies keep postponing such tasks.
The reason, according to Kuznetsov, is quite prosaic: the result of preventive engineering work is poorly visible.
A new feature exists physically — it can be opened and shown. A new design can be put into a presentation. Speed increases can be expressed by a number.
And the result of timely renewed certificates is that nothing happened. DNS did not break. The old key was removed, and no one noticed. The problematic service was automatically restricted before it dragged others. Monitoring raised the alarm earlier than the clients.
“N dull things do not deliver demos. They are not shown in meetings. So they live without an owner until they arrange a meeting themselves.”
This phrasing well describes the conflict between the visibility of work and its real value. The better preventive infrastructure works, the fewer events it creates for the business. Therefore it is easy to treat it as a secondary cost item—until the first major outage.
After an outage, the situation usually reverses. Resources for monitoring, access auditing, documentation, and redundancy suddenly appear. But Kuznetsov considers the more mature opposite approach: not to wait until the infrastructure proves its importance through downtime.
Nikita Kuznetsov on four questions for every critical dependency.
Instead of a huge list of formal rules, Kuznetsov offers a practical approach. For each truly critical dependency the team should be able to answer four questions: who is responsible, when do we check, how do we learn about the problem, and what do we do after the signal.
The certificate must have an owner and automatic expiry control. In DNS — a clear process for changes and the ability to verify where the domain actually resolves. For a technical account — purpose, minimum necessary rights, and a review procedure. For inter-service calls — timeouts, a limited retry policy, and a predictable behavior during failure. For a user request — sufficient telemetry to trace its path through the system.
The most important thing here is to remove memory dependence.
The memory of an experienced employee is useful, but it is not a failure-resilience mechanism. The phrase “Sasha usually monitors certificates” is not a process. “We know who to call” is not a recovery plan. “This key still seems needed” is not a policy of access.
“If critical boredom has no owner, no expiry check, and no signal to a failure, it is not infrastructure. It is hope.”
This phrase effectively encapsulates the entire position of Nikita Kuznetsov. Product reliability is determined not only by code quality or the number of technologies in the architecture. It is also determined by how well the team controls simple, old, and completely inefficient dependencies without which the entire complex product stops existing for the user.
You can build dozens of microservices, use modern AI models, and process huge volumes of data. But if no one knows when a critical certificate expires, who owns the old administrative key, or why a service repeats a request to an already overloaded dependency five times, technological complexity does little to help.
Therefore Kuznetsov suggests looking for future outages not only in the most complex parts of the system. Sometimes it is more useful to open the list of DNS records, certificates, and service accounts, look at the rules by which components interact, and try to reproduce one user request from logs.
And pose a few awkward questions to each such element: who is responsible for it, when was it last checked, how will the system report a problem, and what happens if it stops working right now?
If there are concrete answers to these questions, the “boring” infrastructure truly becomes part of engineering. If there are no answers, the product can look reliable for years simply because it has lucked so far.
Document: PDF proof of the original version of the news item "Никита Кузнецов про невидиму інженерію, через яку хороший продукт одного дня може просто не відкритися". It records the publication content at the moment of the first scan, the preservation date and the source: Zavtra.UA - About What Will Happen Tomorrow.
