Expertise · Complex Architectures
We design and operate platforms made up of multiple services, with a message bus, capped retries, idempotency keys, circuit breakers and publication with automatic rollback. Each pattern below runs in its own system, not in a diagram.
Already builtWe operate four proprietary platforms with different architectures and we can open all of them. The figures are measured today with commands, not taken from documentation: aichat has 28 service directories, 235 data models across four schemas and a message bus with 297 queues and 17,525 links; Kallina has 472 edge functions, 559 migrations and 28 scheduled tasks in the database; MEGA CRM has a 9,516-line backend with 99 routes and 179 row-level access policies; Taskin runs 21 scheduled tasks without any Docker. The reservation we state: in two places our own documentation lagged behind the code — a statement about the number of lines was outdated by 16%, and an infrastructure diagram described a server on which we no longer run anything. We used the measurement, not the document.
“Complex architecture” does not mean many boxes on a diagram. It means knowing what happens when one of them does not respond. A system with three services that call each other synchronously, without a retry limit and without idempotency, is more fragile than a monolith — because each new link is a new mode of failure, and failure modes multiply faster than features.
Our work starts from separating responsibilities and ends with what is seen when something fails. In one of the proprietary platforms, messages do not pass directly between services, but through a message bus with 297 queues, 9 exchanges and 17,525 bindings, of which 278 queues are generated per client, not hand-written. Eight queues have a dead-letter exchange and eight have a five-minute message TTL — that is, a message that cannot be processed ends up somewhere it can be seen, not lost.
The patterns we put in are few and repeat: retries with a limit written in code, idempotency keys that make a second delivery of the same event harmless, a circuit breaker that stops calls to a repeatedly failing service, a consumption gate before actions that cost money, and alerts with repetition thresholds so an intermittent fault does not bury the rest.
The last part, the one that is overlooked: publishing. One of the platforms is published by file synchronisation, without Docker, with a check that the served page contains exactly the identifier of the package just published — not merely that the server returns 200 — and with automatic rollback to the previous version if the check fails. The rule appeared after a real incident in which a superficial check cancelled a good publish.
What it covers
In one proprietary platform there are 28 service directories, of which 16 have their own entry point, and in production five processes run under a process manager. Each communication channel has its own service and its own port, precisely so that a fault in one does not stop the rest. The data are modelled across four separate schemas, with 235 models in total — the separation is not only at process level, it is also at schema level.
297 queues, 9 exchanges, 17,525 bindings. The queue names are per client and, in some cases, per conversation thread — that is, the topology is generated, not hand-written. There is also an exchange with delayed delivery, for things that need to happen later, not now. Eight queues have dead-letter exchange configured and eight have a five-minute message TTL.
Three different limits for three different situations, all written in code: five retries for scheduled messages, with the counter persisted, not kept in memory; three retries at queue level, counted in a message header; and three retries with exponential backoff for sending email, on two provider paths. A retry without a limit is not resilience, it is a loop.
For third-party events we use an insert that fails if the event has already been seen: a database unique constraint violation is treated as “duplicate”, not as an error, together with a freshness check for the timestamp. For payments, the key is a mark on the credit transaction, so resending the same webhook does not credit twice. For messages, deduplication is done by identifier, with TTL and cleanup.
The integration that depends on the session of a third-party site has a real circuit breaker, not just a note in the documentation: after ten consecutive failures, calls stop for two minutes. Without it, a site that does not respond turns a failed channel into a slow platform, because everyone waits in the request.
In one of the platforms, any paid action goes through a four-step verification function, each with its own clear reason returned: suspended account, monthly conversation limit reached, daily spending cap exceeded, insufficient balance. On exception it returns "not allowed", not "allowed" — the gate closes, not opens, when something goes wrong. Behind it sits a log of consumption events with cost per event, broken down by provider, model and mode.
A sliding window kept in the database, with an in-memory map only as a shortcut within one invocation. The reason is written right at the top of the file: the processes serving requests are short-lived, so their memory cannot be the source of truth. Fifteen functions use it, and an external provider's quota has its own limiter, separate.
A 200 response does not mean the task succeeded. The wrapper we use to run scheduled tasks recursively walks the response for failure or error lists and treats "200 with failures" as a distinct result. Alerts have a repetition threshold — ten minutes in one system, two hours in another — and the marker is cleared at the first success, so a recurring fault can alert again immediately.
Publishing checks not only that the address responds 200, but that the served page contains exactly the identifier of the package just published; otherwise it restores the previous version. For the background service, publishing compares the tree by checksums and skips restart if nothing has changed — an unnecessary restart kills a loop already in progress.
What it looks like
For each link between two systems: what happens if the other is slow, if it is down, if it responds twice, if it responds incorrectly. We deliver: the list of links with the expected behaviour in each of the four situations, plus the sync/async decision for each.
Queues, exchanges, dead letters, message time-to-live, and the idempotency key for each event type. We deliver: the topology, the message contracts and the documented behaviour on resend.
Rate limiting on a sliding window in the database, the consumption gate before cost-incurring actions and the breaker on integrations outside our control. We deliver: the set thresholds, the reasons returned clearly and the tests for each refusal step.
Publishing with content verification, not just response code, previous version kept alongside and automatic rollback. We deliver: the publishing procedure, the rollback procedure practised at least once, and the health probes — one shallow for "alive", one deep for "actually working".
The document is checked against the measurement before handover, because documentation left behind is more dangerous than its absence. We deliver: the diagram, the procedures and the explicit list of places where the document and code were found to disagree, with what we corrected.
The data
The questions anyone with a data protection officer would ask — asked here before they do.
A case
The situation
A multi-client platform, each with its own communication channels, its own discussion threads and its own notifications. The naïve version — a shared queue and a filter by client ID — makes a high-volume client block the rest, and an error on one thread stops the queue for everyone.
What we built
The topology is generated per client, not written by hand: out of 297 queues, 278 are of the type "notifications for a specific account", and some go down to the level of a single discussion thread. Above them, nine exchanges separate traffic types — notifications, tokens, webhooks, channels — and there is a delayed delivery exchange for what needs to happen later. Eight queues have a dead-letter exchange, eight have a message time-to-live of five minutes. Orchestration rules can be hot-reloaded, through a publish channel, without restarting consumers.
What came out
A high-volume client does not delay the other clients, and a message that cannot be processed ends up somewhere where it can be seen and retried, instead of disappearing or blocking the queue. The number of links in the bus — 17,525 — shows exactly why the topology must be generated: nobody maintains something like this manually.
What the case does not say
The price is operational: a bus of this size needs its own monitoring and a plan for abandoned queues, otherwise it grows endlessly. And per-client generation means that deleting a client also deletes its topology — if that step is missing, dead queues accumulate.
Questions
Because the first recommendation we often give is not to separate. Every new service is a new failure mode, and failure modes multiply faster than features. One example from our own systems: the backend of one of them is a single file of 9,516 lines with 99 routes. It is not elegant and we say so, but it is published in one step and debugged in one place. Separation is done when there is a measurable reason — a service that must be scaled separately, a separate team, a separate release rhythm.
It depends on what we decided together at the mapping stage, and that is the point. In our systems: the message goes into a queue and is retried a capped number of times, then reaches the dead letter exchange, where it can be seen; the unstable integration has a circuit breaker which, after ten consecutive failures, stops for two minutes instead of keeping requests waiting; and the actions that cost money are stopped by a gate which, on exception, refuses, does not allow.
With an idempotency key, not with an “have I seen this before?” check that has a data race. Concretely: we insert a row with the event identifier and, if the database refuses due to a uniqueness violation, we treat that error code as a duplicate signal, not as a fault. For payments, the marker sits on the credit transaction, so resending the same webhook does not credit twice.
Both, and we justify the choice every time. One system runs in a container, but without mounted volumes, which means that publishing is a copy into the container plus restart — a `docker rm` there loses the state, and it is written exactly like that in the operating document. Another platform has no Docker at all: publishing is file synchronisation, and the publishing scripts are installed manually with administrator rights and exposed to the automated flow through exactly two permitted paths. The reason is written in code: a script that the flow can rewrite is a script through which the flow can escalate.
From bad experience. One of our daily reports was dead for eleven days, between 7 and 17 August, while the scheduled task was triggering every day — the command used exited silently on error and wrote nothing. Since then, every task runs in a wrapper that scans the response for failure lists, treats “200 with failures” as a distinct result, has a maximum execution time and writes a structured line per run. Alerts have a repetition threshold, and the marker is cleared on the first success.
A lot, if the vendor is in a fast-moving area. For voice we have three separate bridges — one for each real-time speech provider — which implement the same interface. The bridge converts audio between the phone exchange and the provider, on a dedicated port, with the health server bound only to the local interface, not exposed. Changing the provider becomes a decision, not a rewrite.
Not everywhere, and we prefer to say where. While preparing this page we measured two systems and found our own document was behind: a statement about the size of a file was about 16% smaller than reality, and an infrastructure diagram described a server we had already moved away from. The rule we apply and require in projects is: when the document and the measurement disagree, the measurement is right, and the document is corrected in the same step.
We do not promise availability targets without measurement — a figure such as “99,9%” requires field data over a period, and where we do not have it we do not state it. We do not design systems we cannot operate or hand over: if the result is an architecture your team cannot maintain, it is the wrong architecture. And we do not carry out compliance certifications — we can build the controls, we cannot issue the certificate.
23 of them are code and files from our repositories. We don't publish their name or line: together, on a single page, they would describe too precisely how systems that aren't only ours are built. We go through them with you, in the repository, on request — verification stays possible, it just happens in a conversation.
Tell us about your process. Together we decide what is worth building, what we can connect, and how we check the result.