"Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (The formalization of CAP, which caused a ton of very poor trade-off thinking in the decade that followed by defining Availability in a very goofy way. Still a classic.)
"Practical Byzantine fault tolerance" (Moved the conversation on Byzantine faults forward significantly).
This is just a short selection. There's so much good stuff going back in the 70s and 80s distributed database literature, for example (and in the modern systems and DB literature too).
Lamport lists his collected works on his site[0] along with some commentary for each one, and directly mentions, The Maintenance of Duplicate Databases, as the origin for logical clock paper. Many of the notes have interesting commentary.
> Many computer scientists claim to have read it. But I have rarely encountered anyone who was aware that the paper said anything about state machines. People seem to think that it is about either the causality relation on events in a distributed system, or the distributed mutual exclusion problem. People have insisted that there is nothing about state machines in the paper. I've even had to go back and reread it to convince myself that I really did remember what I had written.[1]
Inside multiple AWS products (including DynamoDB, Kinesis, and Aurora DSQL) is a system called Journal that moves a ton of data. It uses a variant of chain replication.
EBS is also a chain replication variant at heart, and moves even more data.
Basically naive CAP theorem assumes "no latency (delay) allowed", in which case, yeah, sure, the quorum must agree or you are writing to a split-brain quorum somehow. But what if delay sensitivity is a knob that can be tuned, and writes have conflict resolution?
Martin Kleppmann's book Designing Data-Intensive Applications is considered a useful tome for anyone who needs a insight in the complexity involved in distributed systems (hint: getting data in ram flushed to an SSD also involves multiple moving parts)
Basically, many people convinced themselves that the formalization of CAP means that systems need to choose between highly available and strongly consistent, and hence chose eventual consistency. This is, partially, because Gilbert and Lynch define "availability" to mean "available to all clients, even those on a minority side of a partition". The much more useful "available to a majority of clients" is achievable at the same time as strong consistency in presence of a single partition, and this is the common cloud failure mode.
Other, much more reasonable trade-offs, lead to eventual consistency too. Mostly latency optimizations, but many of those also lead to non-zero RPO and so are undesirable for multiple reasons. We discuss some of this in section 8 of the DSQL paper: https://arxiv.org/pdf/2607.13276
Thank you both! That was really enlightening & made for interesting reading!
Though the note at the bottom of the article to teachers to not start with CAP has me concerned about the quality of other content I’ve seen thus far …
Among the thugs, Heat, and Dirt are all great reads. Heat is my favorite of them. Highly recommended if you've ever worked in a kitchen, or considered it.
You absolutely can run agents on a regular VM. But if you want to build multi-tenant and multi-agent systems with strong security boundaries, then having a VM or MicroVM per agent session (or session with a group of agents) really simplifies things.
When we did AWS AgentCore Runtime last year we introduced session isolation, with MicroVMs per session. You can think of Lambda MicroVMs as the same stack, but generalized to fit a larger number of application patterns.
To be fair to jacobgold, at this point there is more or less an AWS services announcement singularity: if you didn't see the announcement when it happened you may never catch up or even find it in the wretched console website.
Though I did know about this one! (Because I saw the announcement.)
It just seems pretty different to me? I've lots of similar stuff and yet I still don't understand what it's for and how it works after scanning the docs quickly.
> Of course, this assumes independent events. World Cup, super bowls, etc break these assumptions.
Yes, this is very true. The model here works for Poisson arrivals and exponential service time (the M/M), which are poor approximations of real-world traffic patterns (which tend to be non-stationary and non-ergodic, and include substantial seasonality). However, the frequency of that seasonality is typically rather low (e.g. daily cycles), and so these stronger assumptions are quite defensible for short time periods.
A better approach is to do simulation with real traffic patterns, or even with more sophisticated parametric models, and get better answers (e.g. https://stability-sim.systems/). The good news is that kind of simulation is cheaper to do than ever before.
One explanation would be that more load could mean higher (absolute) variance in queue length, and therefore higher latency especially at higher percentiles. It doesn't work out that way (for reasons that Erlang actually writes about in one of his original works), but it's not an entirely unreasonable intuition.
> What's conspicuously missing is the plot of performance when you do have a well tuned queue in front of the service.
As in between the service and the load balancer? There's already an infinite queue in the load balancer. You can try that out on https://stability-sim.systems/ to see the effect, but the short version is that (in this model) it makes things worse.
If you're saying that the queue in the load balancer should be limited in size to reduce tail latency, then I agree.
No, I mean when you have a queue broker that the backends can pull work from when they become idle, rather than relying on load balancing which will send work to backends while they're still busy.
This scenario already works that way. The very first sentence says "servers, each of which can only handle a single concurrent request, and has no internal queuing". This implies that the load balancer waits for a server to finish a request then immediately sends the next one.
I don't believe it does. As I understand it, the load balancer has a queue in which it can buffer infinite requests, but it drains that queue by pushing work to the backend servers in what's probably a round-robin fashion. So there is secondary queueing at each server. Even the "least connections" strategies available through some load balancers do not usually behave as you might expect (by always sending the next request to a server that's idle). Pull-based load balancing via a queue has its own downsides but the big upside is to make latency essentially a constant low overhead regardless of the number of servers in the typical case.
I think your imagination decided to rapidly overcomplicate what is literally (literally literally) Queuing Theory 101 example
If I were to guess there weren't any "backend servers" at all. It was just array of random increasing numbers (that stand for request arrival times) and arrays of numbers with minimum distance (that stand for time each consumer took a request)
there's no connections to "least-ify" the strategy about. There's no difference between consumers, no matter the amount of requests having been processed
> With serializable, you need to be a little careful not to have hot rows. Avoid them by sharding commonly written values
Unfortunately, serializable isolation requires detecting or preventing read-write conflicts (i.e. one transaction writing a row that a concurrent transaction has read). This is the performance impact of serializability: you need to be very careful what you read, because if you read too many rows you prevent any concurrent transactions from updating those same rows. Read-only transactions are OK (because MVCC), and read-only tables are OK (because there's no read-write conflict if a table is mostly read only), but tables that are both written and heavily read are where you get performance problems.
With snapshot isolation (e.g. Oracle's serializable, Postgres repeatable read), only write-write conflicts matter. There it doesn't matter what a transaction reads, and reads never need to block (or abort) writers. So what you say is true for snapshot, but not for serializable.
Interestingly, serializable's lack of need to detect write-write conflicts means that (in some implementations) it can be faster than snapshot for blind writes (i.e. anything that's not a read-modify-write under the covers).
I agree with what you're saying. I think we're saying the same thing. I see that in the text you're quoting I only proposed a way to handle hotly written rows, and didn't address read rows being changed. This is also a problem.
My broader point is that with serializable you need to be aware of these bottlenecks in the database and you need to create a data model and access patterns such that permance hits are avoided as much as possible.
It’s somewhat rare for a transaction to read many rows and then write. Typically a transaction reads a single row, applies business logic, writes a single row. Then you have analytic workloads that do no writes and read entire tables.
The 70s, if you want to be pedantic (e.g. Gifford's "Weighted Voting for Replicated Data" or Thomas's "A Majority Consensus Approach to Concurrency Control for Multiple Copy Databases", both from '79).
For DynamoDB, check out https://www.usenix.org/conference/atc22/presentation/elhemal... https://www.usenix.org/conference/atc23/presentation/idziore... and my analysis of the differences here https://brooker.co.za/blog/2025/08/15/dynamo-dynamodb-dsql.h...
reply