IMO transparent proxies for observability are not the best pattern. And I speak from experience, we developed the Postgres plugin for Envoy [1], [2] and we use it in StackGres [3], among others, for this very same reason, observability.
There's two main problems with said proxies:
* Latency. Yes, yes, yes, they add "microseconds" vs "milliseconds for queries", and that's true, but just part of the story. There's an extra hop. There's two extra sets of TCP layers being traversed. If the hop is local (say a sidecar, as we do in StackGres) it adds complexity in its deployment and management (something we solved by automation, but was an extra problem to solve) and consumes resources. If it's a network hop, then adds milliseconds, and not microseconds.
* Performance. It's not that hard to write a functioning PG wire proxy (it's not trivial either). But it is extremely hard to make it perform well under high load scenarios. Most of the proxies I have seen crack down under moderate to high performance.
What's the solution then? The Postgres extension model to capture the metrics (we also experimented with eBPF, but it causes too many kernel-user space context switches when you can do the same in an extension without them), and a small sidecar to push the metrics out via a standardized protocol like OTEL.
I get what you're saying about a proxy like this, latency & performance would suffer, however minor, and in production DB land this really matters.
I've just not sure it is much of a slight on such proxies.
You don't need to run this always inline in production to get amazingly useful results. Yes, there are lots of production insight solutions out there, but lots of modern stacks can be complex enough that just getting a quick handle on how the page you're debugging talks to your DBs can be incredibly useful, which is where I love the idea of a solution like this.
Sure, it is mytop / pgtop, but trying to offering it at a different layer & with a modern interface. Seems useful to me.
> The Postgres extension model to capture the metrics (we also experimented with eBPF, but it causes too many kernel-user space context switches when you can do the same in an extension without them), and a small sidecar to push the metrics out via a standardized protocol like OTEL.
The extension model is great, but it doesn't work with existing postgres providers (RDS, Aurora, etc.). Unless one such extension becomes standard enough that all providers will support it. That would be ideal, IMO.
To be clear, I don't mean pg_stat_statements, that is standard enough, but an extension that pushes the actual queries in real-time.
> If it's a network hop, then adds milliseconds, and not microseconds.
Are you talking about connection establishing time or for query delay? I think it should normally be under a millisecond for the later.
> The extension model is great, but it doesn't work with existing postgres providers (RDS, Aurora, etc.). Unless one such extension becomes standard enough that all providers will support it. That would be ideal, IMO.
That's true, but that's a problem of the PGaaS providers for them to fix by providing the best functionality available. I'm planning on following this route in a pure OSS basis.
> Are you talking about connection establishing time or for query delay? I think it should normally be under a millisecond for the later.
Network trip. If the proxy is not co-located with the database but rather a network hop away, that's usually adding at least 1ms there, could be more.
Postgres can really scale well vertically (and horizontally for read-only workloads) as the post shows.
However, I'm still surprised about the reasons for not sharding. They have been mentioned before, but I haven't seen a substantial rationale.
Sharding is almost only analyzed from the perspective of write scaling. But sharding may not only be about write scaling, but a path to reducing blast radius. And this is something that triggers much earlier than write scaling needs (especially given how well Postgres scales vertically and reads).
When you shard your database, you end up having N clusters (for HA purposes, each "shard" must be a primary-replica(s) cluster itself), each holding 1/Nth of the data.
There are certain scenarios which, while unlikely, may hit you: data corruption in the WAL replication stream, a problem during a major upgrade, a situation that requires a whole cluster restore from a backup, you name it. For those cases, the whole cluster may experience notable downtime.
If you have a single cluster, 100% of your users experience downtime. If you sharded into N clusters, only 1/Nth of your users experience downtime. For a company servicing 800M users the difference from both scenarios is dramatically different. Even for much much smaller companies.
I'm puzzled why this is not perceived as a risk, and if it is not, how it is mitigated.
While I wouldn't advocate to shard "too early", given that it comes with notable caveats, I believe more and more in sharding your workloads when possible more earlier than later. Way before truly needing it from a write scaling perspective. Because apart from reducing the blast radius, it applies implicitly the principle of "divide-and-conquer", and your problems become much more manageable (your number of connections per cluster decreases at will, backup restore times can be a fraction of the time, logical replication can be considered as a true option for replication/upgrades/etc if you keep shards relatively small and many other operational procedures are greatly simplified if now you have much smaller databases, even if you have many more of them).
Microsoft originally bought CitusData and rebadged it as Azure CosmosDb for Postgres Cluster. Microsoft have been recommending partners to now avoid that product. It does not and will not support Entra federated workload identities (passwordless).
The replacement will be Azure Database for Postgres with Elastic Clusters. I think it is still in preview.
Again it’s Citus based, but without the CosmosDb badge and it will support federated workload identities.
There's a fundamental disconnect: OP refers to senior engineers being replaced with AI, whereas the evidence and logical reasoning points much more to junior engineers being replaced by AI. And that premise seems like a quite plausible one...
>OP refers to senior engineers being replaced with AI, whereas the evidence and logical reasoning points much more to junior engineers being replaced by AI.
If industry cared about future seniors, they'd invest in juniors. But that's not what's happening. AI will effectively replace seniors in 20 years with the current trajectory. Whether or not that replacement is adequate or not is the bigger question.
I think the junior thing started ~24, early ~25. Because back then the level of the current models was at or above that level, with somewhat flaky reliability. In the past year that's changed. We are now at "mostly reliable" in any junior-level stuff, and "surprisingly capable, maybe still needs some hand-holding" at advanced / senior-level stuff. And somewhat super-human if the problem is easily verifiable in a feedback loop (see the atcoder stuff).
> "As part of the acquisition, we made an important decision: not only to keep PeerDB open source"
> "Here goes the Open Source reference to our validation logic."
> "PeerDB Open Source Repository"
I hate to be that guy, but PeerDB seems to be governed by the Elastic License [1] which makes it NOT open source.
The difference is not small, but significant for many. For one, it won't get integrated into other OSS projects.
In my particular case, we have integrated Debezium Embedded into StackGres, as a high level object (CRD) named SGStream [2]. It allows Postgres logical replication from Postgres and exports to another Postgres and/or CloudEvents. No Kafka required. We'd love to consider other alternatives like PeerDB, but not being OSS is a red line we can't cross (having said that, we're really happy with Debezium in general, but having competition and alternatives it's always great).
(this is Sai, the author of the post and also PeerDB co-founder)
The wording in that post was an unintentional miss on my part. Apologies for that. We’ve just fixed it. Thanks for flagging it!
To add some context, PeerDB was originally released under ELv2 well before the acquisition. During the acquisition we made a choice to keep the project as-is rather than change its license, so this wasn’t a new decision made at that time — just continuity with how the project already existed.
We appreciate the feedback, around integration and downstream OSS adoption. That overall makes sense. We’ll take it into account as we think about licensing going forward.
Separately, I really wish you tried PeerDB out. The ease-of-use and performance around larger Postgres datasets (TBs to 10s of TB) would’ve been something you would have probably appreciated. That is something we optimized a lot on over that last few years. May be sometime in the future! :)
Thank you for acknowledging this and updating the blog post correspondingly.
I'd love to test and compare PeerDB with Debezium (Embedded), and even SynchDB. But as said, the licensing is a blocker for us. And given the focus and bandwidth we currently have, we won't have the chance to deeply look at it unless there's a high chance we could integrate it into StackGres.
Anyway feel free to DM me if you'd like to talk more.
I think it is more ClickHouse Marketing being that guy; they have a vaguely aggressive feel to them and slightly-questionable claims like that seem on-vibe to me. Although it is tolerable. Selling databases is hard, the specialists who actually understand the trade-offs are so specialised they usually aren't the person who makes the call on what to use. At least they're selling an interesting [0] DB, Clickhouse has a fun design. They don't mislead anyone who is interested in the details and their documentation is in the end rather detailed.
I have two Framework 13 DIY, top of the line (Ryzen AI 9 HX 370, 96 GB of RAM, 2TB WD BLACK SN850X). Well, TBH just one, the other one was stolen with less than one month, but that's a different story.
Can't be happier with this choice.
First of all, now I feel confident I can easily tinker with my laptop, open it and replace any components. No, I'm not repeating their Marketing, I'm truly confident now. Some months ago my previous Lenovo X1 Carbon stopped charging and I was scared as hell. Sure, I have everything in version control and cloud drives blah blah blah but if it doesn't charge, it's dead. Sure, I can extract the NVMe (is it easy at all?) and rsync the data (for a faster recovery) but is that easy, feasible, can I do it on the go (X1 stopped charging while I was on a trip).
The Framework 13 DIY I built it in less than 10 minutes. I can easily disassemble again in a heartbeat. This is priceless.
Other than this, build quality is higher than expected, and several other people said exactly the same when I showed my unit. Powerful? Most than any desktop out there. Perfectly portable. Works well OOB with Linux. What else do I need? Nothing, it's my laptop and will be the next one.
(Actually the next one may be cheaper, as I may only need to replace the motherboard, we will see)
Writing this on a first-gen 13 DIY, so it's already a bit dated. I've already added additional RAM and swapped in a much bigger SSD (really underestimated how much space you need if you heavily use docker and VMs).
The 1st year was a bit bumpy with 4k monitors over a USB-C hub being somewhat flaky. Ever since a clean Ubuntu reinstall, I'm very happy, no complaints whatsoever.
Sure, it costs more, but the combo of perfectly running Linux, giving me the piece of mind of repairability and easy upgrades for me justifies a higher price.
On the other hand, I'm not willing to pay the kind of premium you have with Apple products, where for incremental steps in more RAM or SSD you pay a multiple of the off-the-shelf price of the added space.
"Most components" seems a bit of an understatement when compared with the Framework.
Sure, some components can be replaced. And not at the same cost (opening and manipulating the Framework vs the ThinkPad). But not all, like the motherboard.
I may have built multiple dozens of computers in my life, so it's not that I'm new to this world.
I've been a very happy FW16 user, so much so that I got my partner an FW13 to replace her very old and very bulky gaming laptop. Similar specs to yours + AMD dGPU module. Recently cleaned the innards and upgraded my mainboard to the HX 370 in ~20min with zero issues. Compared to what I've been reading on various forums/threads, my experience as a whole has been virtually free of problems! FW + NixOS is a potent combo that squarely satisfies the kind of configurability/overall experience I need/want in the tech I use. (And perhaps more importantly, it handily assuages my paranoid mind.)
TBH I haven't deeply tested it yet. Laptop is a few days old and I'm still "automating the install" (Ubuntu autoinstall + Ansible tasks for post-installation including BIOS upgrade and install Nix + Nix and home manager to install everything else; post(s) upcoming), so I haven't done "production use" yet.
Anyway, and while I love long battery life, it's not my main concern. Most of the time I have a power socket available and/or a nice portable battery pack that does the job. Laptop feels so much faster than my X1 Carbon that everything else seems to be a distant second feature.
P.S. Hi, Yorick, again, not the first time we cross paths ;)
We're constantly striving to improve the user experience and the quality of StackGres. Would you mind sharing some feedback as to what made your experience not good with it?
Did you join the Slack Community (https://slack.stackgres.io/) to ask if you were facing some trouble? It always helps, even if it is just by sharing your troubles.
(If you'd like to share feedback and do so privately, please DM on the Slack Community)
I did try slack. Maybe the problem is it was launched much too early. A certificate expiry issue caught me out because there wasn't an automatic process on this version to roll them over. Ironically a single database instance would have been much much more stable. I upgraded but this didn't bring up the database, restoring through the portal failed, so I had to create a new PG cluster to get my site up and I never ended up recovering the data as the process was very tedious involving PVCs rather than just pointing to my bucket.
The ratio of open to closed issues on the repo is much worse than CNPG so I would simply start there.
Thank you for your feedback. I'm trying to extract possible improvement actions from your comment, and here are my thoughts.
That certificate expiry issue was unfortunate, but was resolved (if I'm not mistaken) a couple of years ago.
StackGres is just a control plane, your database is as stable as a standalone one. StackGres itself may fail and it won't affect your database, it's not on the data plane. Indeed, it has a feature to "pause it" if you need to perform some manual operations (otherwise everything is automated).
There are procedures to reconstruct a database from PVC. It's arguably tedious, but should be much simpler than running a Postgres pod without the help of an operator like StackGres.
As for the ratio of issues: most of the issues that we get are feature and/or extensions requests, and certainly we can't tackle them all. Most, if not all, outstanding issues are addressed within a reasonable time frame. Is there any particular issue that would itch you that is open? I'd be happy to personally review it. Yet, there are as of today more than 2K closed issues, I won't call that a small number.
I'd also weight the importance of issues, like the split brain that CNPG suffers [1] and that apparently won't even be solved. StackGres relies instead on the trusted and reputed Patroni, which is known NOT to risk split brains that could lead to severe data loss.
We're constantly striving to improve the user experience and the quality of StackGres. Would you mind sharing some feedback as to what made your experience not good with it?
Did you join the Slack Community (https://slack.stackgres.io/) to ask if you were facing some trouble? It always helps, even if it is just by sharing your troubles.
(If you'd like to share feedback and do so privately, please DM on the Slack Community)
Does the unused index feature look into replicas? (I guess it doesn't) It's risky to delete an unused index by looking at a single instance, since it may be used by other read replicas.
Using natural keys is what actually can prevent duplicate rows. In your above example, if email is the PK, there would be no duplicates. But adding an id as a PK would essentially keep your database with duplicates:
There's two main problems with said proxies:
* Latency. Yes, yes, yes, they add "microseconds" vs "milliseconds for queries", and that's true, but just part of the story. There's an extra hop. There's two extra sets of TCP layers being traversed. If the hop is local (say a sidecar, as we do in StackGres) it adds complexity in its deployment and management (something we solved by automation, but was an extra problem to solve) and consumes resources. If it's a network hop, then adds milliseconds, and not microseconds.
* Performance. It's not that hard to write a functioning PG wire proxy (it's not trivial either). But it is extremely hard to make it perform well under high load scenarios. Most of the proxies I have seen crack down under moderate to high performance.
What's the solution then? The Postgres extension model to capture the metrics (we also experimented with eBPF, but it causes too many kernel-user space context switches when you can do the same in an extension without them), and a small sidecar to push the metrics out via a standardized protocol like OTEL.
[1]: https://www.envoyproxy.io/docs/envoy/latest/configuration/li...
[2]: https://www.cncf.io/blog/2020/08/13/envoy-1-15-introduces-a-...
[3]: https://stackgres.io
Edit: formatting