Under the hood, ParadeDB is built by integrating Tantivy (a Lucene-inspired Rust search library) inside Postgres. This is to say: I agree with you -- We're not trying to claim that Postgres itself is an alternative to Lucene, but rather that something like Lucene can be integrated inside Postgres so that you can get the power of both in a single system (or in a cluster of Postgres instances)
- saves infra costs
- saves infra headaches
- devs only need to be experts in one system (or well I guess one and a half, probably there's something to learn about ParadeDB too, but probably less than in learning Lucine)
- no need to worry about keeping data up to date in the separate seach system
- all data is available when you want to do new queries that you handn't thought of when implementing the data transfer/indexing
(ParadeDB maintainer here) Yes! We've already built some faster analytics in Postgres, and have a lot more coming. Here's some relevant context in case you're curious: https://www.paradedb.com/blog/faceting
(ParadeDB maintainer here). This is super cool. Congrats on the project, and I'm excited to see ParadeDB be used to power this kind of use case. If there's anything else you need to ship Omni, don't hesitate to reach out to me!
One-click deploy with Render, and we're directly in contact with the core team to get it added to their official docs. I hear the PR is up internally :)
That's true. For this reason, most modern search engines support language-aware stemming and tokenization. Popular tokenizers for CJK languages include Lindera and Jieba.
ParadeDB is an alternative to Elasticsearch built on Postgres. We're building a Postgres extension in Rust that offers a new index type optimized for full-text search and aggregate/analytics workloads. We solve three problems with Elasticsearch today:
- Lack of read-after-writes guarantees
- Lack of JOINs
- Infrastructure complexity & cost due to syncing Postgres and Elastic.
We're open-source, and our repository is available at https://github.com/paradedb/paradedb. We're a Series A team of 8 distributed across the US and Canada. Most folks on our team have 10+ years of experience in database internals at companies like Twitter, MongoDB, Oracle, Instacart, etc.
If you know Rust and/or have experience working on DB internals and want to work on cool systems problems, shoot us a note. We hire with conviction, have lots of room to grow, and exciting technical problems to solve. My email is phil@paradedb.com.
You don't need to. Customers usually deploy us on a standalone replica(s) on their Postgres cluster. If a query were to take it down, it would only take down the replica(s) dedicated to ParadeDB, leaving the primary and all other read replicas dedicated to OLTP safe.
Are you saying that the cluster isn't homogenous? It sounds like you're describing an architecture that involves a cluster that has two entirely different pieces of software on it, and whose roles aren't interchangeable.
Bear with me, this will be a bit of a longer answer. Today, there are two topologies under which people deploy ParadeDB.
- <some managed Postgres service> + ParadeDB. Frequently, customers already use a managed Postgres (e.g. AWS RDS) and want ParadeDB. In that world, they maintain their managed Postgres service and deploy a Kubernetes cluster running ParadeDB on the side, with one primary instance and some number of replicas. The AWS RDS primary sends data to the ParadeDB primary via logical replication. You can see a diagram here: https://docs.paradedb.com/deploy/byoc
In this topology, the OLTP and search/OLAP workloads are fully isolated from each other. You have two clusters, but you don't need a third-party ETL service since they're both "just Postgres".
- <self-hosted Postgres> + ParadeDB. Some customers, typically larger ones, prefer to self-host Postgres and want to install our Postgres extension directly. The extension is installed in their primary Postgres, and the CREATE INDEX commands must be issued on the primary; however, they may route reads only to a subset of the read replicas in their cluster.
In this topology, all writes could be directed to the primary, all OLTP read queries could be routed to a pool of read replicas, and all search/OLAP queries could be directed to another subset of replicas.
Both are completely reasonable approaches and depend on the workload. Hope this helps :)
If you have a ETL pipeline specialized for PG logical replication (as opposed to generic JVM based Debizium/Kafka setups), you get some fraction of the same benefits. I'm curious about Conduit and its postgres plugin.
That leaves: ParadeDB uses vanilla postgres + rust extension. This is a technology detail. I was looking for an articulation of the customer benefit because of this technologically appealing architecture.
Our customers typically deploy ParadeDB in a primary-replicas topology, with one primary Postgres node and 2 or more read replicas, depending on read volume. Queries are executed on a single node today, yes.
We have plans to eventually support distributed queries.
One of the ParadeDB maintainers here -- Being PostgreSQL wire protocol compatible is very different from being built inside Postgres on top of the Postgres pages, which is what ParadeDB does. You still need the "T" in ETL, e.g. transforming data from your source into the format of the sink (in your example CrateDB). This is where ETL costs and brittleness come into play.
So, I'm not versed enough in legal matters to be certain about this, so I tend to fallback to caution, but (A) customers I've worked with in the past seem to be wary of such copyleft licenses and (B) the contagious nature of such license would make me think twice about using it in a project of my own as well.
It would be nice to have such notion challenged but I'm not sure what would change my mind.
I would expect that most commercial companies that use Grafana would obtain a commercial license?
reply