Cgo exists because goroutine stacks are small and grow on demand. C/C++ expect large fixed size stacks. While you can technically call into C++ without going through cgo (see https://github.com/petermattis/fastcgo), you can't avoid the stack problem. Cgo solves it by switch from the goroutine stack to the stack of the thread underlying the goroutine.
Pebble and FoundationDB are apples and oranges. Pebble is per-node KV storage engine. FoundationDB is a distributed multi-modal database. Internally, FoundationDB uses a library like Pebble for the per-node data storage. I think at one point it used SQLite. I'm not up to date on what it currently use. I seem to recall FoundationDB was writing their own btree-based node-level storage engine to replace the usage of SQLite.
The equivalent of FoundationDB is present inside of CockroachDB: a distributed, replicated, transactional, KV layer. This is where a big chunk of CockroachDB value resides. This is where our use of Raft resides. Pebble lies underneath this.
The current production storage engine is an old-ish version of the SQLite btree. A new btree engine is being written now and is available but I don’t know if it is being used in production anywhere.
RocksDB is shipping soon thanks to some work by members of the community.
Pebble does not currently implement MultiGet as CockroachDB did not use RocksDB's MultiGet operation. CockroachDB can use multiple nodes to process a query by decomposing SQL queries along data boundaries and shipping the query parts to be executed next to the data. CockroachDB can't directly use MultiGet because that API was not compatible with how CockroachDB reads keys.
RocksDB MultiGet is interesting. Parallelism is achieved by using new IO interfaces (io_uring), not by using threads. That approach seems right to me. See https://github.com/facebook/rocksdb/wiki/MultiGet-Performanc.... My understanding is that io_uring support is still a work in progress. We experimented at one point with using goroutines in Pebble to parallelize lookups, but doing so was strictly worse for performance. Experimenting with io_uring is something we'd like to do.
Indeed the conceptual fork point mentioned is RocksDB 6.2.1 which came before those features. The problem with RocksDB is that one thread only makes one request at a time. I should've phrased my question more succinctly: Is Pebble/CockroachDB capable of saturating the backplane with requests in parallel? Does it multiplex a single query by dispatching smaller requests to a thread-pool?
> Is Pebble/CockroachDB capable of saturating the backplane with requests in parallel?
Yes.
> Does it multiplex a single query by dispatching smaller requests to a thread-pool?
Yes, though it depends on the query. Trivial queries (i.e. single-row lookups) are executed serially as that is the fastest way to execute them. Complex queries are decomposed along data boundaries and the query parts are executed in parallel next to where the data is located.
> So I understand the rationale for writing your own storage layer and think this is an awesome project, but there's something missing for me. One of the issues Peter brings up is they've come across a number of serious bugs in RocksDB. My question is, why would Pebble have less bugs. In fact, I would expect it to have significantly more bugs because Cockroach is the only company using Pebble.
We're only worried about functionality in Pebble used by CockroachDB. RocksDB has a huge number of features that sometimes have bugs due to subtle interactions. There is a very stable subset of RocksDB: the configuration and specific API usage patterns used internally by Facebook. That precise combination has seen extreme testing. But that isn't the subset of RocksDB used by CockroachDB. I would guess that the most significant testing of the subset of RocksDB used by CockroachDB is the testing we do at Cockroach Labs. Now that testing is being directed at Pebble along with the Pebble-specific testing detailed in the post.
> For example, it's possible the filesystem had synced some of the buffered data to disk, but not all of it. There's no guarantee about what buffered data was synced to disk. All you know is that some, all, or none of it made it to disk.
The filesystem does provide guarantees when you use fsync() and fdatasync(). Postgres relies on these guarantees. So does RocksDB. Pebble's usage of fsync/fdatasync mirrors RocksDB's. Our crash testing is not testing the filesystem guarantees, only that we're correctly using fsync/fdatasync (which is hard enough to get right).
> The filesystem does provide guarantees when you use fsync() and fdatasync(). Postgres relies on these guarantees. So does RocksDB. Pebble's usage of fsync/fdatasync mirrors RocksDB's. Our crash testing is not testing the filesystem guarantees, only that we're correctly using fsync/fdatasync (which is hard enough to get right).
As far as I'm aware, the fsync/fdatasync sharp edges are around what happens after an fsync/fdatasync failure. My understanding is that you can't rely on anything. The only sane option is to crash the process and attempt recovery on restart. Even that is fraught because data can be in the OS cache but not synced to disk. Pebble (and RocksDB) both take a fairly pessimistic view of what can be recovered. Sstables that were in the process of being written are discarded. The WAL an MANIFEST (which lists the current sstables) are truncated at the first sign of data corruption. Getting all of this right definitely takes time and effort.
From the Rebello paper:
> However, on restart, since the log entry is in the page cache, LevelDB includes it while creating an SSTable from the log file.
Pebble and RocksDB both inherited this behavior. The nuance here is that the sstable is then synced to disk and no reads are served until the sync is successful. If the machine were to crash before the sstable was synced, upon restart we'd rollback to the durable prefix of the log.
Yes. So far performance was worse in experiments, and the durability improvements are questionable because it is extremely difficult to get a clear understanding of the durability semantics of direct IO. If you can find a pointer to clear documentation of what those semantics are I'd be extremely interested in reading it.
RocksDB supports using Direct IO for flush and compaction (use_direct_io_for_flush_and_compaction), enabling that can improve write throughput for my workload in RocksDB. Any plan to do that in pebble?
Direct IO is on our radar, though when I experimented with enabling direct IO in RocksDB it only hurt CockroachDB benchmarks. This was several years ago. I believe newer releases of RocksDB have made improvements in this area.
The TLDR is that the GC did cause problems so we had to avoid it for the block cache. Luckily we were able to do so without exposing the complexity in the API. Not for the faint of heart. Don't try this at home kids.
Any reason why the block cache needs to be 10s of GB in size? Cassandra, for example, usually has a rather small key cache on heap, and then just relies on the kernel page cache. I don't have experience with cockroach, it looks like the block cache is similar to cassandra row cache, which can be configured to be on heap or off heap, but usually not beneficial to enable.
This is a really hard question to answer because there are so many different scenarios to test. Even if you restrict the question to KV operations a simple answer isn't forthcoming. For example, if you want a distributed system with the fastest writes, then Cassandra is your best choice. Oh, but you better pay a lot of attention to all of the caveats regarding tombstones and node outages and overwriting data.
Overall, performance has improved a lot over the past year, but we still have a lot of work to do. Some of the high-contention scenarios that caused poor performance in the Jepsen tests have seen a huge improvement (10-100x), though admittedly we were starting from a poor position.
So far we've performed the most significant comparisons against Cassandra and MongoDB. In terms of performance Cassandra > MongoDB > CockroachDB, but note that most (all?) benchmarks test idealized situations. YCSB tests ideal write scenarios where data is never overwritten or deleted and data correctness is easily achieved because there are no node outages.
Unsurprisingly, PostgreSQL will outperform a single-node Cockroach cluster for simple operations. The internal target was to be within 2x of PostgreSQL performance for simple INSERT, UPDATE, DELETE and SELECT statements. We were at that target earlier in the year, but a quick check reveals that we've slipped a bit.
Sorry I don't have anything more concrete to offer at this time. I realize I've been saying "we're working on it" for too long. Really, we are.
CockroachDB also stores SQL metadata inside of the KV store, but that metadata is also gossiped around the cluster (i.e. it is replicated to every node) so that SQL execution almost never has to read it from the KV store. Handling changes to this SQL metadata is challenging and required a design [1] that is unable to take advantage of the easy correctness of simply doing reads from the KV store on every operation.
I see -- so how is performance? I see that there are no published numbers for e.g. TPC-C or YCSB.
(For the record -- in case I seem to be especially pessimistic -- I'm actually extremely hopeful that CRDB will actually make scaling OLTP applications significantly easier for everyone. It's just that this probably shouldn't be FDB all over again.)
Performance is "ok". CockroachDB 1.0 won't be winning benchmarks, but that was never the goal. We have seen near linear scaling from 3-64 nodes on a uniformly random write workload. And the new distributed SQL execution engine has shown O(num-nodes-in-cluster) speedup on some queries. But none of this has been wrapped up into published numbers, so take it with a grain of salt.
Hi Peter, sounds pretty awesome! Quick question though - on your front page you say that CockroachDB does SQL - but if it can't do a join, then how can you say it uses SQL? Or is distributed SQL a different thing entirely? It does sounds like a very limited SQL subset though... I'm sure I must be missing something as I'm not familiar with your product.
Also, what levels of isolation do you actually offer? Serialized snapshot isolation appears to be MVCC, but I see you also have just snapshot isolation - what is the difference?
Edit: oh brother, the proof you link to, I just realized I bought that book some time ago and never got around to reading it... Transactional Information Systems by Weikum & Vossen, right? time for me to hit the books I guess. Still trying to get my head around that second graph you have drawn, can't work out how you have gotten it :(
SQL is not a single language but rather a family of languages. Each RDBMS that advertises support for SQL ends up implementing a different flavor of SQL, most of the times they are not even compatible :)
For now CockroachDB's supports a subset of the SQL implemented by other databases, with some extensions of its own. This subset will grow over time.
The two levels of isolation offered are snapshot (SI) and serializable (SSI). Snapshot means that concurrent transactions are atomic with regards to each other and "see" the same initial state from the DB. Serializable adds into this that they can't introduce write skews, ie if there are concurrent transactions they will "see" each others effects in some order.
Hi Chris, we describe CockroachDB as a SQL database because that is what we're aspiring to. The missing functionality (i.e. joins) is on our near-term roadmap.
Joins over a distributed database aren't easy. Love to see what you have going! The main issue I see with distributed joins are that they need to be done in a single transaction - if even one table gets an insert, delete or update then it invalidated the join. But this distributed serialised snapshot isolation, that sounds like it might be the best way around it.
You may find that you will have to choose at best between two of these three: joins, speed and convenience. In other words, you won't be able to join quickly in a convenient way for the users, of if you want the user to join like he can in SQL database, it will be much slower.
