I notice you didn't provide any specific comparison alongside that comment, which makes me feel frustrated because I think the Temporal workflow SDK is very different. Architecturally, Temporal and DBOS are at two opposites of the durable execution spectrum. I'd love to understand what makes you think this work is a mere copy and paste. Would you be willing to share some more with me?
Thanks for the comment (author here). I wanted this post to focus on the Golang specific implementation, not dwell on the durable execution ecosystem at large.
With respect to context, I don't know that anyone invented "having their own context". Go interface are extendable and pretty much every major framework I know of implement their own context.
Would love to learn more about the gaps that offset you. We're constantly improving here ;)
Thanks, I didn't mean it as criticism, I guess my 5am brain thought the way it was worded almost came off as like "look at our unique idea" which was a pretty common pattern.
In review, I think it might have been the workflow versioning being strange, and the lack of any heartbeating/crash detection for longer running activities
Conductor is about enterprise features like automatic workflow recovery, alerting, or RBAC. The GUI is a nice to have -- but all your workflow data are in Postgres. You can access it very easily.
The offering would be enticing if some Web GUI features were behind a paywall. Separate "production" from "enterprise".
Right now the messaging is "you shouldn't use DBOS for production unless you are a paying customer", which is odd considering durable execution itself is a production-level concept. So we rolled our own in a few hundred lines of Python.
Durable execution has already been mentioned as the existing solution for this problem, but I would like to call out a specific pattern that DE makes obsolete: the outbox pattern. Imagine just being able to do do
send a()
send b()
And know both will be sent at least once, without having to introduce an outbox and re-architect your code to use a message relay. We can nitpick the details, but being able to "just write normal code" and get strong guarantees is, imo, real progress.
The hype is because DE is such an dev exp improvement over building your own queue. Good DE frameworks come with workflows, pub/sub, notifications, distributed queues with tons of flow control options, etc.
DBOS stores all the workflow metadata in postgres, which is readily queryable for observability. We've recently seen a user setup an entire Grafana dashboard to observe their numerous workflows.
A postgres server can host many databases, and multiple applications can use the same server. The same dashboard can be used to monitor them all.
With respect to recovery: A new Transact process will run a round of recovery at startup. Transact also exposes an admin server with a recovery endpoint.
For more elaborate scenarios, we have control plane options commercially available.
Temporal has a full-fledged UI where I can drill down into individual workflow runs and see the graph of activities, detail logs, retry counts, inputs and outputs, and so on. Temporal also has an API to introspect this without reaching into a database.
You can share a database server with DBOS, but it's common to give applications dedicated database resources (one Postgres cluster per app in different regions), meaning it won't work with DBOS unless you write your own federated control layer that can speak to multiple instances. Which is also not offered out of the box. Sharing one DBOS-specific server across all apps would introduce a single point of failure.
Again, I like DBOS, but right now the value proposition isn't that great given that Temporal has already nailed this.
I remembered reading about the DBOS paper a while back - https://arxiv.org/abs/2007.11112. Is this an evolution of that research work? If so, how did an OS for databases morph into a workflow orchestration service?
It is an evolution. The DBOS workflow orchestrator places a DB at the center of your application to handle most of the complicated state management problems.
Did you consider using NATS? While I haven't tried this deployment model, you can embed it in a go program as a library. If you wanted something really minimal this might be an option.
I use NATS to acheive this type of durable processing. It works well. Of course, idempotent code is needed but I don't think this can be avoided.
Is it possible for you guys to write a blog post analyzing the usage of the DB (reads, writes, what is stored for each workflow any events etc) to help users planning for scale to really understand what they are signing up.
The library seems fantastic but my team did not use this because at scale they believe that the number of DB reads and writes becomes very significant for a large number of workflows with many steps and that with PG vs Cassandra/ScyllaDB it would not be feasible for our throughput. I tried to convince them otherwise but it is difficult to quantify from the current documentation.
Good call. We'll see how to integrate it in our docs better.
The cost of DBOS durable execution is 1 write per step (checkpoint the outcome) and 2 additional writes per workflows (upsert the workflow status, checkpoint the outcome). The write size is the size of your workflows/steps output.
Postgres can support several thousands writes per seconds (influenced by the write size, ofc): DBOS can thus support several thousands of workflows/steps per second.
Postgres scales remarkably well. In fact, most org will never out scale a single, vertically scaled Postgres instance. There's a very good write up by Figma telling how they scaled Postgres horizontally: https://www.figma.com/blog/how-figmas-databases-team-lived-t...
Great project! Love the library+db approach. Some questions:
1. How much work is it to add bindings for new languages?
2. I know you provide conductor as a service. What are my options for workflow recovery if I don't have outbound network access?
3. Considering this came out of https://dbos-project.github.io/, do you guys have plans beyond durable workflows?
How does DBOS scale in a cluster? with Temporal or Dapr Workflows, applications register running their supported workflows types or activities and the workflow orchestration framework balances work across applications. How does this work in the library approach?
Also, how is DBOS handling workflow versioning?
Looking forward for your Java implementation.
Thanks
DBOS naturally scales to distributed environments, with many processes/servers per application and many applications running together. The key idea is to use the database concurrency control to coordinate multiple processes. [1]
When a DBOS workflow starts, it’s tagged with the version of the application process that launched it. This way, you can safely change workflow code without breaking existing ones. They'll continue running on the older version. As a result, rolling updates become easy and safe. [2]
So applications continuously poll the database for work? Have you done any benchmarking to evaluate the throughput of DBOS when running many workflows, activities, etc.?
In DBOS, workflows can be invoked directly as normal function calls or enqueued. Direct calls don't require any polling. For queued workflows, each process runs a lightweight polling thread that checks for new work using `SELECT ... FOR UPDATE SKIP LOCKED` with exponential backoffs to prevent contentions, so many concurrent workers can poll efficiently. We recently wrote a blog post on durable workflows, queues, and optimizations: https://www.dbos.dev/blog/why-postgres-durable-execution
Throughput mainly comes down to database writes: executing a workflow = 2 writes (input + output), each step = 1 write. A single Postgres instance can typically handle thousands of writes per second, and a larger one can handle tens of thousands (or even more, depending on your workload size). If you need more capacity, you can shard your app across multiple Postgres servers.
The durability guarantees are similar--each workflow step is checkpointed, so if a workflow fails, it can recover from the last completed step.
The big difference, like that blog post (https://www.dbos.dev/blog/durable-execution-coding-compariso...) describes, is the operational model. DBOS is a library you can install into your app, whereas Temporal et al. require you to rearchitect your app to run on their workers and external orchestrator.
:wave: Hey there, I'm working on the Go library and just wanted to confirm your suspicion:
"since Golang doesn't have decorators in the same way Python does, we still have to have code doing the kind of "manual callback" style I mentioned"
That's exactly right, specifically for steps. We considered other ways to wrap the workflow calls (so you don't have to do dbos.RunWorkflow(yourFunction)), but they got in the way of providing compile time type checking.
As Qian said, under the hood the Golang SDK is an embedded orchestration package that just requires Postgres to automate state management.
