Reactive access to incrementally parsed JSON for Python. Consume fields and array items as they arrive — no need to wait for the full payload.
jsontap builds a reactive node tree on top of ijson's streaming parser. You await scalar/object values and async for over arrays while the JSON is still being fed in, one chunk at a time. This makes it ideal for LLM token streams, chunked HTTP responses, or any scenario where JSON arrives incrementally.
FauxSpark is a discrete event simulator built with SimPy whose objective is to model the internals of Apache Spark. It lets you experiment with Apache Spark workloads under various cluster configurations without spinning up a real cluster – perfect for testing failures, different job schedules, or capacity planning to observe the impact it has on your workload.
The current version includes:
- DAG scheduling with stages, tasks, and dependencies (but perhaps, designing around "RDD" would've been the right call)
- Modeling the input, output, shuffle partition sizes as probability distributions.
- Automatic retries on executor or shuffle-fetch failures
- Single-job execution
- Simple CLI to tweak cluster configuration, simulate failures, and scaling up executors
This tool might be relevant to the following folks:
- Data & Infrastructure engineers running Apache Spark who want to experiment with cluster configurations
- Anyone curious about Spark internals
I'd appreciate feedback from anyone with experience in discrete event simulation, particularly regarding the planned features as well as from anyone who may find this useful to shape its development.
A walkthrough section in the README demonstrates how it can be used.
jsontap builds a reactive node tree on top of ijson's streaming parser. You await scalar/object values and async for over arrays while the JSON is still being fed in, one chunk at a time. This makes it ideal for LLM token streams, chunked HTTP responses, or any scenario where JSON arrives incrementally.