mongopuff is a logical replication service for MongoDB and turbopuffer that I've been working on as a side-project. I was prototyping with turbopuffer at Huntr when I came across the puffgres project by a24 films. I wasn't able to find an equivalent for MongoDB, and after watching the talk the puffgres team cited on toolmaking I decided to start one myself.
In MongoDB, the fastest path to logical replication is to listen to the native change stream. Under the hood the change stream is a tailable cursor that reads events from the oplog. The oplog is an internal capped collection that acts as an append-only replication log. Each oplog event contains a resume token that a consumer can use to resume the stream in the event of a disconnection.
Delivery Guarantee
mongopuff was designed to guarantee at-least-once delivery, and strive for effectively-once delivery. Each record persisted to turbopuffer is appended with the clusterTime from the change stream event, which is used with turbopuffer's conditional writes to ensure the latest data is not overwritten. The change stream cursor is only advanced after the event is durably written, preventing data loss in case the service crashes or disconnects unexpectedly.
Oplog Exhaustion
MongoDB's oplog is a capped collection, which means it grows until a certain size and then new entries evict older ones. By default, MongoDB allocates 5% of available disk space as the max size for the oplog collection. Therefore, it's possible that if the consumer cannot advance its cursor fast enough the resume token can get evicted from the oplog forcing the stream to end. With no resume checkpoint, this forces the operator to run a potentially expensive backfill operation to re-sync the collections.
In most circumstances, this should never be a concern in mongopuff. Each consumer runs concurrently in its own goroutine and can realistically process hundreds of thousands of events per second without issue. The failure point lies in potential write latency to turbopuffer. mongopuff blocks event consumption while waiting for the turbopuffer write to respond successfully, to confirm delivery before advancing the change stream token. Against turbopuffer's reported p50, p90 and p99 write latencies, mongopuff can theoretically handle the following throughputs.
| Percentile | Write Latency | Throughput |
|---|---|---|
| p50 | 165ms | 6,113 events/sec |
| p90 | 248ms | 4,082 events/sec |
| p99 | 850ms | 1,197 events/sec |
However, in a scenario where the writer experiences sustained degradation, the distance between the cursor and the oplog head can balloon quickly. To demonstrate this, I set up a MongoDB instance with a 10MB oplog and a writer inserting 3,000 documents per second. mongopuff ran in CDC mode with a simulated write latency of 5ms - a pace the service can handle comfortably. After 15 seconds, I increased the latency to 1,000ms to simulate a degradation. The cursor was evicted approximately 14 seconds later.
A Durable Consumer
Consistent thousands of writes per second would exceed my own use case for this tool, but the failure requiring a full backfill warranted addressing the issue. The service can effectively read change stream events as fast as they are emitted, so the solution is to write change stream events to some internal buffer without blocking for the writer. In memory would run the risk of a crash losing events after the token was advanced, so some level of persistence is required. In keeping mongopuff simple and self-contained, the solution is a spool directory written to disk functioning as a write-ahead log.
The disk write is on the order of microseconds, so we always keep up with the oplog head — at the cost of disk space. The previous benchmark, run with the spool directory enabled, shows the lag stays ~0 and during the degradation the spool directory size grows linearly. As soon as the lag resolves, the spool directory is quickly flushed and disk usage goes back to 0.
If the mongopuff project is interesting, you can find it on Github. Contributors or feedback are gladly welcomed - if you end up using mongopuff, please reach out!