Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This must be common or we worked at the same company, seen this exact pattern.

It went like:

App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka

The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.



Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: