Hacker Newsnew | past | comments | ask | show | jobs | submit | zobeirhamid's commentslogin

This is insanely fast! How many records is it processing?


38,118,782 as of right now. It was a massive undertaking on the ops side of things. Imo, the most impressive bit is that we eventually made our ingest fast enough to ingest the entire dataset in <24hrs. In February, it took two weeks.

Speeding up ingest made experimenting with different indexing strategies a lot more viable.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: