In my experience the only time I worked on a big data project was the public Twitter firehose. The team built an amazing pipeline and it did actually deal with masses of data. Any other team I've been on were delusional and kept building expensive and overcomplicated solutions that could be replaced with a single Postgres instance. The most overcomplicated system I've seen could not process 24hrs-worth of data in under 24 hours... I was happy to move on when an opportunity presented itself.