> Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

No need to act smug and superior, especially since nothing about OP's plan here actually precludes having all the nice things you mentioned, or even having them inside $your_favorite_enterprise_environment.

You risk coming across as a person who feels threatened by simple solutions, perhaps someone who wants to spend $500k in vendor subscriptions every year for simple and/or imaginary problems... exactly the type of thing TFA talks about.

But I'll ask the question.. why do you think safety, auditablity, access control, and knowledge sharing are incompatible with CLI tools and a specific choice of file system? What's your preferred alternative? Are you sticking with that alternative regardless of how often the work load runs, how often it changes, and whether the data fits in memory or requires a cluster?

> No need to act smug and superior

I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .

Another comment mentions this classic meme:

> Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.

A lot of industry work really does fall into this category, and it's not controversial to say that going the wrong way on this thing is mind-blowing. More than not being controversial, it's not confrontational, because his comment was essentially re: the industry, whereas your comment is directed at a person.

Drive by sniping where it's obvious you don't even care to debate the tech itself might get you a few "sick burn, bro" back-slaps from certain crowds, or the FUD approach might get traction with some in management, but overall it's not worth it. You don't sound smart or even professional, just nervous and afraid of every approach that you're not already intimately familiar with.

i repurposed the parent comment

"not understanding the scale of "real" big data was a no-go in my eyes when hiring." , "real winner" ect.

But yea you are right. I shouldn't have directed it at commenter. I was miffed at interviewers who use "tricky questions" and expect people to read their minds and come up with their preconceived solution.

The classic putting words in people's mouths technique it is then. The good old straw man.

If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]".

I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.

> that people don't try simpler and proven solutions FIRST

Well why don't people do that according to you ?

Its not 'mind blowing' to me because you can never guess what angle interviewer is coming at you. Especially when they use the words like ' data stack'.

> you can never guess what angle interviewer is coming at you

Why would you guess in that situation though?

It’s an interview, there’s at least 1 person talking to you — you should talk to them, ask them questions, share your thoughts. If you talking to them is a red flag, then high chances that you wouldn’t want to work there anyway.

I don't know why and this is why I said it's mind-blowing. Because to me trying stuff that can work on most laptops comes naturally in my head as the first viable solution.

As for interviews, sure, they have all sorts of traps. It really depends on the format and the role. Since I already disclaimed that I am not actual data scientist and just a seasoned dev who can make some magic happen without a dedicated data team (if/when the need arises) then I wouldn't even be in a data scientist interview in the first place. ¯\_(ツ)_/¯

Thats fair. My comment wasn't directed at you. I was trying to be smart and write an inverse of original comment. Where I as an interviewer was looking for a proper 'data stack' and interviewee responded with a bespoke solution.

"not understanding the scale of "real" big data was a no-go in my eyes when hiring."

Sure, okay, I get it. My point was more like "Have you tried this obvious thing first that a lot of devs can do for you without too much hassle?". If I were to try for a dedicated data scientist position then I'd have done homework.

Abstractly, "safety, auditablity, access control, knowledge sharing" are about people reading and writing files: simplifying away complicated management systems improves security. The operating system should be good enough.

What about his answer prevents any of that? As stated the question didn't require any of what you outline here. ZFS will probably do a better job of protecting your data than almost any other filesystem out there so it's not a bad foundation to start with if you want to protect data.

Your entire post reeks of "I'm smarter than you" smugness while at the same time revealing no useful information or approaches. Near as I can tell no one should trust you with any data.

"blows my mind" is similar in tone yes. But I wasn't replying to the OP. Further the OP actually goes into some detail about how he would approach the problem. You do not.

You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.

Honest question: was "blows my mind" so offensive? Thought it was quite obvious I meant that "it blows my mind people don't try the simpler stuff first, especially having in mind that it works for much bigger percentage than cloud providers would have you believe"?

I guess it wasn't but even if so, it would be legitimately baffling how people manage to project so much negativity in three words that are slightly tongue-in-cheek casual comment on the state of affairs in an area whose value is not always clear (in my observations, only after you start having 20+ data sources it starts to pay off to have dedicated data team; I've been in teams only 3-4 devs and we still managed to have 15-ish data dashboards for the executives without too much cursing).

An anecdote, surely, but what isn't?

I generally don't find that sort of thing offensive when combined with useful alternative approaches like your post provided. However the phrase does come with a connotation that you are surprised by a lack of knowledge or skill in others. That can be taken as smug or elitist by someone in the wrong frame of mind.

Thank you, that's helpful.

I already qualified my statement quite well by stating my background but if it makes you feel better then sure, show me the door. :)

I was never a data scientist, just a guy who helped whenever it was necessary.

He didn't say he didn't have any data background. He's clearly worked with data on several occasions as needed.

Are you trolling? Did you miss the part where I said I worked with data but wouldn't say I'm a professional data scientist?

This negative cherry picking does not do your image any favors.

Edit: for above comment.

My comment wasn't directed at parent. I was trying to be smart and write an inverse of original comment. Opposite scenario Where I as an interviewer was looking for a proper 'data stack' and interviewee responded with a bespoke solution.

"not understanding the scale of "real" big data was a no-go in my eyes when hiring."

i was trying to point out that you can never know where the interviewer is coming from. Unless i know interviewer personally i would bias towards playing it safe and go with 'enterpisey stack'