Feel free to be horrified: a data scientist who doesn't understand where and why to use unix command line tools for data preparation and ETL is about as useful to me as one who doesn't understand the conditions where a t-test breaks down or what a ROC curve is.

Generally speaking, people like this have never actually dealt with large data sets, never dealt with issues involved with installing "unapproved software" on a machine (ridiculously common in The Real World), has probably never cleaned a dirty data set (what do you do when your giant csv is formatted in a way that Wes McKinney didn't think of?), and will in a senior role be a long term liability for a data science team that works on serious problems. Sure at one point I didn't know about them either: I wasn't a senior data scientist then. I submit that if you don't know about them and haven't actually used them, you aren't either.

So I think this is a perfect example of what happens in tech interviews.

OP is using knowledge of a specific technology as a heuristic for "has experience in role x"

But this always makes me wonder, couldn't you see that experience from a resume? If the candidate filled a data science role at somewhere reputable for 3 years, and you verify that they successfully filled that role, why rely on that heuristic?

As you say testing for the specific technology, when it can be learnt in 10 minutes, does not seem logical.

> Databases are generally known for their good performance

If the data is on a filesystem, then sed, grep and cut pipelines will likely be your fastest option (Yahoo! processed petabytes of logfiles for decades that way.)

If the data is already inside a database table and indexed well, that could be fast enough. But generally speaking, the ETL is often a bottleneck. And DBAs are $$$$ compared to "the UNIX way."

Source: DBA