undefined | Better HN

0 pointsshikharja6y ago0 comments

ska, our aim with the challenge was to allow candidates to not be biased by a fixed outcome and try to solve the problem as they would solve any real data science problem.

This meant we couldn't automatically score/rank a candidate's solution. We do provide them with an evaluation metric in the problem description (Mean Absolute Error). Here is scoring rubric we provide to the interviewers when they review the submission - https://d.pr/i/hNYY0u

Would love to hear more opinions on our scoring rubric

0 comments

3 comments · 1 top-level

ska6y ago· 2 in thread

That's useful information, thanks. I'll add some thoughts on the rubric.

[edit]. Initial thoughts:

- "data wrangling" scoring difficult given this task - more weight to "rationale", that's more important the "performance", here.

- not enough focus on communication capabilities

- really need something on validation

- "proficiency" measure you use is pretty much impossible to accurately evaluate from your example question

- way too much weight to modeling section overall

shikharjaOP6y ago

By 3rd bullet, you mean Data Validation?

ska6y ago

both data validation and model validation are important.

Of course more abstractly you want to validate the entire process - but I was referring to those two, as it is hard to see how to address the latter in a format like this.

j / k navigate · click thread line to collapse