It quickly became obvious to me that there was a flaw in the method: If the candidate didn't get a chance to answer all the questions, their score would be really low. This is through no fault of their own: they don't know they're on question 4 of 12, and there's only 15 minutes left in the interview. They think they're just answering the question thoroughly and thoughtfully. So, it left me in the position of having to do a "lightning round" style assault at the end of the interview, to make sure they at least got a chance to answer everything, and not be torpedoed by the scoring system.

The solution to this is likely one of:

* ask everyone different questions, or different numbers of questions

* ask fewer questions to everyone, and potentially not cover some areas.

* timebox each response, and cut them off to say "time's up, let's move on".

* ignore questions they didn't answer, which means deciding on a candidate you know less about.

* increase the length of the interview.

I think I'd be happy asking everyone a different set of questions, but companies are apparently scared about the bias that could introduce (at any rate, we were instructed not to do that). Of the remaining options, the best one seems to be to increase the length of the meeting.