I think it would be better (and simpler) to just decrease the potential score according to the time the player takes to look at the numbers (time of first click minus time the level was first displayed).
Also, I played several games and ended with a different score each time; if time doesn't matter, what does the score come from?