The question I have is how can you be sure you're testing for something meaningful? By which I mean that testing new college grads on knowledge of Ruby is going to favor people with experience in Ruby and not necessarily those with a sound grasp of data structures and in the long run. Now if all you care about is the ability of someone to code up Ruby at a beginner level, then it's a good test, and in three years you may have someone who still codes Ruby at a beginner level. But if you want someone who can contribute at a higher level over the long term, then maybe taking an advanced data structures elective over a Rails elective is more important than the equivalent of a few weeks of on the job training.
Going further, a test to filter out candidates is a social act not a technical one. Some people, and perhaps a higher percentage of strong candidates who have more options, may forgo the test, or if it is easier than another company's think less of your company. Unless you're Google class as a resume builder, your pool of candidates will probably drop simply as a result of increased friction in the application process.
Sure you may get the desired effect of reducing the number of resumes on your desk, but you could do that by simply flipping a coin for each one and ignoring any that come up tails. And as crazy as that sounds at least then you would know that you were not filtering in a way that was having counter-productive results.
Good luck.
