That's also, of course, true of those who send their kids to private school. I think that there's an excellent argument for vouchers which enable parents to allocate the funds that would have gone to a public school to a private school. There are some dangers (because once something is private, it can be advertised and the system can be gamed), but overall I think this is far better than our current 51%-of-parents-decide model.
I don't feel that it'd be a good idea to give homeschooling parents a similar voucher, but only because it's too ripe for abuse: an abusive parent could take the voucher and spend it, and not bother to educate his kids. I don't know how likely it is, but it's worrisome nonetheless.