The other group are very religious who don't want their children learning about evolution or many other secular things.
The only real issue I have with homeschooling in the US is that regulations vary wildly by state. Some states have so little enforcement that it is possible to teach a child essentially NOTHING by the time they are 18 and face no punishment for ruining that child's life.