My experience of building accessible systems for low-literacy user groups is there's pretty much no such thing as dumbing down too much.

But the one big lesson is user test, user test and user test again. People will get tripped up by all sorts of stuff you can't see coming.

The good news is it sounds like your app could be mocked-up cheaply to get it infront of people to test your hypothesis before you've spent too much time building stuff.

There are umpteen agencies around that can help with research like this, including finding appropriate cohorts of people and even doing the user testing on your behalf. It sounds like it would be well worth the money.