The bigger concern is the overhead of the Linux OS in terms of interacting with it vs just flashing a microcontroller... but linux lets you run an SSH and FTP servers and wifi and a debugger on the thing easily.
So you get easier access to remotely playing with the programming of your gizmo, but you have the OS in the way of just talking to hardware in real-time. I haven't done projects like that since my undergrad, does it really make that much difference?
To me the big deal-breaker would probably be if the thing I was building was battery-powered.