As for symlinks, heh, yes I do know they are not silly and actually extremely useful and yes I know how to use them. I simply think they are silly for managing dotfiles. If your intent is to version your dotfiles and have different 'profiles' across your systems, using the tools at hand -- well, in theory you can write a bunch of script to create the symlinks and 'manage' them as well -- but I just find /that/ silly.
My approach: git repo with a very simple setup script that creates symlinks to the dotfiles/dotdirs. It also manages some sensitive things like gpg/ssh keys.
Now, you're using Git submodules in your setup. This, yes, is a nasty and dangerous thing to deal with.