Quoting is a much bigger problem than differentiating flags from paths, and on Unix that's a solved problem: the shell always handles quoting, and unix programs only expect a list of words which can contain arbitrary characters (except NUL, of course). If you invoke a program directly using the exec-family of syscalls, you don't need to quote anything.
Whereas AFAIU Windows programs expect quoted words to be passed via main(), and must parse them. The only benefit is that you can disambiguate a filename with a dash (or slash) based on whether it was escaped, but that's a quite rare necessity, and of course still relies on the caller quoting them. (Does cmd.exe quote pathname expansions?)
The dash problem is also solved as long as programs use getopt() or getopt_long(). First, getopt() knows which flags take arguments and which don't. Knowing this, if a flag takes an argument it doesn't matter whether the argument begins with a dash or not. One consequence is that there's no such thing as an "optional" argument to a flag when using getopt and friends, as that ambiguity cannot be handled cleanly. People who roll their own argument processing code just so they can get "optional" arguments to flags invariably don't appreciate the security problem.
Second, a double-dash (--) terminates the argument list. getopt stops consuming command-line arguments at that point, and optind will index the first non-flag argument. So if passing a list of filenames to a command, the correct idiom in Unix is something like, `foo -- /path/to/*`. Of course, that presumes that the foo is using getopt or getopt_long, or a compatible argument processing implementation. Fortunately the vast majority do.
Smart programmers should rarely if ever roll their own argument processing code. Any headaches (real or imagined) related to a mismatch between the semantics offered by getopt and what the application might want is usually dwarfed by the usability and security benefits of adhering to the system facilities.
On a related note, I've always disliked the way GNU's getopt and getopt_long permuted (reordered) argument lists. I have an inkling it could introduce needless security issues, though I haven't thought it through carefully.
