Patents in all fields clearly slow down innovation by allowing the holder to slow down or stop derivative works. The common counter to removing patents altogether is big pharmacy. I don't know how much sympathy I can have for big pharmaceuticals either with how much they get away with: even with unpatented drugs we see anti-competitive behavior with companies like Turing. Clearly the innovation is profitable enough that funding it publicly and then reaping the benefit publicly shouldn't be a problem, right?
The other objection is that it promotes secrets over open innovation, but I'm not sure that is too valid either. The current status quo gags profitable utilization of research for 20 years. Surely there's an acceptable alternative.