The point of patents (at least in the US) is to benefit society. The question is if granting a monopoly on ideas is actually benefiting society. Thomas Jefferson wrote:
"That ideas should freely spread from one to another over the globe ... seems to have been peculiarly and benevolently designed by nature, when she made them, like fire, ... incapable of confinement or exclusive appropriation. Inventions then cannot, in nature, be a subject of property."[1]
He then goes on to point out that patents "may or may not be done, according to the will and convenience of the society, without claim or complaint from anybody".
Given the abuse of the patent system and little evidence of it benefiting society, it's hardly an "extreme position" to be against patents.
1: http://press-pubs.uchicago.edu/founders/documents/a1_8_8s12....