I don't really know how this might or might not work. My gut says that since the .spec is meaningless without the sources, it's a Modification of the work and thus the spec, patches, and the resulting SRPM is definitely a Derived Work. But every time anything quasi-legal is being brought up here or anywhere else, it gets drowned in the arguments over what the meaning of the word "is" is, so I don't know how you can twist it, legally.
But then, the elephant in the room is that IBM may decide to give away only the GPLed SRPMs, and say a big fuck you to anything more permissive. People rallying against copyleft have made quite an impact, and the GPLed landscape is shrinking.