It's more nuanced. If you've included GPL code
and modified or redistributed it, then you either have to comply with the GPL to have permission for that use, or you've potentially committed a copyright violation.
To comply with the GPL you only need to publish the one specific snapshot of source code you've combined and redistributed with GPL code, but you don't need to permanently relicense your project if it doesn't contain any code you don't have rights to use. The "tainted" version will be granted as GPLed forever, but other earlier or later versions that don't use any GPL code don't have to.
Or you can go the copyright way, and claim it wasn't a copyright violation (because it was a fair use, or non-copyrightable code) or settle the matter in whatever way the law lets you get away with.