The conservative message machine is determinative, and they would find something to effectively raise a storm about: immigration, inflation, etc. If Biden cut inflation, they would have demonized him regarding employment. Or just make something up - they can say anything at this point, and the Dems and others have made themselves helpless. They will always find something - Biden and Dems were being called pedophiles in 2000, the election was stolen, etc.
Remember that the GOP stopped immigration reform in Congress for many years, including killing the agreed-upon bipartisan immigration reform bill at Trump's behest during the election. If your theory is correct, that would have disqualified the GOP among those voters.