Given that GPT-4 is purported to be not bigger than GPT-3, but simply a better trained one or with some minor modifications, I'm inclined to believe that we are almost there and existing hardware does most of what is needed, even though better hardware would make things cheaper.