No, Turing complete means that whatever we are talking about can solve all the problems that a Turing machine can solve. A language that has only jump statements and absolutely nothing else is not Turing complete even though its programs can run forever.
We only equate running forever with Turing completeness in contexts where something would be Turing complete iff it did not miss an instruction or something that would allow it to run forever (consider a language that has addition, subtraction, if statements, etc but no jump or function) or if it has something that does not allow it to run forever (such as a type system, consider simply typed lambda calculus).
Our modern computers do not have unbounded tape nor are they infinite state machines - with their limited memory and finite amount of states they can't solve all the problems that a Turing machine can. They are basically glorified finite state machines with a lot of states.
