You don't lose it completely, though. And it's easier to relearn and get back on track than to learn it the first time.
I learned C like ten years ago and then I moved on to PHP and then to JavaScript. I'm sure I forgot most of it. But thanks to that, I'm now learning Golang and when it came to pointers it clicked almost instantly.