It's not so much as trouble with pointers as trouble with visualizing algorithms and data structures. Try implementing some moderately complex one in C. It reads like a monster. A dynamic language like Python would have a simpler implementation but at the cost of performance (Why do beginners need performance).

In retrospect I don't understand either. It's such a simple concept. But it really befuddled me for a while.

My suspicion is that since everything involved pointers in C, even things like string manipulation, it was just an extra layer of confusion in a sea of new to me stuff.

It’s possible to use pointers successfully without understanding them fully - (having a sense of) incomplete knowledge can frustrate the learning process. Spending time dropping down to assembler to learn a few variations of MOV or looking at compiler output would make pointers achingly clear.