In the case of programming languages, yes, it's relative to the difficulty of other PLs. I've learned many over the years, and found Rust by far the hardest (it's the only one that defeated me). And it's not the most different from others I've learned - lisps are far further from the common languages than Rust is.
> I suspect a major reason I found it easy was because I didn't try to solve lifetime problems,
Well yes anything's easy if you skip the hard bits. Learn C without using pointers.
I personally didn't find ownership & borrows the hardest part - in my case it's the great complexity of many of the commonly used libraries. Rust's complexity bleeds out into the entire ecosystem (a cultural thing).