Conflating the two is why some people have trouble understanding why religions like Buddhism and Christianity seemed to tolerate so much inequality and violence; or more generally just assumed people writ large were historically more callous and uncaring than today.
Arguably one of the downsides, though, to a focus on rights vs intrinsic value is that rights are typically couched in materialist terms. Most of the time that's probably for the better, but sometimes maybe not.