Sure, heritability is orthogonal to the question of whether a trait is determined by genetics.
But traits like IQ, height, and eye color are both (A) highly heritable and (B) substantially shaped by genetic factors. In casual online discourse, I believe that (B) is usually taken for granted, so it's glossed over, and when people say that any given trait is "heritable" they're also assuming that (B) is true for the trait. At least, I am guilty of that lapse.
And I take your point about language.