People (and, really, adult mammals that aren't humans of European descent) mostly are lactose intolerant, which is a non-immune issue (and thus not an allergy.) Milk allergies (allergies to one or more of whey, cassein, or other milk proteins) are a different issue, but also a real thing.