Global lat/long coordinates are defined in terms of coordinate systems like WGS84 or ITRF2020, which are themselves the result of relative measurements between reference stations.
The earth's crust floats on top of liquid rock. This matters at relevant length and time scales; in most places, these effects alone are on the order of millimeters per year. One reason why it's better to use NAD83 over WGS84 in North America is that NAD83 latitudes and longitudes move with the North American plate.
Positions _are_ relative, and the closer you can put your datum, the less drift you'll accumulate.
