If you take a reference with you on a journey through the solar system, that reference will stay accurate for an observer travelling with it and drift as seen from an observer that stayed behind. Both views are accurate. Locally, the amount of time that has passed for each observer was different. If you would correct for any observed drift, you would mess with your fundamental unit of time in your local frame of reference and get incorrect measures of time passed in your own frame.
