Points don't have to track time intervals to give you a sense of progress. Averages are a powerful thing, hopefully management is focused on long term graphs not getting het up about the number of hours difference between your tasks for two similarly pointed stories.
Oh and Scrum makes no requirement re using story points - it only requires that backlog items be estimated. We estimate tasks in hours and stories in points for instance, Scrum doesn't care. All our Product Owner shows to the stakeholders is a burnup with just numbers on the x axis.