The points are (also) an estimating mechanism. They let you estimate a new task in points (effort measurement) instead of hours (time measurement), and then use history to predict a likely timeframe based on your typical rate.
The extra layer of indirection helps to account for uncertainties in the task, imprecision in the estimate, and chaos (in the scientific sense) in how long individual tasks take relative to aggregated historical metrics.