Sharing and using location are different things. Like I expect to be able to use my address book with my phone app, but the app sharing it seems like a completely different use case.
The chat interface shows the AI like another friend you chat with, and the share-location feature is specifically to share the location with friends, so its perfectly logical to believe that not sharing location includes not sharing it with the AI.
