The file with all this location data is stored locally on your own phone or computer. Let’s say, hypothetically, your phone was stolen. They have your email, contacts, photos, text messages and documents. Plus access to any accounts that log in automatically.
Are you really worried about the fact that they also can get a general idea of where you were six months ago?