willclarke.net

There is an absurd meme that has been circulating in conservative circles for the past year involving redistribution of wealth. I first encountered it by word of mouth from my dad, then a libertarian columnist from Florida, and now Fox News. It goes something like this:

Liberal college students are offered the opportunity to contribute their GPA/test scores to classmates who have a lower GPA - “redistribution of academic wealth.” Students balk at the suggestion that they should contribute the fruits of their labor to others that didn’t work as hard.

It’s not hard to see why this notion is catchy to conservatives. It makes young liberal college students look naïve. It makes academia look hypocritical. And it makes conservatives look sagacious and worldly. Of course, it’s also a really, really stupid analogy, for several reasons:

• The rich aren’t being asked to give all of their money to the poor, they are being asked to give a tiny fraction more of their income to the federal government to be used overwhelmingly for projects that benefit everyone, themselves included. Only a small percentage of the budget goes to the poor.
• The stakes are not the same - college students that fail tests don’t become homeless and starve on the street. They just change their major or drop out and move home with their parents.
• A college population is not analogous to the general population - college students are in college to learn, and have all been pre screened by application boards to assure they are capable of doing so. The reality is that the general population has a large number of people incapable of participating in the marketplace, either due to physical disability or inadequate labor market.
• The distribution of grades on a test does not match the distribution of income in America. On tests, scores are between 0 and 100, and the median score usually falls around a 70. Income ranges from $0 to several billion dollars, and the median is $44k (in 2003). If a professor consistently gave tests with a distribution like that, he or she would be fired for incompetence.

The analogy is bad, but that doesn’t mean we can’t fix it. Here’s a better one: let’s say that 25% of your class failed a test for one of the following reasons:

• They already took the class years ago but it has been a really long time and they no longer have the mental ability to pass the test (analogous to a Social Security recipient)
• They had a death in the family or some other reasonable excuse that prevented them from taking the test (a welfare recipient)
• They were unable to attend the class because there were not enough chairs in the classroom and the fire marshall would not allow anyone else in (an unemployment insurance recipient)

Now let’s say that all the students that made an A on the test had the ability to band together and contribute to these failing students’ grades. They would still maintain their A, just with slightly fewer points. And the students failing would receive the lowest possible grade that still allowed them to pass.

You would be an asshole for not saying yes. Even if a small percentage of those failing students were lying, or somehow cheating the system, wouldn’t you still contribute? Even if you had a B, wouldn’t you contribute a few points?

That is a better analogy to the situation that we find ourselves in today. The rich will not be materially affected by a slight increase in taxes.

With Apple set to announce iCloud in a few hours, I thought I’d write up my preliminary guesses about what the service will be. This is not based on any inside information I have, just my guesses based on how I think Apple thinks. I’m typing it up because, if I’m right, I want to be able to gloat.

I think iCloud is going to be more than just storage for iTunes. It will be storage for any app. Any iOS or Mac app developer will be able to leverage it and store user documents in the cloud.

Lets say you’re a developer that makes an image editing program. You can write a version for iPhone, iPad, and Mac, and they all will share the same document sandbox that lives in iCloud. A user can edit the image on the iPad, then immediately go to their Mac and keep editing the same image. This is probably why Apple released iWork last week for iPhone and iPod Touch - these apps will soon leverage iCloud for document syncing across all your devices.

There are some challenges that I am going to be interested to see how Apple addresses:

• Cost: how will Apple cover costs on iCloud? It could either a) eat them, considering how much sales iCloud will inevitably generate, b) charge end users by total iCloud usage, or c) charge App developers who want to take advantage of iCloud in their app. It’s this last option I find most interesting, maybe the could change the sales split from 70/30 to 60/40, or require them to use a subscription model which would mean more money for Apple. I think this scenario is more likely than many consider, mainly because of Apple’s proven willingness to shift cost burdens off of end users and on to App developers.
• File structure: currently there is no notion of file browsing or document hierarchies in iOS. In apps that have different documents, like Pages or GarageBand, there is just a list of all your documents, with no mention of “where” they are stored. As apps and documents get more numerous, this is going to become necessary.
• File sharing between Apps: Apps are going to need a way to pass around documents, or access documents used in other apps. Maybe you have multiple image editors and want to edit an image in one and then pass it on to the other. If they solve the “file structure” problem mentioned previously, this shouldn’t be a problem. But if they don’t then they need to come up with a way to share of expose documents to other Apps.
• Availability: will these features become available to developers immediately? I feel pretty confident that this is the direction that Apple is going in, but it may not be today. Step 1 may just be to get iCloud working for iTunes and other Apple products like Pages, and integration for third party apps may come later on down the line.

Hopefully we’ll get answers shortly.

Apple released a Q&A about the location data stored on the iPhone.

The iPhone is not logging your location. Rather, it’s maintaining a database of Wi-Fi hotspots and cell towers around your current location, some of which may be located more than one hundred miles away from your iPhone, to help your iPhone rapidly and accurately calculate its location when requested. Calculating a phone’s location using just GPS satellite data can take up to several minutes. iPhone can reduce this time to just a few seconds by using Wi-Fi hotspot and cell tower data to quickly find GPS satellites, and even triangulate its location using just Wi-Fi hotspot and cell tower data when GPS is not available (such as indoors or in basements). These calculations are performed live on the iPhone using a crowd-sourced database of Wi-Fi hotspot and cell tower data that is generated by tens of millions of iPhones sending the geo-tagged locations of nearby Wi-Fi hotspots and cell towers in an anonymous and encrypted form to Apple.

You read it here first. (And second. And third.) Okay, nobody wants to read a post about gloating, but I’m sure enjoying writing it.

The file with all this location data is stored locally on your own phone or computer. Let’s say, hypothetically, your phone was stolen. They have your email, contacts, photos, text messages and documents. Plus access to any accounts that log in automatically.

Are you really worried about the fact that they also can get a general idea of where you were six months ago?

Part of what is so intriguing about this iPhone location data dust-up is the question of why. Why is Apple storing this data on your phone? Are they trying to provide information to law enforcement? Do they have ideas for future apps that look at your location history? Do they want to provide it to advertisers?

Turns out, Apple told us nine months ago. Well, not us, but two of our esteemed Congressmen, Reps. Edward Markey (D-Mass.) and Joe Barton (R-Texas). In response to an inquiry about iPhone location data usage, they said (see pages 5 and 6):

To provide the high quality products and services that its customers demand, Apple must have access to comprehensive location-based information. For devices running the iPhone OS versions 1.1 .3 to 3.1, Apple relied on (and still relies on) databases maintained by Google and Skyhook Wireless (“Skyhook”) to provide location-based services. Beginning with the iPhone OS version 3.2 released in April 2010, Apple relies on its own databases to provide location-based services and for diagnostic purposes. These databases must be updated continuously to account for, among other things, the ever-changing physical landscape, more innovative uses of mobile technology, and the increasing number of Apple’s customers. Apple always has taken great care to protect the privacy of its customers.

Basically, when you can’t get a good GPS signal, your iPhone guesses based on WiFi and cell sites near you. Apple used to rely on other people’s databases of where WiFi and cell locations are, but now they do it in house. How do they do this?

To provide location-based services, Apple must be able to determine quickly and precisely where a device is located. To do this, Apple maintains a secure database containing information regarding known locations of cell towers and Wi-Fi access points. The information is stored in a database accessible only by Apple and does not reveal personal information about any customer.

Information about nearby cell towers and Wi-Fi access points is collected and sent to Apple with the GPS coordinates of the device, if available: (1) when a customer requests current location information and (2) automatically, in some cases, to update and maintain databases with known location information. In both cases, the device collects the following anonymous information:

• Cell Tower Information: Apple collects information about nearby cell towers, such as the location of the tower(s), Cell IDs, and data about the strength of the signal transmitted from the towers. A Cell ID refers to the unique number assigned by a cellular provider to a cell, a defined geographic area covered by a cell tower in a mobile network. Cell IDs do not provide any personal information about mobile phone users located in the cell. Location, Cell ID, and signal strength information is available to anyone with certain commercially available software.
• Wi-Fi Access Point Information: Apple collects information about nearby Wi-Fi access points, such as the location of the access point(s), Media Access Control (MAC) addresses, and data about the strength and speed of the signal transmitted by the access point(s)…

The consolidated.db is the file is how they store that data locally on the phone¹. The cell tower information is in the CellLocation table. The WiFi access point information is in the WiFiLocation table. There is also a table called CdmaCellLocation - for me it’s empty, but I’d be willing to bet that people with a CDMA (Verizon) iPhone will find if chock full of cell locations. As I’ve been saying all along, these tables are of the locations of the access points, and when you’re in those areas again, your phone can look at what kinds of signal strength is has with them to determine it’s own location. It does this because this takes much less battery than locking on to a GPS signal, and although the accuracy isn’t as good, for many applications it’s good enough.

I suspect the data in these tables is probably two way - you upload some anonymized data to Apple about cell and wifi locations that you have located, and Apple may send some back that other anonymous people have located. That would explain why we all get weird data from time to time for places we haven’t been - for some reason Apple pushed out cell locations to us because an App requested data about that location. Or maybe it knows about cell locations that have been moved and wanted everyone to have accurate information. I know this is just a hypothesis but it’s a much better explanation than I’ve heard from anyone else.

Now in this document, Apple says that when they send the approximate locations of the cell sites and WiFi access points, they also are sending the GPS coordinates (if available). However, there is no evidence that I can find that they are storing those coordinates on the phone. The coordinates are sent to Apple, which stores them in an anonymous database on their end. They aren’t in the consolidated.db anywhere - at least not that I can find. They’re definitely not in the CellLocation table that iPhoneTracker uses, because GPS coordinates tend to be very precise and nothing in that table has an accuracy of closer than 500 meters.

Why store a whole year’s worth of data? Because that’s the whole point! In order to find your location without using GPS, you want a huge database of where cell towers are and what their location is. Now Apple could probably do this in a way that better protects your privacy - they could munge the timestamps more effectively, or take all the data whenever you sync and replace it with more reliable cell location data from their servers that covers the areas that you frequently travel. Kind of like Genius syncing in iTunes; they look at where you’ve been, and give you reliable cell and wifi locations based on those locations. Smart, elegant, and secure. Why they do it this way is unclear, and surely they will provide an additional clarification now that more people are asking questions.

1. To clarify, this is my opinion. Apple has not said exactly what the purpose of the consolidated.db file is, and until they do, we won’t know for sure. I strongly suspect this is the purpose of the file, for reasons mentioned both in this post and other posts on my blog.

Dan Goodin put together a pretty good article on the iPhone tracking issue, “No, iPhone location tracking isn’t harmless and here’s why.” I have to take issue with how casually he disregarded my assertion that what is being stored in the database is cell locations and not the device’s location, but he was trying to show both sides. After all, Alasdair Allan and Pete Warden are researchers and I’m a blogger.

However, I still think I’m right. And in Dan’s defense, he hadn’t read my most recent post when he put the article together, so he hadn’t seen my evidence regarding the non-overlapping “halos” in the data points.

In the article, Alasdair and Pete were indirectly quoted as saying:

Some of the extracted databases they examined plotted literally thousands of unique coordinates in a small part of a single city. It’s almost impossible that there could be that many corresponding nodes in such a confined area, they said.

I believe this is because what the iPhone is doing is guessing the network nodes’ position using its GPS. They are so many unique ones because they are different guesses at different times.

I think the distinction of exactly what data the iPhone is collecting is important, because it speaks to both what level of detail we’re talking about and what Apple’s motivations are for collecting that data. I hope this is settled by Apple, and I still believe the evidence is on my side.

Where I agree with Dan, Alasdair, and Pete is that it isn’t, as The Register puts it, “harmless.” Alasdair said it can place you “a bit above the block level” and I’d say it’s more like neighborhood level, but yes. It can place you at around that level. Being able to access that level of data on someone is not good. And even in the cases where the data is really inaccurate, it’s bad because law enforcement is using this inaccurate data to prosecute or question potentially innocent people!

Another thing I noticed, however, is that it is not always collecting this data. In my 1+ years of using iOS 4 and collecting data, I only have 1112 distinct timestamps in my data. That’s about three times a day - hardly enough to determine my jogging route or reveal to my girlfriend which of her friends I’m cheating on her with. (It’s a joke! I love you baby!)

(Previously: Apple is not “recording your moves”)

A lot of people have said that while the data from my backcountry bike trip is interesting, there is still a large security concern for people that live in cities. Since the urban density of “cell phone towers” (or more realistically, wireless network nodes) is much greater, couldn’t someone who stole your iPhone find you in a city much more easily?

From the data I’m seeing, no. I have been looking through the table of my data more thoroughly and what I’ve found is interesting. It doesn’t log one data point at a time - it will log a couple dozen data points all at once. For example, here is my data visualized on a graph, for the timestamp of April 3rd at 5:15:25.865 PM:

Ignore the lines that are drawn - the utility I used to convert the points to something readable by Google Earth adds them. All the points were added to the file with the exact same timestamp. What gets interesting is when you start looking at the data behind these points. Take a look at the following two screenshots:


Note the Horizontal Accuracy of the two points. This is a measure, in meters, of the confidence in that location - like when you load up maps and it shows a blue “halo” around your location indicating the area you may be in. The first point has an accuracy of 549 meters, the second 500 meters - and they are over 2000 meters from each other. Now, if these data points are supposed to be where you are, then their horizontal accuracies should all overlap on some point that reveals your actual location. But they don’t - which is why I believe they are locations of nearby cell sites, and the horizontal accuracy is a measure of how confident it is that the cell is there.

“But it’s still very revealing!” you must be thinking. After all, if it’s cell sites around you, you must be right in the middle of that circle. Fortunately for my privacy, no. At 5:15 PM on April 3rd I was in the bottom left of that circle, over a block away from the nearest dot on the map. I had just finished a 155 mile bike trip and was pretty happy to be sitting there, not moving.

A vast majority of the data in consolidated.db is in clumps like this. I grouped all the data by timestamp and found that 68% of the groups (and 96% of the points) were in clumps of more than 12 points.

Of the 40k valid entries in my CellLocation table, there were 106 entries with just a single timestamp. I looked at several of these, thinking I had perhaps found something that would give a close indication of my actual location. Still, no. One was at 1:30 AM, and it was 700 meters, or seven blocks, away from where I lay asleep, despite claiming an accuracy of 500 meters. Interestingly, this was in downtown Seattle, where there were tons of other cell nodes that it could have logged. Why did it pick just this one? Who knows.

Now there may be individual points in your consolidated.db file that happen to be the exact same as your location at the time. I’m sure there are some in mine. But that that is coincidental - the phone seems to be logging locations of nodes on the wireless cell network, and their presence in the file mean that you MAY have been near them. The Las Vegas mystery is still confounding me, but it only further proves the point that this data should not be relied on for any kind of forensic analysis.

Fire up your iPhoneTracker and zoom in on southern Nevada. Remember that fun trip?

I don’t either. Neither does my coworker, who apparently took the exact same trip around the same time, late June. My bet is that your iPhone has these same points. Did we all drink that much? Was there a convention of amnesiac iPhone users?

Data from my iPhone	Data from @bp1222’s iPhone

This is what concerns me about the idea that these data points are your location. If forensics experts are supposedly using this file to pinpoint suspects’ locations and help prosecute them for crimes, shouldn’t we be SURE that what’s in these tables are really location data and not cell phone tower location?¹

Here are the data points from my phone, from Vegas:

Timestamp	Latitude	Longitude	Horizontal Accuracy
299183575.518057	36.42393219	-115.39707958	2549.0
299183575.518057	36.38181227	-115.37701904	2622.0
299183575.518057	36.46913003	-115.44647067	1030.0
299183575.518057	36.46861606	-115.44789755	2837.0
299183575.518057	36.36917614	-115.35957294	2782.0
299183575.518057	36.36333626	-115.35199129	1530.0
299183575.518057	36.36041134	-115.34166538	2551.0
299183575.518057	36.36049932	-115.33027505	2353.0
299183575.518057	36.49515825	-115.49839264	2872.0
299183575.518057	36.49757117	-115.49900907	2709.0
299183575.518057	36.3041771	-115.40962135	500.0
299183575.518057	36.30460888	-115.41722583	697.0
299183575.518057	36.32789695	-115.29256647	500.0
299183575.518057	36.32790172	-115.29153209	500.0
299183575.518057	36.32790309	-115.29110014	2795.0
299183575.518057	36.30432605	-115.33195531	500.0
299183575.518057	36.30418097	-115.33188027	500.0
299183575.518057	36.30395781	-115.33031117	500.0
299183575.518057	36.30274063	-115.33094429	1315.0
299183575.518057	36.31804084	-115.29408824	1912.0
299183575.518057	36.31821024	-115.29372918	500.0

The “Horizontal Accuracy” field is what the iPhone uses to give the radius of confidence for a point on a map. It is in meters. When you see the little blue dot with a circle around it in Maps, that’s the Horizontal Accuracy. The Timestamp is in seconds since January 1st, 2001.

Why would there be so many points in there with a relatively high accuracy, all registered at the exact same time? If my theory that the data points are cell phone towers is correct, then maybe it gets these records from the network? Maybe some bug with AT&T sent out the approximate location of cell towers in Las Vegas to all our iPhones and they all diligently cached them away? I am going to do some more digging but this certainly is getting interesting…

1. I have frequently been saying “cell tower” location. This terminology is anachronistic, usually what your phone is communicating with is not a “tower” but a wireless network node on the side of some building, more akin to the WiFi base station in your house.

A pretty sensational piece was published on O’Reilly Radar this morning titled “Got an iPhone or 3G iPad? Apple is recording your moves.” From their article:

Today at Where 2.0 Pete Warden and I will announce the discovery that your iPhone, and your 3G iPad, is regularly recording the position of your device into a hidden file. Ever since iOS 4 arrived, your device has been storing a long list of locations and time stamps. We’re not sure why Apple is gathering this data, but it’s clearly intentional, as the database is being restored across backups, and even device migrations.

They also released an application, iPhoneTracker that you can use to browse the data. In the FAQ they ask:

Why is Apple collecting this information?
It’s unclear. One guess might be that they have new features in mind that require a history of your location, but that’s pure speculation. The fact that it’s transferred across devices when you restore or migrate is evidence the data-gathering isn’t accidental.

I don’t mean to denigrate the work they’ve done. Nor do I mean to imply that there aren’t security concerns with this file. But after looking at the raw data that the iPhone stores, I want to point out that it seems that they are technically incorrect. Apple is not storing the device’s location, it’s storing the location of the towers that the device is communicating with.

How do you know?

When I first read the article, I was both concerned and intrigued. Obviously storing a ton of detailed location data for a period of a year is a concern, but I was also interested in what kind of data my phone had on it. After all, I just finished a fun weekend bike tour, and it would be neat to see the detailed route we took. I was using the Maps application a lot and it was giving me very accurate information at the time, so I figured surely there would be at least portions of the route that would be very detailed.

When I used their application, I was disappointed. They throw all the data points on a grid heat map that doesn’t give you much detail. They do this intentionally; they want you to see that it stores the data while not providing a tool that can be easily exploited by others. But that wasn’t good enough for me.

So I followed their instructions to get at the raw data. I extracted the location database from my last iPhone backup, used SQLite to limit it to just the data points during my bike trip and data points that were within a certain level of accuracy, exported this data to a CSV file, used an online tool to convert this to KML, and imported my route into Google Earth.

What I found was disappointing. Almost all the points were way off. Here is a map that is generated from Google Earth; the red points are the ones pulled from my phone. The blue line is the route we actually took to get to Long Beach Island and the Orange Line is the route we used to leave.

We did not do anything on the island at all. We went straight ahead to the beach, took a picture, and then turned around and went back.

Now keep in mind, I was using the iPhone’s GPS constantly, and it was giving me very good readings. It had me exactly along Main St near Tuckertown, where I actually was. I took a photo on the beach, and it geotagged me with almost the exact location. Yet none of these very accurate points were in the dataset that is stored on the phone. Instead, the datapoints were all along the highway, where I definitely wasn’t, or in nearby towns, which we biked around.

The datasets also contain accuracy information. I am limiting the results to only data points that claim to be accurate within 1500 meters. However most of them are much farther than that from my known route - for example, the ones on the Garden State Parkway are more than twice that distance away. And those individual points claim an accuracy of less than 1000 meters.

The only thing that makes sense is that the iPhone is actually storing the locations of the cell phone towers that it communicates with. My guess is that the iPhone uses this data to help it locate cell towers if it is in the same location again in the future.

Other Indications

Here is other evidence I’ve found to confirm this theory:

• The name of the table this data is stored in is CellLocation. Other tables in the DB include WifiLocation and CdmaCellLocation. These names strongly indicate that they are used to store locations of access points, not of the device itself.
• I looked at the data from a friend’s phone who traveled recently to Germany. While there, he used his iPhone roaming on their 3G network. He took a lot of photos, many of them geotagged correctly, meaning a lot of location information was stored. But there were no points stored in the CellLocation table in Germany. So it’s probably only collecting data about the location of cell towers on the phone’s primary network (meaning AT&T). UPDATE: I was wrong - he had it in Airplane Mode with WiFi on in Germany, so it was geotagging based on WiFi positioning. I’ve talked to another friend who confirmed that it DOES collect data abroad.
• None of the points in the table have an accuracy of more than 500 meters. That’s probably the best a phone can do to place an individual cell phone tower, but triangulation with multiple cell towers can place a device on a map much more accurately.
• Note the way-off points on the above map on Long Beach Island. When crossing the bridge, I should have had incredible accuracy since I was over water. It should have placed me right on the bridge. Instead, what I had was incredible accuracy to see the far-away towers. It was alternating between recording towers behind me and towers in front of me.

Who cares? This still means there is a log of where I’ve been!

Yes, it does. Like I said originally, I’m not saying there aren’t privacy concerns here. I just think we should be honest about what they are. According to the Huffington Post, “your iPhone or iPad is keeping a record of every step you take.” The Guardian claimed the phone “saves every detail of [where you go] to a secret file on the device.”

The data that is exposed basically reveals which city you were in at a given time. Nothing more specific than that. It can’t tell what house you live in, it can’t tell what route you jog on, nothing like that. It’s misleading because people know their phone can locate them within several meters of their actual location. But if all it’s showing is cell tower location, you could be anywhere within a 2 to 3 mile radius. That’s several orders of magnitude less precise.

There is also the issue of motivation. If Apple was collecting data on the user’s location, for use in future apps that “require a history of your location” as the researchers speculate, then that is pretty creepy. Nobody would want that, and it would be a huge breach of trust to collect data with that level of specificity. But if they are just collecting data about cell tower locations, perhaps because it helps you get a better signal in the future or improve network reliability, and they do so in a way that doesn’t reveal very much about your location, that’s certainly less creepy.

UPDATE: I’ve added a new post with a bunch of details about how the tracking works in an urban environment, and still believe what it’s tracking is cell nodes and not your location. Check it out

If you’re viewing this, it’s probably because I emailed you about a place to live and you are stalking me before replying. It’s cool. When you reply, I’ll probably use your email address to do the same to you.

Feel free to read my posts, but I thought I’d just preface this with the fact that I don’t post to my blog very often and when I do it’s about random things that probably don’t “define” me very well. Also, the “About” section is very old and for some reason my blog software won’t let me edit it. I could take time to figure it out but I don’t really care. Enjoy the investigation! And email me back, damnit.