willclarke.net

There is an absurd meme that has been circulating in conservative circles for the past year involving redistribution of wealth. I first encountered it by word of mouth from my dad, then a libertarian columnist from Florida, and now Fox News. It goes something like this:

Liberal college students are offered the opportunity to contribute their GPA/test scores to classmates who have a lower GPA - “redistribution of academic wealth.” Students balk at the suggestion that they should contribute the fruits of their labor to others that didn’t work as hard.

It’s not hard to see why this notion is catchy to conservatives. It makes young liberal college students look naïve. It makes academia look hypocritical. And it makes conservatives look sagacious and worldly. Of course, it’s also a really, really stupid analogy, for several reasons:

• The rich aren’t being asked to give all of their money to the poor, they are being asked to give a tiny fraction more of their income to the federal government to be used overwhelmingly for projects that benefit everyone, themselves included. Only a small percentage of the budget goes to the poor.
• The stakes are not the same - college students that fail tests don’t become homeless and starve on the street. They just change their major or drop out and move home with their parents.
• A college population is not analogous to the general population - college students are in college to learn, and have all been pre screened by application boards to assure they are capable of doing so. The reality is that the general population has a large number of people incapable of participating in the marketplace, either due to physical disability or inadequate labor market.
• The distribution of grades on a test does not match the distribution of income in America. On tests, scores are between 0 and 100, and the median score usually falls around a 70. Income ranges from $0 to several billion dollars, and the median is $44k (in 2003). If a professor consistently gave tests with a distribution like that, he or she would be fired for incompetence.

The analogy is bad, but that doesn’t mean we can’t fix it. Here’s a better one: let’s say that 25% of your class failed a test for one of the following reasons:

• They already took the class years ago but it has been a really long time and they no longer have the mental ability to pass the test (analogous to a Social Security recipient)
• They had a death in the family or some other reasonable excuse that prevented them from taking the test (a welfare recipient)
• They were unable to attend the class because there were not enough chairs in the classroom and the fire marshall would not allow anyone else in (an unemployment insurance recipient)

Now let’s say that all the students that made an A on the test had the ability to band together and contribute to these failing students’ grades. They would still maintain their A, just with slightly fewer points. And the students failing would receive the lowest possible grade that still allowed them to pass.

You would be an asshole for not saying yes. Even if a small percentage of those failing students were lying, or somehow cheating the system, wouldn’t you still contribute? Even if you had a B, wouldn’t you contribute a few points?

That is a better analogy to the situation that we find ourselves in today. The rich will not be materially affected by a slight increase in taxes.

With Apple set to announce iCloud in a few hours, I thought I’d write up my preliminary guesses about what the service will be. This is not based on any inside information I have, just my guesses based on how I think Apple thinks. I’m typing it up because, if I’m right, I want to be able to gloat.

I think iCloud is going to be more than just storage for iTunes. It will be storage for any app. Any iOS or Mac app developer will be able to leverage it and store user documents in the cloud.

Lets say you’re a developer that makes an image editing program. You can write a version for iPhone, iPad, and Mac, and they all will share the same document sandbox that lives in iCloud. A user can edit the image on the iPad, then immediately go to their Mac and keep editing the same image. This is probably why Apple released iWork last week for iPhone and iPod Touch - these apps will soon leverage iCloud for document syncing across all your devices.

There are some challenges that I am going to be interested to see how Apple addresses:

• Cost: how will Apple cover costs on iCloud? It could either a) eat them, considering how much sales iCloud will inevitably generate, b) charge end users by total iCloud usage, or c) charge App developers who want to take advantage of iCloud in their app. It’s this last option I find most interesting, maybe the could change the sales split from 70/30 to 60/40, or require them to use a subscription model which would mean more money for Apple. I think this scenario is more likely than many consider, mainly because of Apple’s proven willingness to shift cost burdens off of end users and on to App developers.
• File structure: currently there is no notion of file browsing or document hierarchies in iOS. In apps that have different documents, like Pages or GarageBand, there is just a list of all your documents, with no mention of “where” they are stored. As apps and documents get more numerous, this is going to become necessary.
• File sharing between Apps: Apps are going to need a way to pass around documents, or access documents used in other apps. Maybe you have multiple image editors and want to edit an image in one and then pass it on to the other. If they solve the “file structure” problem mentioned previously, this shouldn’t be a problem. But if they don’t then they need to come up with a way to share of expose documents to other Apps.
• Availability: will these features become available to developers immediately? I feel pretty confident that this is the direction that Apple is going in, but it may not be today. Step 1 may just be to get iCloud working for iTunes and other Apple products like Pages, and integration for third party apps may come later on down the line.

Hopefully we’ll get answers shortly.

Apple released a Q&A about the location data stored on the iPhone.

The iPhone is not logging your location. Rather, it’s maintaining a database of Wi-Fi hotspots and cell towers around your current location, some of which may be located more than one hundred miles away from your iPhone, to help your iPhone rapidly and accurately calculate its location when requested. Calculating a phone’s location using just GPS satellite data can take up to several minutes. iPhone can reduce this time to just a few seconds by using Wi-Fi hotspot and cell tower data to quickly find GPS satellites, and even triangulate its location using just Wi-Fi hotspot and cell tower data when GPS is not available (such as indoors or in basements). These calculations are performed live on the iPhone using a crowd-sourced database of Wi-Fi hotspot and cell tower data that is generated by tens of millions of iPhones sending the geo-tagged locations of nearby Wi-Fi hotspots and cell towers in an anonymous and encrypted form to Apple.

You read it here first. (And second. And third.) Okay, nobody wants to read a post about gloating, but I’m sure enjoying writing it.

The file with all this location data is stored locally on your own phone or computer. Let’s say, hypothetically, your phone was stolen. They have your email, contacts, photos, text messages and documents. Plus access to any accounts that log in automatically.

Are you really worried about the fact that they also can get a general idea of where you were six months ago?

Part of what is so intriguing about this iPhone location data dust-up is the question of why. Why is Apple storing this data on your phone? Are they trying to provide information to law enforcement? Do they have ideas for future apps that look at your location history? Do they want to provide it to advertisers?

Turns out, Apple told us nine months ago. Well, not us, but two of our esteemed Congressmen, Reps. Edward Markey (D-Mass.) and Joe Barton (R-Texas). In response to an inquiry about iPhone location data usage, they said (see pages 5 and 6):

To provide the high quality products and services that its customers demand, Apple must have access to comprehensive location-based information. For devices running the iPhone OS versions 1.1 .3 to 3.1, Apple relied on (and still relies on) databases maintained by Google and Skyhook Wireless (“Skyhook”) to provide location-based services. Beginning with the iPhone OS version 3.2 released in April 2010, Apple relies on its own databases to provide location-based services and for diagnostic purposes. These databases must be updated continuously to account for, among other things, the ever-changing physical landscape, more innovative uses of mobile technology, and the increasing number of Apple’s customers. Apple always has taken great care to protect the privacy of its customers.

Basically, when you can’t get a good GPS signal, your iPhone guesses based on WiFi and cell sites near you. Apple used to rely on other people’s databases of where WiFi and cell locations are, but now they do it in house. How do they do this?

To provide location-based services, Apple must be able to determine quickly and precisely where a device is located. To do this, Apple maintains a secure database containing information regarding known locations of cell towers and Wi-Fi access points. The information is stored in a database accessible only by Apple and does not reveal personal information about any customer.

Information about nearby cell towers and Wi-Fi access points is collected and sent to Apple with the GPS coordinates of the device, if available: (1) when a customer requests current location information and (2) automatically, in some cases, to update and maintain databases with known location information. In both cases, the device collects the following anonymous information:

• Cell Tower Information: Apple collects information about nearby cell towers, such as the location of the tower(s), Cell IDs, and data about the strength of the signal transmitted from the towers. A Cell ID refers to the unique number assigned by a cellular provider to a cell, a defined geographic area covered by a cell tower in a mobile network. Cell IDs do not provide any personal information about mobile phone users located in the cell. Location, Cell ID, and signal strength information is available to anyone with certain commercially available software.
• Wi-Fi Access Point Information: Apple collects information about nearby Wi-Fi access points, such as the location of the access point(s), Media Access Control (MAC) addresses, and data about the strength and speed of the signal transmitted by the access point(s)…

The consolidated.db is the file is how they store that data locally on the phone¹. The cell tower information is in the CellLocation table. The WiFi access point information is in the WiFiLocation table. There is also a table called CdmaCellLocation - for me it’s empty, but I’d be willing to bet that people with a CDMA (Verizon) iPhone will find if chock full of cell locations. As I’ve been saying all along, these tables are of the locations of the access points, and when you’re in those areas again, your phone can look at what kinds of signal strength is has with them to determine it’s own location. It does this because this takes much less battery than locking on to a GPS signal, and although the accuracy isn’t as good, for many applications it’s good enough.

I suspect the data in these tables is probably two way - you upload some anonymized data to Apple about cell and wifi locations that you have located, and Apple may send some back that other anonymous people have located. That would explain why we all get weird data from time to time for places we haven’t been - for some reason Apple pushed out cell locations to us because an App requested data about that location. Or maybe it knows about cell locations that have been moved and wanted everyone to have accurate information. I know this is just a hypothesis but it’s a much better explanation than I’ve heard from anyone else.

Now in this document, Apple says that when they send the approximate locations of the cell sites and WiFi access points, they also are sending the GPS coordinates (if available). However, there is no evidence that I can find that they are storing those coordinates on the phone. The coordinates are sent to Apple, which stores them in an anonymous database on their end. They aren’t in the consolidated.db anywhere - at least not that I can find. They’re definitely not in the CellLocation table that iPhoneTracker uses, because GPS coordinates tend to be very precise and nothing in that table has an accuracy of closer than 500 meters.

Why store a whole year’s worth of data? Because that’s the whole point! In order to find your location without using GPS, you want a huge database of where cell towers are and what their location is. Now Apple could probably do this in a way that better protects your privacy - they could munge the timestamps more effectively, or take all the data whenever you sync and replace it with more reliable cell location data from their servers that covers the areas that you frequently travel. Kind of like Genius syncing in iTunes; they look at where you’ve been, and give you reliable cell and wifi locations based on those locations. Smart, elegant, and secure. Why they do it this way is unclear, and surely they will provide an additional clarification now that more people are asking questions.

1. To clarify, this is my opinion. Apple has not said exactly what the purpose of the consolidated.db file is, and until they do, we won’t know for sure. I strongly suspect this is the purpose of the file, for reasons mentioned both in this post and other posts on my blog.

Dan Goodin put together a pretty good article on the iPhone tracking issue, “No, iPhone location tracking isn’t harmless and here’s why.” I have to take issue with how casually he disregarded my assertion that what is being stored in the database is cell locations and not the device’s location, but he was trying to show both sides. After all, Alasdair Allan and Pete Warden are researchers and I’m a blogger.

However, I still think I’m right. And in Dan’s defense, he hadn’t read my most recent post when he put the article together, so he hadn’t seen my evidence regarding the non-overlapping “halos” in the data points.

In the article, Alasdair and Pete were indirectly quoted as saying:

Some of the extracted databases they examined plotted literally thousands of unique coordinates in a small part of a single city. It’s almost impossible that there could be that many corresponding nodes in such a confined area, they said.

I believe this is because what the iPhone is doing is guessing the network nodes’ position using its GPS. They are so many unique ones because they are different guesses at different times.

I think the distinction of exactly what data the iPhone is collecting is important, because it speaks to both what level of detail we’re talking about and what Apple’s motivations are for collecting that data. I hope this is settled by Apple, and I still believe the evidence is on my side.

Where I agree with Dan, Alasdair, and Pete is that it isn’t, as The Register puts it, “harmless.” Alasdair said it can place you “a bit above the block level” and I’d say it’s more like neighborhood level, but yes. It can place you at around that level. Being able to access that level of data on someone is not good. And even in the cases where the data is really inaccurate, it’s bad because law enforcement is using this inaccurate data to prosecute or question potentially innocent people!

Another thing I noticed, however, is that it is not always collecting this data. In my 1+ years of using iOS 4 and collecting data, I only have 1112 distinct timestamps in my data. That’s about three times a day - hardly enough to determine my jogging route or reveal to my girlfriend which of her friends I’m cheating on her with. (It’s a joke! I love you baby!)

(Previously: Apple is not “recording your moves”)

A lot of people have said that while the data from my backcountry bike trip is interesting, there is still a large security concern for people that live in cities. Since the urban density of “cell phone towers” (or more realistically, wireless network nodes) is much greater, couldn’t someone who stole your iPhone find you in a city much more easily?

From the data I’m seeing, no. I have been looking through the table of my data more thoroughly and what I’ve found is interesting. It doesn’t log one data point at a time - it will log a couple dozen data points all at once. For example, here is my data visualized on a graph, for the timestamp of April 3rd at 5:15:25.865 PM:

Somewhere in Philadelphia

Ignore the lines that are drawn - the utility I used to convert the points to something readable by Google Earth adds them. All the points were added to the file with the exact same timestamp. What gets interesting is when you start looking at the data behind these points. Take a look at the following two screenshots:


Note the Horizontal Accuracy of the two points. This is a measure, in meters, of the confidence in that location - like when you load up maps and it shows a blue “halo” around your location indicating the area you may be in. The first point has an accuracy of 549 meters, the second 500 meters - and they are over 2000 meters from each other. Now, if these data points are supposed to be where you are, then their horizontal accuracies should all overlap on some point that reveals your actual location. But they don’t - which is why I believe they are locations of nearby cell sites, and the horizontal accuracy is a measure of how confident it is that the cell is there.

“But it’s still very revealing!” you must be thinking. After all, if it’s cell sites around you, you must be right in the middle of that circle. Fortunately for my privacy, no. At 5:15 PM on April 3rd I was in the bottom left of that circle, over a block away from the nearest dot on the map. I had just finished a 155 mile bike trip and was pretty happy to be sitting there, not moving.

A vast majority of the data in consolidated.db is in clumps like this. I grouped all the data by timestamp and found that 68% of the groups (and 96% of the points) were in clumps of more than 12 points.

Of the 40k valid entries in my CellLocation table, there were 106 entries with just a single timestamp. I looked at several of these, thinking I had perhaps found something that would give a close indication of my actual location. Still, no. One was at 1:30 AM, and it was 700 meters, or seven blocks, away from where I lay asleep, despite claiming an accuracy of 500 meters. Interestingly, this was in downtown Seattle, where there were tons of other cell nodes that it could have logged. Why did it pick just this one? Who knows.

Now there may be individual points in your consolidated.db file that happen to be the exact same as your location at the time. I’m sure there are some in mine. But that that is coincidental - the phone seems to be logging locations of nodes on the wireless cell network, and their presence in the file mean that you MAY have been near them. The Las Vegas mystery is still confounding me, but it only further proves the point that this data should not be relied on for any kind of forensic analysis.

If you’re viewing this, it’s probably because I emailed you about a place to live and you are stalking me before replying. It’s cool. When you reply, I’ll probably use your email address to do the same to you.

Feel free to read my posts, but I thought I’d just preface this with the fact that I don’t post to my blog very often and when I do it’s about random things that probably don’t “define” me very well. Also, the “About” section is very old and for some reason my blog software won’t let me edit it. I could take time to figure it out but I don’t really care. Enjoy the investigation! And email me back, damnit.

Bush as president made sense. A country with citizens this profoundly stupid deserve him. Now we have accidentally elected an intelligent president, refused to implement any of his policies except in their most watered-down forms, and proceeded to hate him for it. It’s heartbreaking to watch.

Perhaps Palin in 2012 is inevitable. We’ve certainly gotten stupider since 2008. Like how telling a child not to play with fire isn’t enough - you have to let them go ahead and burn themselves.

The Muslims are coming! They are going to build a “community center” near Ground Zero. I case you forgot (and your car’s bumper promised you never would!), Ground Zero is the Hallowed Ground where Muslims killed a bunch of people. Ever since then, we hold the ground sacred and don’t allow anything but the most somber and respectful establishments to operate there, such as Christian Churches, bank headquarters (they’d never do anything to destroy America), one hundred thousand Muslim-run hot dog stands, and of course strip clubs.

The Imam who is behind this, Feisal Abdul Rauf, is a seriously suspicious character. Less than a year ago, his wife was on Fox News [sic] where Patriot Laura Ingraham (no, not Dr. Laura, the younger one) referred to him as a moderate Muslim and praised his efforts to open this community center. Today he is a Muslim extremist who wants to do something comparable to placing a Nazi sign next to the Holocaust museum. That’s some serious radicalization in just one (midterm election) year. Just imagine how radical he will be by October 2012!

Besides, it’s just insensitive! When members of a religion do something bad, it’s the entire religion’s fault. They should be more sensitive to the victims, kind of like how we don’t build Catholic Churches around elementary schools anymore*. To do otherwise would be caving to their evil demands, and you’d have to be a lilly-livered effete east-coast liberal to do that.

They should build it somewhere else - if for no other reason than to avoid violating our Constitution’s guarantee of Free Exercise of Religion Unless It Bothers Anyone In Any Way Or Involves Brown People. Just relocate! But not Staten Island (still too close)… or Kentucky (they’re always about eight years behind, so it’s still too soon)… or Georgia (too religious already)… or California (too gay). But somewhere else. Less hallowed ground. I’m not saying Alaska but we’re all thinking it.

There, that’s settled. Now if we can just get rid of those Mexicans. And Iranians. And Koreans. BE AFRAID

*shamelessly stolen from The Daily Show last night.