The challenge of the “knowledge problem” is just one example of a general truth: What we do and don’t know about the social (as opposed to the natural) world is not inherent in its nature, but is itself a function of social constructs. Much of what we can find out about companies, governments, or even one another, is governed by law. Laws of privacy, trade secrecy, the so-called Freedom of Information Act— all set limits to inquiry. They rule certain investigations out of the question before they can even begin. We need to ask: To whose benefit?

More benignly, perhaps, these companies influence the choices we make ourselves. Recommendation engines at Amazon and YouTube affect an automated familiarity, gently suggesting offerings they think we’ll like. But don’t discount the significance of that “perhaps.” The economic, political, and cultural agendas behind their suggestions are hard to unravel. As middlemen, they specialize in shifting alliances, sometimes advancing the interests of customers, sometimes suppliers: all to orchestrate an online world that maximizes their own profits.

So why does this all matter? It matters because authority is increasingly expressed algorithmically. Decisions that used to be based on human reflection are now made automatically. [...]

[...] In their race for the most profitable methods of mapping social reality, the data scientists of Silicon Valley and Wall Street tend to treat recommendations as purely technical problems. The values and prerogatives that the encoded rules enact are hidden within black boxes.

[...] Companies were gathering millions of records from pharmacies. They then sold them on to insurers eager to gain a competitive advantage by avoiding people likely to incur high medical fees. Since 1 percent of patients account for over one-fifth of health care costs, and 5 percent account for nearly half of costs, insurers who can “cherry-pick” the healthy and “lemon-drop” the sick will see far more profit than those who take all comers. Prescription data gave insurers the information they needed to tailor policies to exclude preexisting conditions and to impose higher charges for some members.

[...] Marketers plot to tout beauty products at moments of the day that women feel least attractive. There’s little to stop them from compiling digital dossiers of the vulnerabilities of each of us. In the hall of mirrors of online marketing, discrimination can easily masquerade as innovation.

[...] Runaway data can lead to cascading disadvantages as digital alchemy creates new analog realities. Once one piece of software has inferred that a person is a bad credit risk, a shirking worker, or a marginal consumer, that attribute may appear with decision-making clout in other systems all over the economy. [...]

[...] software engineers construct the datasets mined by scoring systems; they define the parameters of data-mining analyses; they create the clusters, links, and decision trees applied; they generate the predictive models applied. Human biases and values are embedded into each and every step of development. Computerization may simply drive discrimination upstream.

[...] Laws prevent the government from collecting certain kinds of information on citizens, but data brokers are not so constrained. And once someone else has collected that information, little stops the government from buying it, demanding it, or even hacking into it.

Our off- and online actions are logged in hundreds of private-sector databases. Aptly called “big brother’s little helpers” by privacy expert Chris Hoofnagle, private-sector data brokers gather files that police would never be able to gather on their own, and then sell them to the police. This is not a “bug” in our surveillance system, but a “feature.” Note that the very definition of fusion centers includes their willingness to receive information from private parties. The Snowden leaks make the shared infrastructure of state and private data collection incontrovertible. Never again can data deregulationists claim that corporate data collection is entirely distinct and far less threatening than government surveillance. They are irreversibly intertwined.

[...] search services, social and not, are “must-have” properties for advertisers as well as users. As such, they have made very deep inroads indeed into the sphere of cultural, economic, and political influence that was once dominated by broadcast networks, radio stations, and newspapers. But their dominance is so complete, and their technology so complex, that they have escaped pressures for transparency and accountability that kept traditional media answerable to the public

We pay no money for Google’s services. But someone pays for its thousands of engineers, and that someone is advertisers. Nearly all the company’s revenue comes from marketers eager to reach the targeted audiences that Google delivers so abundantly. We pay with our attention and with our data, the raw material of marketing. [...]