There is much still to be won and lost in the battle for our online autonomy in the future. As the next generation of web technology improves the integration of all our digital activities, allowing machines to organize even more of our lives, others will continue to learn more about us than we even know ourselves. In this context, focusing on our power over this process as consumers is a mistake: the power being exercised over us is precisely based on our being socialized as consumers. This prepares us to accept a city where every park is paved over to build freeways and every sports field and roller-skating rink is demolished to build a shopping mall. Such a city would not be a functional, let alone enjoyable, place to live. But it would be a place where data traders and retailers made a lot of money.
love this
This is not an attempt to pin blame on evil engineers or designers. The people who made these cars were working in a specific corporate climate. Their organizations were led by ruthless executives. The leadership of companies like Ford and GM ignored safety concerns in competition with other companies that did the same. It was not even a problem confined to the auto industry; there have been many other similar scandals involving corporate indifference to the human consequences of poorly designed consumer products. These scandals are not aberrant; they occur in a context, and to avoid them happening again requires a political strategy to attack the logic that produces them.
on the ford pinto
Computer code itself functions as a form of law. It is written by humans and it regulates their behavior, like other systems of power distribution. It is not an objective process or force of nature. It expresses a power relation between coder and user, and it will reflect the system in which coders work. “Code is never found,” Lawrence Lessig reminds us. “It is only ever made, and only ever made by us.” Letting the free market determine these matters means that digital technology risks reproducing discrimination under the cover of an inscrutable process. [...]
The implication was that these modifications to the standard testing process were done in response to public outrage, generated by Dowie’s article and the lawsuits. Ford was held to standards that other companies were not—standards it had no way of knowing that it was required to meet. Ford has never admitted that it did anything wrong.
Yet the lesson is this: as we learn more about an industry and what is possible technologically, we need to update our expectations in terms of safety and accountability. We need to organize activists, lawyers and journalists to highlight the human consequences of badly designed technology, and force the industry to adapt to design culture that values safety and works to mitigate bias. We need to demand that governments intervene in this industry to establish publicly determined standards and methods for holding companies accountable when they are breached. The standards must be constantly updated and responsive to changing circumstances as we learn more about the problems and experiment with solutions. Just like we would not let a car onto the road without crash tests, the parameters of which are subject to public scrutiny and influence, algorithms and products should not be inflicted on the public if they have not met certain standards or been tested for certain biases before shipping. We need to create a feedback loop for good design that allows lessons learned in the field to inform improvements to a product.
Digital technology, at least as much as any innovation over the last two centuries, offers us the opportunity to create a society that can meet the needs of every human being and allow them to explore their potential. But at present, too much power over the development of technology rests in the hands of technology capitalists and political elites who do their bidding. These people are good at what they do and also at convincing us that they are the best people to do it. They talk about egalitarianism and social connection in their public relations pitches and marketing campaigns, but what they hope to gain from digital technology is different: they aspire to wealth and power.
what if some of them actually believe in it tho. that's the scariest thing to me
These are serious and frightening design problems, but they also result in an immense waste of human potential. It is not just that companies do not invite feedback on their software or input from users on their design. Their objective, the purpose of their software, is not to service the user. Their primary goal is to retain control of that software. They want to control who uses it (that is, only paying customers). Proprietary software design makes a fetish of creativity—turning it into something abstract and commodified, geared to the purpose of money-making, rather than a collective or public good.
Paine’s writings mark the beginning in Western political thought of a very long tradition, namely that people are worth more than their productive capacity. People are not simply economic units whose moral worth is measured by what they produce for the market. Every person is entitled to a share in the wealth that is created collectively in the world, as part of recognizing his or her existence. The capacity to feed, clothe and shelter oneself ought not be dependent on the capacity to do some kind of productive work, especially when being productive is defined within the narrow confines of material value determined by market capitalism. The moral hierarchy created around people’s economic success as a productive unit is false and dehumanizing. People have a claim upon society by virtue of their humanity.
The aristocracy represented the ossified capriciousness of wealth distribution that Paine confronted in pre-revolutionary America; Engels denounced the horrors imposed on workers by industrialists in nineteenth-century England. The leaders of technology capitalism fall into a similar category today. Their much-lauded intellect and innovations are directed toward inflating and jealously guarding the wealth they accumulate and helping their fellow captains of industry to do the same. They are indifferent to the miseries they inflict on countless people; they see themselves as noble individuals pursuing their divine right to make money. Yet this is only possible through the exploitation of labor. Which also means that organized labor still holds the power to transform society.
The survival of capitalism is also dependent on using technology to extract more from workers wherever possible. Companies are investing in technology that allows them to reduce their labor requirements through data-driven scheduling, and the costs of such arrangements fall to workers to bear. The increasing adoption of this kind of technology is responsible in part for the rise in casualization of labor over the last several decades. Nearly 60 percent of American workers (around 80 million people) are paid by the hour, and nearly half of these are subject to just-in-time scheduling, with no certainty about hours or start times. When we are at work, we are watched more closely than ever. We carry devices connected to the cloud, for example, which monitor every task and the time it takes to perform. The much-derided consumer version of Google Glass has been reincarnated as a workplace management tool, to better track every task assigned to a worker. Amazon is patenting a wristband to squeeze more out of its workers by guiding their movements so they can pack items more quickly. Employers are making the most of technology for the purposes of optimizing how they use and monitor labor.
Automation under our current system—where retraining is expensive and the welfare net is minimal—wastes the potential of both workers and technology. It creates the paradox of both increased productivity and impoverishment. Outsourcing work to machines, particularly if the task is boring or dangerous, is a highly worthwhile goal, but it is not one that we are moving toward under the current system. And those who will suffer under successive waves of automation will often be the most vulnerable layers of society—the worst-paid, with the fewest skills and the fewest resources to fall back on in times of crisis. [...]