How to stop AI from reinforcing biases
August 2, 2019
August 2, 2019
In the fall of 2018, the New York Times published a piece about an experiment in which they used an algorithm to produce Halloween costume ideas. The results were amusing: “baseball clown”, “cat witch”, “king dog.” The algorithm combined random letters to make words, which it then compared to the set of real words in its training data. If it found a match, it kept the word and paired it with another one.
In this example—and in many more serious ones like it—the algorithm was given pre-existing patterns and taught to replicate them. The meaning of the words was never “understood” by the algorithm, but it simply produced results that matched the pattern it had learned. If a word had been spelled incorrectly in the training data, the algorithm would perpetuate the error. If a word was obsolete, the algorithm, too, would be out-of-date. If a word had negative associations, the algorithm would be “rude.”
There have been several examples that show how bad feedback loops, such as those mentioned above, could lead to negative impacts on society.
To name one, Amazon developed an algorithm to inform recruitment decisions and was using historical data from the past 10 years, a time period in which men dominated the technology industry. As a result, Amazon found its algorithm discriminated against female applicants. Or take another example from the US criminal justice system. A biased algorithm—that gave black defendants a higher risk score—was used to predict the risk of recidivism and influence sentencing in several states.
The problem is that simple algorithms treat all data as immutable, even data about our preferences, income, life situation or countless other shifting patterns. What can happen then, is that algorithms can trap people in their origins, history or a stereotype. This should compel all who develop algorithms to pin-point and address potential, unintended consequences.
The problem is that simple algorithms treat all data as immutable, even data about our preferences, income, life situation. What can happen then, is that algorithms can trap people in their origins, history or a stereotype.
The Future of Privacy Forum (FPF), a nonprofit think tank, has identified four main types of harm—or unintended consequences—that algorithms can cause.
Most organizations are committed to avoiding these harms, and indeed, the majority of biases are accidental and negligent, not intentional. But to avoid accidents and negligence, we should ask the following questions in the planning, design and evaluation phases of our work with algorithms and related AI applications:
In order to understand whether the results of an algorithm are fair, the creator needs to engage the impacted audiences to understand the context of fairness in the application of the algorithm.
We program algorithms to give us exactly what we have asked for, so we shouldn’t be surprised when they do.
None of the issues mentioned in this article are inherent with machine learning algorithms themselves. Instead, issues arise from the way they interact with society and the unintended consequences that can result from those interactions. As such, putting the ethical implications at the heart of the development of each new algorithm is vital.
One way to ensure this is by embracing public health models of governance, which treat issues as indicative of underlying drivers, rather than problems to be solved per se. Another would be to ensure algorithms can be adapted more readily to newer or better data, in ways that do not exaggerate historical patterns. We see this every day in the way AI at Spotify or Amazon quickly adapts recommendations to our latest searches.
Finally, targeted research identifying individual problems and solutions is critical to the success of any effort to create more ethical AI. We need to see more resources—and more senior leadership attention—directed at ensuring algorithms do not have negative impacts on individuals or society. Just as data privacy and cyber security have moved from departmental to board-level issues, responsible governance of AI must be quickly elevated in importance by all organizations that use it.