{"id":81477,"date":"2021-11-29T07:30:00","date_gmt":"2021-11-29T06:30:00","guid":{"rendered":"https:\/\/www.hiig.de\/?p=81477"},"modified":"2023-03-28T14:02:54","modified_gmt":"2023-03-28T12:02:54","slug":"bias-in-natural-language-processing","status":"publish","type":"post","link":"https:\/\/www.hiig.de\/en\/bias-in-natural-language-processing\/","title":{"rendered":"How to identify bias in Natural Language Processing"},"content":{"rendered":"\n

Why do translation programmes or chatbots on our computers often contain discriminatory tendencies towards gender or race? Here is an easy guide to understand how bias in natural language processing works. We explain why sexist technologies like search engines are not just an unfortunate coincidence.<\/strong><\/p>\n\n\n\n

What is bias in translation programmes?<\/strong><\/h2>\n\n\n\n

Have you ever used machine translation for translating a sentence to Estonian? In some languages, like Estonian, pronouns and nouns do not indicate gender. When translating to English, the software has to make a choice. Which word becomes male and which female? However, often it is a choice grounded in stereotypes. Is this just a coincidence?<\/p>\n\n\n\n

\"\"
Figure 1<\/strong>: Screenshot of a translation from English to Estonian (and vice versa) by Google Translate. There is no grammatical distinction for gender in Estonian but for English it is necessary, the programme has to decide which word becomes grammatically male or female. The algorithms are often based on stereotypes, as in this example.<\/span><\/figcaption><\/figure><\/div>\n\n\n\n

These kinds of systems are created using large amounts of language data \u2013 and this naturally occurring language data contains biases: systematic and unfair discrimination against certain individuals or groups of individuals in favour of others. The way these systems are created can even amplify these pre-existing biases<\/a>. In this blog post we ask where such distortions come from and if there is anything we can do to reduce it? <\/p>\n\n\n\n

We will start with a technical explanation of how individual words from large amounts of language data are transformed into numerical representations \u2013 so-called embeddings \u2013 so that computers can make sense of them. This isn\u2019t to bore you but to make clear that bias in word embeddings is no coincidence but rather a logical byproduct. We will then discuss what happens when biased embeddings are used in practical applications.<\/p>\n\n\n\n

Word embeddings: what, why, and how?<\/strong><\/h2>\n\n\n\n

Word embeddings are simply lists of numbers. When you search Google for \u201cwomen\u201d, what the program sees is a list of numbers and the results are based on calculations on those numbers. Computers cannot process words directly. And this is why we need embeddings.<\/p>\n\n\n\n

\"Eine
Figure 2: <\/strong>Numerical representation or word embedding of the word \u201cwoman\u201d. It is nothing but a long list, or vector, of numbers.<\/span><\/figcaption><\/figure>\n\n\n\n

The main goal is to transform words into numbers such that only a minimum of information is lost. Take the following example. Each word is represented by a single number. <\/p>\n\n\n\n

\"\"
Figure 3<\/span><\/strong><\/figcaption><\/figure><\/div>\n\n\n\n

However, we can capture even more information. For instance, \u201cnurse\u201d, \u201cdoctor\u201d, and \u201chospital\u201d are somehow similar. We might want similarity to be represented in the embeddings, too. And this is why we use more sophisticated methods.<\/p>\n\n\n\n

Word embeddings and Machine Learning<\/strong><\/h2>\n\n\n\n

We will focus on a particular embedding method called GloVe<\/a>. GloVe falls into the category of machine learning. This means that we do not manually assign numbers to a word like we did before but rather learn the embeddings on a large corpus of language (and by large we mean billions of words).<\/p>\n\n\n\n

The basic idea is often expressed by the phrase: \u201cYou shall know a word by the company it keeps<\/a>\u201d. This simply means that the context of a word says much about its meaning. Take the previous sentence again. If we were to mask one of the words, you could still roughly guess what it is: \u201cThe ___ and the doctor work at the hospital.\u201d This idea forms the foundation for GloVe and its algorithm consists of three steps.<\/p>\n\n\n\n

  1. First, we sweep a window over a sentence from our language corpus.<\/strong> The size of the window is up to us and depends on how big we want the surrounding context of a word to be. There have been experiments<\/a> that with a window size of 2 (like below) \u201cHogwarts\u201d is stronger associated with other fictional schools like \u201cSunnydale\u201d or \u201cEvernight\u201d but with a size of 5 it is associated with words like \u201cDumbledore\u201d or \u201cMalfoy\u201d.<\/li><\/ol>\n\n\n\n
    \"\"
    Figure 4<\/span><\/strong><\/figcaption><\/figure><\/div>\n\n\n\n
    1. Second, we count the co-occurrences of the words.<\/strong> By co-occurrence we mean the words which appear in the same window as the given word. \u201cNurse\u201d, for example, co-occurs with \u201cand\u201d and twice with \u201cthe\u201d. The result can be displayed in a table. <\/li><\/ol>\n\n\n\n
      \"\"
      Figure 5<\/span><\/strong><\/figcaption><\/figure><\/div>\n\n\n\n
      1. The third step is the actual learning.<\/strong> We won\u2019t get into detail because this would require a lot of maths. However, the basic idea is simple. We calculate co-occurrence probabilities. Words that co-occur often, such as \u201cdoctor\u201d and \u201cnurse\u201d, get a high probability and unlikely pairs like \u201cdoctor\u201d and \u201celephant\u201d get a low probability. Finally, we assign the embeddings according to the probabilities. The embedding for \u201cdoctor\u201d and the one for \u201cnurse\u201d, for instance, will have some similar numbers and they will be quite different to the numbers for \u201celephant\u201d.<\/li><\/ol>\n\n\n\n
        \"\"
        Figure 6:<\/strong> This code snippet outputs the 20 most similar words to \u201cbanana\u201d. The numbers next to the words display their similarity to \u201cbanana\u201d. High numbers mean high similarity.<\/span><\/figcaption><\/figure><\/div>\n\n\n\n

        That\u2019s it! Those are all steps needed to create GloVe embeddings. All that was done was to swipe a window over millions of sentences, count the co-occurrences, calculate the probabilities, and assign the embeddings. <\/p>\n\n\n\n

        \"\"
        Figure 7<\/strong>: Word embeddings let you do actual maths on words. If you subtract the embedding for \u201cMan\u201d from the one for \u201cKing\u201d and add the embedding for \u201cWoman\u201d, your result is \u201cQueen\u201d.<\/span><\/figcaption><\/figure><\/div>\n\n\n\n

        Whilst GloVe is not used anymore in most modern technologies, and has been replaced by slightly more advanced methods, the underlying principles are the same: similarity (and meaning) are assigned based on co-occurrence.<\/p>\n\n\n\n

        Bias in Natural Language Processing: Is it just a fantasy?<\/strong><\/h2>\n\n\n\n

        But back to the original topic: what is it about this method that makes it so vulnerable to biases? Imagine that words like \u201cengineer\u201d often co-occur with \u201cman\u201d or \u201cnurse\u201d with \u201cwoman\u201d. GloVe will assign strong similarity to those pairs, too. And this is a problem. To emphasise this point: bias in word embeddings is no coincidence. The very same mechanism that gives them the ability to capture meaning of words is also responsible for gender, race and other biases. <\/p>\n\n\n\n

        Bias in embeddings is certainly not cool. But is it merely a theoretical problem or are there also real-world implications? Well, here are some examples to convince you of the latter.<\/p>\n\n\n\n

        Word embeddings constitute the input of most modern systems, including many ubiquitous technologies that we use daily. From search engines to speech recognition, translation tools to predictive text, embeddings form the foundation of all of these. And when the foundation is biased, there is a good chance that it spreads to the entire system.<\/p>\n\n\n\n

        Male-trained word embeddings fuel gender bias: It\u2019s real life<\/strong><\/h2>\n\n\n\n

        When you type \u201cmachine learning\u201d into a search engine, you will get thousands of results \u2013 too many to go through them all. So you need to rank them by relevance. But how do you determine relevance? As it has been shown<\/a>, one efficient way is to use word embeddings and rank by similarity. Results that include words similar to \u201cmachine learning\u201d will get higher rankings. However, what has been shown<\/a>, too, is that embeddings are often gender biased. Say, you search for \u201cComputer science PhD student at Humboldt University\u201d. What might happen<\/a> is that the webpages of male PhD students will be ranked higher because their names are stronger associated with \u201ccomputer science\u201d. And this in turn reduces the visibility of women in computer science.<\/p>\n\n\n\n

        Maybe it\u2019s no surprise that these word embeddings grow up to be biased, given what they are fed with. The training data of GPT-2, one of the biggest language models, is sourced by scraping outbound links from Reddit, which is mainly populated by males between the ages of 18 and 29<\/a>. However, there are also embeddings<\/a> that have been trained on Wikipedia \u2013 a seemingly unbiased source \u2013 and yet they turn out to be biased. This might be<\/a> because only about 18% of biographies on (English) Wikipedia are about women and only 8\u201315% of contributors are female. Truth is, finding large balanced language data sets is a pretty tough job.<\/p>\n\n\n\n

        We could go on<\/a> and go on<\/a> and go on<\/a> and go on<\/a> with examples but let us now turn to something more optimistic: solutions!<\/p>\n\n\n\n

        So what can we do against  bias in Natural Language Processing?<\/strong><\/h2>\n\n\n\n

        So, the bad news: language-based technology is often racist, sexist, ableist, ageist\u2026 the list goes on. But there is a faint glimmer of some good news: we can do something about it! Even better news: awareness that these problems exist is already the first step. We can also attempt to nip the problem in the bud, and try to create a dataset that contains less bias from the offset. <\/p>\n\n\n\n

        As this is nigh on impossible, as linguistic data will always include pre-existing biases<\/a>, an alternative is to properly describe your dataset<\/a> \u2013 that way, both research and industry can ensure that they only use appropriate datasets and can assess more thoroughly what impact using a certain dataset in a certain context will have. <\/p>\n\n\n\n

        On the technical level, various techniques have been proposed to reduce bias in the actual system. Some of these techniques involve judging the appropriateness of a distinction, for example, the gender distinction between \u2018mother\u2019 vs. \u2018father\u2019 is appropriate, whereas \u2018homemaker\u2019 vs. \u2018programmer\u2019 rather inappropriate<\/a>. The word embeddings for these inappropriate word pairs are then adjusted accordingly. How successful these methods actually are in practice is debatable<\/a>. <\/p>\n\n\n\n

        Basically, the people who may be affected by a technology \u2013 particularly those affected adversely \u2013 should be at the centre of research. This also involves consulting with all potential stakeholders at all stages of the development process \u2013 the earlier the better! <\/p>\n\n\n\n

        Watch your data and try participation<\/strong><\/h2>\n\n\n\n

        Biases that exist in our society are reflected in the language we use; this is further reflected and can even be amplified in technologies that have language as a basis. As a minimum, technologies that have critical outcomes \u2013 such as a system that automatically decides whether or not to grant loans \u2013 should be designed in a participatory manner, from the beginning of the design process to the very end. Just because people can be discriminatory, this does not mean that computers have to be too! There are tools<\/a> and methods<\/a> to decrease the effects and we can even use word embeddings to do research<\/a> on bias. So: be aware of what data you are using, try out technical de-biasing methods and always keep various stakeholders in the loop.<\/p>\n