With nearly a billion registered users globally, AI's reach is undeniable. Yet, if half of its training data is in only one language, how truly 'global' can its educational impact be?
Imagine an AI chatbot designed by the best tech companies to teach, but it accidentally makes stereotypes worse or gives inaccurate feedback to a student. This issue is more pervasive than we think and is related to cultural bias in the fast-growing world of AI. There are over 200 countries in the world and we collectively speak over 7000 languages. But a majority of the AI models that we are using, from Google, OpenAI and Anthropic are all generated in just one country, the US, with half of its training data in only one language, English. And these provide information and generate billions of responses to people all over the world.
So I ask: Can a model trained in the US create culturally relevant educational material for secondary students in rural India?
The answer: Not to the same quality it can for materials aimed at students in Urban USA.
The reason: bias on multiple levels.
And unless we pay attention to it, the situation isn’t going to get better. First we need to understand the issue in detail…
The Root of Algorithmic Bias in LLMs
The roots of algorithmic bias, particularly in areas like unrepresentative training data, have a significant history, as discussed in detail in my article on the Evolution of Cultural Representation in AI Models. This issue of bias comes from three main areas. These are:
1. Unrepresentative Training Data (Data-driven bias)
2. Biased Model Architecture / Post-training Fine-Tuning
3. Bias towards default use
1. Unrepresentative Training Data (Data-driven bias)
Cultural bias in AI basically comes down to how these big language models are trained. They learn from a huge amount of data scraped from the internet, but about half of that is in the English language. This means the models end up leaning heavily on Western ways of thinking and social values. Because of this, they often reproduce existing inequalities and tend to align with the cultural views of English-speaking, Protestant European countries. This lopsided training means the models favor Western (10% of the world population) cultural stories and might miss out on what non-Western cultures (the remaining 90% of the world population) have to say. It's a big problem because it limits access for people who speak less common languages and just reinforces the dominance of English.
The specific language the training happens in further engrains specific characteristics into the model. That’s because the models break down the text into tokens (small words or word parts) and figures out how these tokens relate to each other based on how often they show up together in the training data. For instance, a model trained mostly on English will closely associate the words ‘cats’, ‘dogs’ and ‘raining’ together because of the idiom 'it's raining cats and dogs'. Even though this token association has no relevance in any other language or culture, it could still be used in those contexts. At the same time, idioms and nuances of languages like Hawaiian ('Ōlelo Hawai'i), or Twi (Ghana) that have not been analyzed and tokenized in the same way would be misinterpreted.
2. Biased Model Architecture / Post-training Fine-Tuning
After the model is initially trained on the data above, it goes through another process, where developers fine-tune the outputs using a method called Reinforcement Learning from Human Feedback (RLHF) to try and make models safer and more in line with what they think end-users want. But if the people involved in this process are mostly US-based, their specific cultural preferences get baked right into the system, making the potential for bias even greater.
3. Bias towards default use
More frequently, generative AI users go with the first answer that the AI gives them without further checks or research. This bias towards the AIs answer without checks makes the situation even worse. Even though this issue is different from the first 2 which is a direct bias toward western culture, it has the same effect of not considering alternatives.
Ultimately, these three points taken together lead to sociocultural blind spots and algorithmic monoculture. It is a direct result of most of the talent, money, and know-how being concentrated in a few Western tech hotspots. A small number of companies get to call the shots on what the industry focuses on and how things are designed. Because of this, the developers' own views end up shaping how algorithms work and how their results are understood. The outcome? Models that better reflect the opinions of people who are already well-represented and privileged online, rather than those from the rest of the world, which just happens to be an overwhelming majority of people.
Pedagogical consequences of cultural bias in EdTech
Given these inherent biases, what problems can this actually lead to? I want to highlight 3 main issues for education.
1. Distortion and Misassessment of Student Performance
Bias towards specific languages and cultures skew how we judge student work. Automated grading often prefers certain writing styles and doesn't appreciate other ways of writing, which means unfair grades. Plus, language bias can misinterpret grammar from non-native speakers or different regions. This can lead to bad teaching choices and inaccurate feedback generated by the LLMs, misguiding students and damaging motivation and confidence.
2. Widening of Educational Inequalities and Digital Neocolonialism
When AI in education has cultural biases, it makes existing problems worse. For example, it can push gender stereotypes in career advice and mess up how we evaluate students. This also means we end up with a narrower curriculum because if we rely too much on AI tools with a limited cultural view, we get a "one-size-fits-all" approach that ignores non-Western knowledge and leads to what some call "digital neocolonialism." Plus, AI systems trained globally often miss what local learners actually need, pushing Western teaching styles and policies that just don't work or are completely wrong in many countries and regions. For example, an AI may (and actually did) recommend that I use it to augment the features of a UAE landmark or famous figure to demonstrate AI literacy to secondary students. In the UK and the US, this practice would not be cause for concern. However this practice is illegal in the UAE, stemming from their already strict privacy and digital media use laws and culture, and could lead to hefty fines.
3. Erosion of Identity, Trust, and Accountability
AI tools trained on mostly Western data, might ignore or misrepresent Indigenous knowledge and history, giving a biased view that can disadvantage students and their cultural identity. This can really affect students and make them lose trust in the systems intended to teach them.
In essence, if educational AI is designed without different cultures in mind, it acts less like a universal mentor and more like a mirror reflecting a single dominant culture. This makes learning less effective for a global audience and can be academically harmful to students from marginalized or non-Western backgrounds. To add to this, the ‘Black box’ nature of AI systems - how we don’t really know how AI decisions are made - makes it hard to hold them accountable. There are attempts to improve this, but huge issues still exist.
Strategies for Ensuring Cultural Awareness in AI Outputs
Addressing these serious issues, in an era where AI use is not just vast, but greatly increasing, is vital. Luckily there are four things that can be done. The first three are the responsibility of the developers and policy makers, so other than giving external pressure, changes may be out of the control of the regular user. However, the final one is something all of us can actively do, and will make a difference.
1. Use Local Data & Decentralized Tokenization
Developers can build AI models with local, culturally relevant data and tokenization. This helps avoid AI always defaulting to Western ideas and English language concepts. China and the UAE have already done this.
2. Fine-Tune Existing Models
Developers can take current AI models, even those trained in the West, and fine-tune them with diverse, non-English data. This makes them better at understanding different cultures, especially for languages that don't have a lot of digital resources. Japan has already done this.
3. Policy Changes for Diverse Data
We can encourage existing developers, like Google, OpenAI and Anthropic to fix their products, and make them better. We can make them collect and use diverse datasets that actually reflect all kinds of users and learners. Plus, we should push for transparent AI audits, explainable AI, and even work with diverse, indigenous and international communities to create AI solutions.
4. Smart Prompt Engineering
Educators and students should get good at telling AI what cultural context to use. Instead of vague prompts that lead to Western biases, we can use techniques like specific "Cultural Prompting" to guide the AI.
Conclusion
The issue of cultural bias in education is nothing new. Discussions regarding culturally relevant pedagogy - separate from edtech - in Western societies have been going on for decades, and have been an issue in edtech for a long while. It’s just that with the recent introduction and growth of generative AI, this powerful new technology, it would be detrimental to see the same old issues go unaddressed. We can't ignore cultural bias in AI education, but it's also a huge chance for us to do something transformative! If we consciously bake cultural intelligence into every part of AI, from the data we use to how we write prompts, we won't just avoid problems. We'll actually open up a world of possibilities for learning globally. Imagine an AI that transcends a single culture, moving beyond the lack of diversity within its model. Instead it actively reflects the rich tapestry of human knowledge. This would help every student see their own world and themselves as part of the whole, valued in the digital age. This isn't just about making better tech, it's about creating a fairer and more exciting future for education and other sectors, everywhere.
Sources
- Map created by www.mapchart.net
- Algorithmic Bias in AI-Enhanced Education: Cultural Dimensions and Pedagogical Impact
- Algorithmic Bias in Educational Systems: Examining the Impact of AI-Driven Decision Making in Modern Education
- Cultivation of Human Centered Artificial Intelligence: Culturally Adaptive Thinking in Education (CATE) for AI
- Cultural Bias and Cultural Alignment of Large Language Models
- Decolonising Educational Technology
- Digital Equity and Inclusion in Education: An Overview of Practice and Policy in OECD Countries
- Ed-Tech Landscape and Challenges in Asia and the Middle East and North Africa
- EdTech Culturation: Integrating a Culturally Relevant Pedagogy into Educational Technology
- Edtech Platforms from Below: A Family Ethnography of Marginalized Communities and Their Digital Learning Post-Pandemic
- Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology
- Generative AI and Digital Neocolonialism in Global Education: Towards an Equitable Framework
- Investigating Cultural Alignment of Large Language Models
- Navigating Algorithm Bias in AI: Ensuring Fairness and Trust in Africa
- Tokenising Culture: Causes and Consequences of Cultural Misalignment in Large Language Models
- Towards Decolonising the Ethics of AI in Education
- “I Would Never Trust Anything Western”: Kumu (Educator) Perspectives on Use of LLMs for Culturally Revitalizing CS Education in Hawaiian Schools