Fang Xiao, Che Shangkun, Mao Minjia, Zhang Hongzhe, Zhao Ming, Zhao Xiaohang
University of Delaware, Newark, USA.
Tsinghua University, Beijing, China.
Sci Rep. 2024 Mar 4;14(1):5224. doi: 10.1038/s41598-024-55686-2.
Large language models (LLMs) have the potential to transform our lives and work through the content they generate, known as AI-Generated Content (AIGC). To harness this transformation, we need to understand the limitations of LLMs. Here, we investigate the bias of AIGC produced by seven representative LLMs, including ChatGPT and LLaMA. We collect news articles from The New York Times and Reuters, both known for their dedication to provide unbiased news. We then apply each examined LLM to generate news content with headlines of these news articles as prompts, and evaluate the gender and racial biases of the AIGC produced by the LLM by comparing the AIGC and the original news articles. We further analyze the gender bias of each LLM under biased prompts by adding gender-biased messages to prompts constructed from these news headlines. Our study reveals that the AIGC produced by each examined LLM demonstrates substantial gender and racial biases. Moreover, the AIGC generated by each LLM exhibits notable discrimination against females and individuals of the Black race. Among the LLMs, the AIGC generated by ChatGPT demonstrates the lowest level of bias, and ChatGPT is the sole model capable of declining content generation when provided with biased prompts.
大语言模型（LLMs）有潜力通过其生成的内容，即所谓的人工智能生成内容（AIGC），来改变我们的生活和工作。为了利用这一变革，我们需要了解大语言模型的局限性。在此，我们研究了包括ChatGPT和LLaMA在内的七个代表性大语言模型所产生的AIGC的偏差。我们从以提供无偏见新闻而闻名的《纽约时报》和路透社收集新闻文章。然后，我们将每个被研究的大语言模型应用于以这些新闻文章的标题为提示来生成新闻内容，并通过比较AIGC和原始新闻文章来评估大语言模型所产生的AIGC的性别和种族偏差。我们通过在由这些新闻标题构建的提示中添加性别偏见信息，进一步分析每个大语言模型在有偏见提示下的性别偏差。我们的研究表明，每个被研究的大语言模型所产生的AIGC都表现出相当大的性别和种族偏差。此外，每个大语言模型生成的AIGC对女性和黑人个体表现出明显的歧视。在这些大语言模型中，ChatGPT生成的AIGC偏差水平最低，并且ChatGPT是唯一一个在收到有偏见的提示时能够拒绝生成内容的模型。
