Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases (2112.07868v2)

Published 15 Dec 2021 in cs.CL and cs.AI

Abstract: Detecting social bias in text is challenging due to nuance, subjectivity, and difficulty in obtaining good quality labeled datasets at scale, especially given the evolving nature of social biases and society. To address these challenges, we propose a few-shot instruction-based method for prompting pre-trained LLMs (LMs). We select a few class-balanced exemplars from a small support repository that are closest to the query to be labeled in the embedding space. We then provide the LM with instruction that consists of this subset of labeled exemplars, the query text to be classified, a definition of bias, and prompt it to make a decision. We demonstrate that large LMs used in a few-shot context can detect different types of fine-grained biases with similar and sometimes superior accuracy to fine-tuned models. We observe that the largest 530B parameter model is significantly more effective in detecting social bias compared to smaller models (achieving at least 13% improvement in AUC metric compared to other models). It also maintains a high AUC (dropping less than 2%) when the labeled repository is reduced to as few as $100$ samples. Large pretrained LLMs thus make it easier and quicker to build new bias detectors.

User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Shrimai Prabhumoye (40 papers)
  2. Rafal Kocielnik (13 papers)
  3. Mohammad Shoeybi (60 papers)
  4. Anima Anandkumar (236 papers)
  5. Bryan Catanzaro (123 papers)
Citations (19)

Summary

We haven't generated a summary for this paper yet.