Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving (2112.05194v1)

Abstract: With widening deployments of NLP in daily life, inherited social biases from NLP models have become more severe and problematic. Previous studies have shown that word embeddings trained on human-generated corpora have strong gender biases that can produce discriminative results in downstream tasks. Previous debiasing methods focus mainly on modeling bias and only implicitly consider semantic information while completely overlooking the complex underlying causal structure among bias and semantic components. To address these issues, we propose a novel methodology that leverages a causal inference framework to effectively remove gender bias. The proposed method allows us to construct and analyze the complex causal mechanisms facilitating gender information flow while retaining oracle semantic information within word embeddings. Our comprehensive experiments show that the proposed method achieves state-of-the-art results in gender-debiasing tasks. In addition, our methods yield better performance in word similarity evaluation and various extrinsic downstream NLP tasks.

• The paper's main contribution is a causal inference approach that reduces gender bias in word embeddings while preserving semantic meaning.
• It employs two causal models using projection and resolving variable techniques to disentangle bias components from semantic content.
• Experimental evaluations show improved bias reduction and high performance on downstream NLP tasks such as POS tagging and NER.

Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving

This essay reviews the research paper titled "Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving" (2112.05194). The paper proposes a novel approach to address gender bias in word embeddings by leveraging causal inference techniques, while ensuring that semantic information is preserved. The authors introduce two primary methodologies within the causal inference framework to improve upon existing debiasing techniques.

Introduction

The proliferation of NLP systems into various facets of daily life has brought to light the issue of inherited biases within word embeddings, particularly gender biases. Traditional debiasing methods, which either incorporate bias modeling or post-processing, often disregard the complex causal relationships between bias and semantics. This paper addresses these limitations by integrating causal inference frameworks, providing a comprehensive solution for gender bias removal that maintains the integrity of semantic information in word embeddings.

Methodology

Causal Inference Framework

The authors propose two causal models to address different facets of gender bias: potential proxy bias and unresolved bias. These models employ causal graphs, depicted in (Figure 1) and (Figure 2), to define the interdependencies between word embeddings, gender bias, and semantic information components.

Figure 1: A causal graph for proxy bias removal.

Figure 2: A causal graph for unresolved bias removal.

Removing Potential Proxy Bias

The methodology for removing potential proxy bias involves decomposing word embeddings into gender bias and non-gender bias components. The bias component is isolated through projections onto a gender bias subspace defined by a proxy matrix. The non-gender bias component, orthogonal to this subspace, retains the semantic information. Partial Least Squares (PLS) is employed to estimate the model parameters, ensuring effective disentanglement of gender bias from semantic content.

Removing Unresolved Bias

For scenarios where proxy variables are unavailable, the framework addresses unresolved bias by leveraging resolving variables correlated with gender bias vectors. This intervention on unresolved bias employs similar causal techniques to those used for proxy bias, ensuring that gender information does not inadvertently influence semantic relationships.

Experimental Evaluation

Comprehensive experiments demonstrate that the proposed methods not only achieve leading performance in various gender bias tasks, such as bias-by-projection and the SemBias Analogy Task but also preserve crucial semantic information. The authors report reductions in gender clustering of biased words and improve correlations in bias-by-neighbors assessments.

Figure 3: t-SNE visualization.

Results in Downstream Tasks

The efficacy of the debiased embeddings is further evidenced in downstream NLP tasks, where the balance between debiasing and semantic integrity is evaluated. Embedding matrix replacement and model retraining techniques show that the authors' methods maintain competitive performance across POS tagging, chunking, and NER tasks, with the least degradation relative to the original biased embeddings.

Conclusion

The introduction of causal inference frameworks for debiasing word embeddings marks a significant step forward in addressing gender bias while preserving semantic content. The authors demonstrate that by focusing on causal relations, bias can be effectively reduced without sacrificing the utility of embeddings across various tasks. Future work may extend these frameworks to non-linear causal models or other forms of bias, broadening the applicability of these methodologies in AI development.