Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity (2209.12106v2)

Abstract: LLMs have demonstrated impressive capabilities in generating fluent text, as well as tendencies to reproduce undesirable social biases. This study investigates whether LLMs reproduce the moral biases associated with political groups in the United States, an instance of a broader capability herein termed moral mimicry. This hypothesis is explored in the GPT-3/3.5 and OPT families of Transformer-based LLMs. Using tools from Moral Foundations Theory, it is shown that these LLMs are indeed moral mimics. When prompted with a liberal or conservative political identity, the models generate text reflecting corresponding moral biases. This study also explores the relationship between moral mimicry and model size, and similarity between human and LLM moral word use.

• The paper shows that LLMs generate politically tailored moral rationalizations based on Moral Foundations Theory.
• It employs controlled experiments using varied prompts and models (GPT-3/3.5, OPT) to measure foundation-specific effect sizes.
• Findings indicate that model architecture and post-training methods significantly influence mimicry, with implications for AI ethics and political polarization.

Moral Mimicry in LLMs: Alignment of Moral Rationalizations with Political Identity

Introduction and Problem Formulation

This essay analyzes “Moral Mimicry: LLMs Produce Moral Rationalizations Tailored to Political Identity” (2209.12106), which investigates the extent to which state-of-the-art LLMs—specifically those in the GPT-3/3.5 and OPT families—reproduce and modulate moral rationalizations as a function of explicit political identity priming. The paper leverages Moral Foundations Theory (MFT), which operationalizes human morality along five axes (Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Sanctity/Degradation), to probe whether these models' outputs exhibit the characteristic foundational biases observed in U.S. liberal and conservative groups.

Given that LLMs can be highly influential in digital discourse, their ability to align rationalizations with sociopolitical identities has practical and ethical significance, including downstream risks around polarization and targeted influence.

Experimental Methodology

The core experimental loop constructs prompts from parameterizable templates comprising: (1) a scenario warranting moral judgment, (2) a political identity string (e.g., "as a liberal"/"as a conservative"), and (3) a stance (moral/immoral). Prompts are paired with scenarios drawn from the Moral Stories, ETHICS, and Social Chemistry datasets, covering both action-centered and situational narratives. Completions are generated from multiple model variants, holding decoding parameters fixed (temperature=0, max_tokens=64).

Figure 1: Overview of the experimental pipeline, illustrating prompt assembly, generation, foundation content analysis, and effect size estimation.

The output text is then analyzed for relative foundation-specific lexical content using three versions of the Moral Foundations Dictionary (MFDv1, MFDv2, eMFD). For each completion, the proportion of words associated with each foundation serves as a proxy for moral rationalization content. The principal signal of interest is the effect size: the absolute difference in foundation activation, for a given scenario and stance, when the political identity string alternates between "liberal" and "conservative."

LLM vs. Human Foundation Use

A two-tiered evaluation is conducted to assess:

• Criterion A: Whether LLMs increase references to a particular foundation when the prompt scenario is salient for that foundation.
• Criterion B: Whether LLMs match or diverge from human consensus in foundation use, compared to human inter-annotator variability.

  Figure 2: Left—LLMs increase use of foundation-appropriate language in ground-truth foundation scenarios; Right—Deviation from human consensus is generally larger for LLMs than for individual humans, measured with text-davinci-002 on the Social Chemistry Situations dataset.

LLMs consistently satisfy Criterion A, indicating lexical sensitivity. However, per Criterion B, deviations from human consensus are systematically larger than human-human differences, highlighting persistent representation misalignment.

Emergence of Moral Mimicry

The principal hypothesis interrogated is whether political identity priming induces foundation-specific shifts in model output, mirroring empirical differences between liberal and conservative populations. The paper finds robust evidence: effect sizes align with the MFT-predicted directions—liberal identity primes increase Care/Harm and Fairness/Cheating, while conservative primes augment Authority/Subversion, Loyalty/Betrayal, and Sanctity/Degradation.

Figure 3: Effect sizes for liberal vs. conservative prompting across four models (OPT-30B, text-davinci-001/002/003), three dictionaries, and all moral foundations; directionality matches MFT predictions, with large effect sizes for some foundations.

Of the 60 foundation/model/dictionary combinations, only 11 yield effect sizes contrary to expectations, and these are of negligible magnitude. This constitutes strong evidence that present-day large LLMs are "moral mimics" in the defined sense.

Scaling, Architecture, and Training Effects

To map the scaling behavior of moral mimicry, effect sizes and a summary MFH-Score are computed as a function of parameter count for multiple GPT-3 and OPT models. The relationship is nontrivial: For OPT, MFH-Score correlates positively with size (r=0.69), but anomaly is observed in the 13B variant. In the GPT-3.5 models, which share parameter counts but differ in fine-tuning protocols, the largest effect sizes are observed, surpassing even those of the base 175B GPT-3 variant.

Figure 4: Top—Effect size vs. model parameters for Care/Harm, Fairness/Cheating, etc.; Bottom—MFH-Score vs. model parameters, with marked correlation across models and families.

This suggests that model architecture and post-training procedures (e.g., RLHF, SFT) mediate the emergence and fidelity of moral mimicry, in addition to scale.

Prompt and Dataset Robustness

Effects are robust across both prompt templates and datasets, including the variously sourced Moral Stories, ETHICS, and Social Chemistry data. The directionality and magnitude of political identity effect sizes persist, with only minor deviations in less-salient foundations or atypical prompt/dictionary combinations.

Qualitative Output Evolution

The paper provides qualitative output slices (randomly sampled) across GPT-3 model scales, demonstrating that as model size increases, moral rationalizations become more contextually attuned, nuanced, and lexically competent.

Figure 5: Illustrative completions for prompts under different model sizes, evidencing increasingly sophisticated and foundation-relevant rationalizations.

Implications and Theoretical Considerations

The results establish that LLMs function as conditional simulators of group-typical moral rationalizations, not only echoing lexical cues but capturing deeper alignment with documented human group differences. This has implications for generative systems embedded in digital discourse: adversarial or deliberate prompting may be used to surface group-congruent or polarizing moral rhetoric at scale.

For computational social science, these models offer a living, high-fidelity probe for attitudinal distributions—but underscore the need for careful conditioning and interpretive guardrails given their training-data aggregation effects.

The variances with human consensus reinforce a well-recognized risk: LLMs are not perfect moral agents, and dictionary-based evaluation, though tractable, is blunt—future work must turn to more sophisticated, possibly neural, foundation detectors and incorporate continual human-in-the-loop evaluation ([gartenMoralityLinesDetecting2016], [royFewShotIdentificationMorality2022]).

Limitations

• Use of dictionary methods for foundation detection—limited nuance, possible bias.
• Evaluation datasets are U.S.-centric; cross-cultural generalization is untested here, but directly motivated.
• Lack of model interpretability regarding the mechanisms of moral mimicry—opacity leaves open questions concerning causal drivers and potential levers for intervention.
• API-accessible models condition downstream reproducibility and limit fine-grained mechanistic analysis.

Conclusion

The paper provides strong empirical and quantitative evidence that LLMs tailored with political identity labels generate distinct, foundation-specific moral rationalizations consonant with canonical sociopolitical group biases. The magnitude of this "moral mimicry" scales with model size and is modulated by both architecture and training regimes, with prompt and data selection exerting secondary influence. These findings have immediate consequences for the deployment, governance, and interpretation of LLM-based systems in sociotechnical contexts.

Future AI developmental priorities should incorporate explicit mechanisms for managing subgroup alignment, transparency into conditional simulation capabilities, and systematic investigation of emergent behaviors in other axes of social cognition. In addition, establishing culturally aware and human-in-the-loop systems for moral content evaluation will be critical as LLMs become further entangled in public-facing applications.