A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians

Evaluating the Use of LLMs as Creativity Support Tools for Comedy Writing

The paper "A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians" by Mirowski et al. addresses the potential and limitations of LLMs in the domain of comedy writing. The researchers conducted an empirical study with twenty professional comedians to assess the utility and drawbacks of using LLMs like ChatGPT and Bard as tools for generating humorous content. The study incorporated structured comedy-writing sessions, surveys evaluating the creativity support provided by LLMs, and focus group discussions to understand the participant's perceptions on various aspects of using AI in comedy. The findings from the study expose significant challenges and considerations in adopting LLMs for creative applications, particularly in a culturally sensitive and context-dependent domain such as comedy.

Study Design and Methodology

The study was carefully structured to capture a broad spectrum of comedians' experiences and attitudes towards using LLMs. The participants included professional comedians who were familiar with AI tools. The study involved:

1. Comedy Writing Sessions: Participants engaged in writing comedy sketches using LLMs. They were encouraged to generate material they would be comfortable presenting in live shows, focusing on setups, structures, and joke generation.
2. Surveys:

• Quantitative Evaluation: Utilizing a Likert scale and the Creativity Support Index (CSI) to measure various dimensions like helpfulness, expressiveness, immersion, and collaboration facilitated by the LLMs.
• Qualitative Questions: Open-ended questions to capture participants' direct experiences and thoughts on AI-generated content.

Focus Group Discussions: This qualitative component explored deeper insights into the participants' creative processes, ethical considerations regarding AI use, moderation and censorship concerns, and the perceived effectiveness of LLMs in producing high-quality comedic material.

Quantitative Findings

The quantitative results, as summarized by the Creativity Support Index, highlighted a mediocre performance of the LLMs in aiding comedy writing with an average score of 54.6 out of 100. Notably:

• Comedians who had previously used AI tools or performed AI-written material rated the LLMs more positively.
• Users who generated content in multiple languages also reported a higher CSI.

Despite some participants finding the AI-generated content helpful for preliminary drafts and structuring, the general consensus was that the output lacked the uniqueness and expressivity required for high-quality comedy. The participants did not feel substantial ownership or pride over the AI-generated material, indicating that the current capabilities of LLMs are limited when it comes to producing engaging and authentic comedic content.

Qualitative Insights

The focus group discussions provided nuanced perspectives on the limitations and potential of LLMs in comedy writing. Several critical themes emerged:

1. Quality and Blandness of Generated Outputs: Participants overwhelmingly characterized the AI-generated material as uninspired and generic. They noted the difficulty in steering LLMs towards producing specific or interesting content that felt fresh and original.
2. Creative Agency and Moderation: The intrinsic moderation and censorship imposed by current LLMs were seen as significant hindrances. Comedians expressed frustration over the tools' inability to handle offensive or edgy humor, which is often a staple in the comedy domain. There was a strong preference for comedians to have control over the moderation process, rather than relying on the AI's pre-tuned safety filters.
3. Marginalization and Representation: The default outputs of LLMs were perceived to reinforce hegemonic viewpoints, often sidelining minority perspectives. Attempts to generate content involving non-dominant identities were met with superficial adjustments, revealing the models' inability to authentically capture diverse cultural nuances.
4. Contextual Understanding: Effective comedy relies heavily on context—personal experiences, audience demographics, and cultural settings. Participants highlighted the fundamental limitation of LLMs in lacking situational awareness and the ability to incorporate subtext, which are crucial elements in delivering humor.
5. Ownership and Ethical Concerns: While there was less concern over ownership due to the low quality of outputs, the ethical implications of training models on copyrighted material raised significant questions. The participants debated the balance between enhancing model performance and respecting intellectual property rights.

Discussion and Implications

The study's findings underline the complexity of aligning LLMs with the specific creative needs of comedians. The current global, one-size-fits-all approach towards fine-tuning these models, often based on broad criteria like being "Harmless, Helpful, and Honest" (HHH), proves inadequate for creative tasks that demand cultural sensitivity and contextual understanding. The authors suggest a pivot towards community-based value alignment, where AI tools are fine-tuned with data and feedback from specific user groups to better reflect their norms and values.

Additionally, incorporating more explicit relational contexts into LLM moderation mechanisms could mitigate the overly cautious filtering that currently stifles creativity. There is also a call for transparency and improvements in data governance to responsibly curate training datasets and empower artists in understanding and controlling the AI tools they use.

Conclusion

The paper by Mirowski et al. offers a comprehensive examination of the role and limitations of LLMs in comedy writing. The empirical insights gathered from professional comedians reveal that while LLMs can assist with preliminary drafting and structural support, they fall short in generating genuinely humorous and contextually rich material. Achieving effective AI-assisted creative tools requires addressing issues of cultural value alignment, context incorporation, and ethical data usage, paving the way for more nuanced and adaptable AI systems tailored to the creative industries.