Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications (2402.07927v1)

Published 5 Feb 2024 in cs.AI, cs.CL, and cs.HC

Abstract: Prompt engineering has emerged as an indispensable technique for extending the capabilities of LLMs and vision-LLMs (VLMs). This approach leverages task-specific instructions, known as prompts, to enhance model efficacy without modifying the core model parameters. Rather than updating the model parameters, prompts allow seamless integration of pre-trained models into downstream tasks by eliciting desired model behaviors solely based on the given prompt. Prompts can be natural language instructions that provide context to guide the model or learned vector representations that activate relevant knowledge. This burgeoning field has enabled success across various applications, from question-answering to commonsense reasoning. However, there remains a lack of systematic organization and understanding of the diverse prompt engineering methods and techniques. This survey paper addresses the gap by providing a structured overview of recent advancements in prompt engineering, categorized by application area. For each prompting approach, we provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized. We also delve into the strengths and limitations of each approach and include a taxonomy diagram and table summarizing datasets, models, and critical points of each prompting technique. This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022.
  2. Language models are few-shot learners, 2020.
  3. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022.
  4. Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310.14735, 2023.
  5. Contrastive chain-of-thought prompting. arXiv preprint arXiv:2311.09277, 2023.
  6. Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint arXiv:2311.04205, 2023.
  7. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495, 2023.
  8. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246, 2023.
  9. Chain-of-symbol prompting elicits planning in large langauge models, 2023.
  10. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  11. Large language models understand and can be enhanced by emotional stimuli. arXiv preprint arXiv:2307.11760, 2023.
  12. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint arXiv:2312.04474, 2023.
  13. Structured chain-of-thought prompting for code generation. arXiv preprint arXiv:2305.06599, 2023.
  14. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources, 2023.
  15. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
  16. Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291, 2023.
  17. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.
  18. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014, 2023.
  19. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  20. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313, 2024.
  21. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  22. Chain-of-table: Evolving tables in the reasoning chain for table understanding, 2024.
  23. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  24. System 2 attention (is something you might need too). arXiv preprint arXiv:2311.11829, 2023.
  25. Visual chatgpt: Talking, drawing and editing with visual foundation models, 2023.
  26. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  27. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
  28. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
  29. Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2305.16582, 2023.
  30. Chain-of-note: Enhancing robustness in retrieval-augmented language models, 2023.
  31. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022.
  32. Enhancing zero-shot chain-of-thought reasoning in large language models through logic, 2023.
  33. Take a step back: evoking reasoning via abstraction in large language models. arXiv preprint arXiv:2310.06117, 2023.
  34. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
  35. Thread of thought unraveling chaotic contexts. arXiv preprint arXiv:2311.08734, 2023.
Citations (133)
View on Semantic Scholar

Summary

  • The paper presents a comprehensive taxonomy of prompt engineering techniques that extend LLM capabilities without retraining underlying models.
  • It details advanced methods like chain-of-thought prompting and retrieval augmented generation to enhance reasoning and reduce hallucinations.
  • It outlines future directions, including meta-learning approaches and ethical considerations, to guide next-generation prompt engineering research.

A Systematic Survey of Prompt Engineering in LLMs: Techniques and Applications

Introduction

Prompt engineering presents itself as a seminal technique in the enhancement and adaptation of pre-trained LLMs and vision-LLMs (VLMs) across a broad spectrum of tasks without modifying their underlying parameters. This paper compiles a comprehensive survey of the field, delineating the evolution, methodologies, and applications of prompt engineering, stretching from zero-shot and few-shot learning to complex reasoning methodologies. This detailed overview addresses the pressing need for a systematic organization within this burgeoning field, aiming to pave the way for future research by identifying both current strengths and potential gaps.

Overview of Prompt Engineering Techniques

The taxonomy introduced in this survey categorizes prompt engineering techniques into several critical areas, emphasizing their application domains and contributions to advancing LLM capabilities.

New Tasks Without Extensive Training

  • Zero-shot prompting, leveraging the innate knowledge of models to tackle tasks without specific training, and Few-shot prompting, which requires minimal examples, signify foundational techniques that extend model applicability without significant data or computational overheads.

Reasoning and Logic

  • Advanced techniques like Chain-of-Thought (CoT) prompting showcase how models can be guided to generate step-by-step reasoning, replicating a thought process for complex problem-solving tasks. Innovations like Automatic Chain-of-Thought (Auto-CoT) and Self-Consistency further refine this idea, introducing automation and variety in reasoning generation to enhance performance and reliability.

Reducing Hallucination

  • Techniques such as Retrieval Augmented Generation (RAG) and Chain-of-Verification (CoVe) aim to curb the issue of hallucination in outputs by augmenting prompts with retrieval mechanisms and verification steps, leading to more accurate and reliable responses.

Code Generation and Execution

  • The survey explores the domain of code generation, highlighting approaches like Scratchpad Prompting, Structured Chain-of-Thought (SCoT), and Chain-of-Code (CoC), which elucidate methods to enhance the precision and logical flow in code-related tasks.

Managing Emotions and Tone

  • A novel area explored is the management of emotions and tone through prompting, illustrating the method's versatility beyond technical applications to include human-like understanding and generation of content.

Theoretical and Practical Implications

This survey not only underscores the practical successes achieved across various tasks, including language generation, reasoning, and code execution, but also explores the theoretical understanding of how different prompt engineering methodologies can influence model behavior. It highlights the dual benefit of prompt engineering: enhancing model performance while providing insights into model cognition and decision-making processes.

Future Directions in Prompt Engineering

The comprehensive analysis in this survey pinpoints several future directions, including the exploration of meta-learning approaches for prompt optimization, the development of hybrid models combining different prompting techniques, and the imperative need for addressing ethical concerns. The ongoing efforts to mitigate biases, enhance factual accuracy, and improve the interpretability of models through advanced prompt engineering methodologies are critical focal points for future research.

Conclusion

In conclusion, this survey not only serves as a critical resource for researchers and practitioners exploring the field of prompt engineering but also lays down a roadmap for future investigations. By systematically organizing and analyzing the plethora of existing techniques and their applications, it brings to light the immense potential of prompt engineering in harnessing the capabilities of LLMs and VLMs, advocating for a balanced approach towards its development, with an eye toward ethical and responsible AI use.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube