FlowMind: Automatic Workflow Generation with LLMs (2404.13050v1)

Abstract: The rapidly evolving field of Robotic Process Automation (RPA) has made significant strides in automating repetitive processes, yet its effectiveness diminishes in scenarios requiring spontaneous or unpredictable tasks demanded by users. This paper introduces a novel approach, FlowMind, leveraging the capabilities of LLMs such as Generative Pretrained Transformer (GPT), to address this limitation and create an automatic workflow generation system. In FlowMind, we propose a generic prompt recipe for a lecture that helps ground LLM reasoning with reliable Application Programming Interfaces (APIs). With this, FlowMind not only mitigates the common issue of hallucinations in LLMs, but also eliminates direct interaction between LLMs and proprietary data or code, thus ensuring the integrity and confidentiality of information - a cornerstone in financial services. FlowMind further simplifies user interaction by presenting high-level descriptions of auto-generated workflows, enabling users to inspect and provide feedback effectively. We also introduce NCEN-QA, a new dataset in finance for benchmarking question-answering tasks from N-CEN reports on funds. We used NCEN-QA to evaluate the performance of workflows generated by FlowMind against baseline and ablation variants of FlowMind. We demonstrate the success of FlowMind, the importance of each component in the proposed lecture recipe, and the effectiveness of user interaction and feedback in FlowMind.

• The paper introduces a two-stage framework that dynamically generates workflows by integrating LLMs with structured API information.
• It employs a generic lecture prompt recipe—including context setting, detailed API descriptions, and explicit code prompting—to mitigate hallucinations and improve performance.
• User feedback and rigorous evaluation on the NCEN-QA dataset demonstrate enhanced accuracy and practical applicability in secure, data-sensitive industries.

An Expert Analysis of "FlowMind: Automatic Workflow Generation with LLMs"

The paper "FlowMind: Automatic Workflow Generation with LLMs" presents a novel approach to enhancing Robotic Process Automation (RPA) by leveraging the capabilities of LLMs to generate workflows dynamically, specifically addressing scenarios requiring spontaneous or unpredictable user tasks. The method combats common LLM issues, such as hallucinations, by grounding LLM reasoning with reliable Application Programming Interfaces (APIs). Additionally, the system implements mechanisms to ensure data privacy and integrates user feedback to refine the generated workflows—a critical feature for practical deployment in industries with stringent data security requirements, such as finance.

Methodology

The FlowMind framework is a two-stage process. The first stage involves "lecturing" the LLM on the context and available APIs using a generically crafted prompt recipe consisting of three components:

1. Context Setting: Introduces the overall task domain to the LLM.
2. API Descriptions: Provides structured and semantically meaningful descriptions of the available APIs, including function names, input arguments, and output variables.
3. Code Prompting: Explicitly instructs the LLM to write code using the described APIs upon receiving a user query or task.

In the second stage, FlowMind uses the information gained from the first stage to generate and execute workflow code dynamically in response to user queries. Importantly, the system incorporates a feedback loop whereby users can review and provide input on generated workflows, allowing the LLM to refine the workflows based on this feedback. This interaction ensures greater accuracy and usability of the generated workflows.

Empirical Evaluation

The paper introduces an extensive evaluation using a newly created dataset called NCEN-QA, derived from N-CEN reports on funds. The dataset is structured into three parts—NCEN-QA-Easy, NCEN-QA-Intermediate, and NCEN-QA-Hard—each varying in complexity from straightforward questions about specific fund information to more intricate queries requiring mathematical operations and aggregation across multiple funds.

For performance benchmarks, FlowMind was compared against a baseline method, GPT-Context-Retrieval, which is a common approach wherein an LLM retrieves contextually relevant information before answering queries. FlowMind, even without user feedback, significantly outperformed the baseline, demonstrating the effectiveness of the integration of APIs and structured workflow coding. Detailed accuracy results across all dataset complexities highlight the robustness of FlowMind, achieving close to or full accuracy in most cases.

Further, an ablation paper analyzed the importance of each component of the generic lecture recipe:

• FlowMind-NCT (No Context) lacked context setting.
• FlowMind-BA (Bad APIs) employed non-informative arguments for API descriptions.
• FlowMind-NCP (No Code Prompt) did not explicitly instruct the LLM to write code.

The results underscored the necessity of each component for the overall success of FlowMind. Particularly, the explicit code prompting showed critical significance, with FlowMind-NCP showing markedly lower accuracy.

Incorporation of user feedback further augmented the system's performance, demonstrating that user input allows for real-time corrections and refinement of workflows, driving accuracy to near-perfect levels across all datasets.

Practical and Theoretical Implications

The introduction of FlowMind holds several significant implications for both practical applications and theoretical development:

• Practical Implications: FlowMind offers a sophisticated solution for industries that demand high levels of data integrity and security, such as finance. By ensuring that LLMs do not interact directly with proprietary data, FlowMind circumvents potential data privacy issues—critical in highly regulated sectors. Additionally, the integration of user feedback positions FlowMind as a highly adaptable tool, capable of evolving based on user interaction, thereby ensuring practical usability and continuous improvement.
• Theoretical Implications: This work provides a foundational approach to combining LLMs with robust, domain-specific APIs to mitigate the hallucination problem inherent in LLMs. Furthermore, the clear delineation of the lecture recipe components and their empirical validation offers substantial ground for future research into how structured prompts and methodical context setting can optimize LLM task performance.

Future Directions

Several avenues for future research are suggested:

• Scalability: Investigating the scaling of user feedback mechanisms to enhance workflow precision at scale, potentially through crowdsourcing strategies.
• Lifelong Learning: Exploring the possibility of resolving past user-approved workflows for a continuous learning approach, thereby incrementally improving FlowMind's performance.
• Enhanced API Management: Developing methods to manage extensive libraries of APIs, selecting relevant ones for LLM tasks based on contextual embeddings.

In summary, the research presented in "FlowMind: Automatic Workflow Generation with LLMs" provides a compelling advance in the field of workflow automation, blending the robustness of domain-specific APIs with the adaptive capabilities of LLMs, and introduces a valuable new dataset for the evaluation of workflow generation systems in finance.