Can LLMs Learn Macroeconomic Narratives from Social Media?

Can LLMs Learn Macroeconomic Narratives from Social Media?

Introduction

The study explores the feasibility and potential of using LLMs to extract macroeconomic narratives from social media (specifically Twitter) and evaluate their ability to improve macroeconomic forecasting. Focusing on the Narrative Economics hypothesis, the researchers aim to understand whether narratives disseminated through social media can drive economic fluctuations and be harnessed for predictive modeling.

Data Resources

The researchers curated two distinct datasets from Twitter (now known as X): one spanning from Twitter's inception to the onset of the COVID-19 pandemic, and another from late 2021 onward. These datasets were designed to capture economy-related narratives through posts discussing varied topics like politics, business, and current affairs, among others.

Additionally, the study targets three key macroeconomic indicators for prediction tasks:

1. Federal Funds Rate (FFR): Analyzed for daily changes, it acts as a barometer of economic stability and monetary policy effectiveness.
2. S&P 500: Reflects collective investor confidence and market performance.
3. Volatility Index (VIX): Measures market expectations of future volatility, often referred to as the 'fear gauge'.

Experimental Setup

The study focused on three prediction tasks: the next value, percentage change, and direction change of the financial indicators over three horizons (next-day, next-week, and next-month). The evaluation metrics used were Mean Squared Error (MSE) for regression tasks, and Accuracy and F1-Score for classification tasks. The models were rigorously compared against several financial and textual baselines to assess their predictive power.

Models and Methods

The models employed were divided into three categories based on their input signals:

1. Financial (F): Utilized historical financial data.
2. Textual (T): Leveraged narrative data from tweets or LLM-generated analyses.
3. Textual + Financial (TF): Combined both textual and financial inputs.

The study experimented with various representation techniques:

• Daily Sentiments: Aggregating sentiment scores from tweets.
• Embedding-Based Representations: Using embeddings from pre-trained language models (BERT, RoBERTa, T5) to encode tweets.
• LLM-Generated Analyses: GPT-3.5 was used to create narrative analyses of tweets combined with corresponding historical financial data.

Major Findings

Upon analyzing the datasets using these methods, several insights emerged:

1. Presence of Narratives: The study demonstrated the existence of significant economic narratives within the Twitter datasets. Temporal shifts in narrative token frequencies aligned with real-world events, indicating the potential of social media content for understanding economic sentiments.
2. Predictive Challenges:

• Sentiment-Based Predictions: Sentiment scores alone provided marginal improvements in prediction accuracy. Incorporating financial data largely overshadowed the textual sentiment's contributions.
• Embedding-Based Representations: Despite using advanced LLMs for embedding tweets, these models struggled to translate the narrative information into enhanced predictive signals for financial indicators.
• LLM-Generated Analyses: Although GPT-3.5 successfully extracted and summarized narratives, its direct use for predictions revealed inconsistencies. Summaries provided nuanced insights into economic contexts but showed limited added value over traditional financial data for macroeconomic forecasting.

Implications and Future Directions

The paper illuminates the complexity of leveraging social media narratives for macroeconomic predictions. Although narrative extraction using LLMs showed promise, incorporating this information into predictive models presented challenges. The marginal improvements observed suggest that while LLMs can learn macroeconomic narratives, translating these narratives into actionable predictive insights remains difficult.

Future work could focus on:

• Enhanced Representation Methods: Improving techniques to more effectively aggregate and encode diverse narratives from social media.
• Refined Prediction Models: Developing novel architectures tailored to integrating narrative data with financial indicators.
• Broader Economic Indicators: Testing additional macroeconomic variables or integrating microeconomic indicators to capture a more comprehensive economic picture.

Conclusion

The research provides an in-depth exploration into the Narrative Economics hypothesis through empirical testing with advanced NLP tools. The findings highlight the potential and limitations of LLMs in learning and leveraging macroeconomic narratives from social media for predictive purposes. The nuanced analyses suggest avenues for future exploration and underscore the need for innovative models to fully realize the value of narrative-driven financial forecasting.