In this article, I review the research in the 2024 NeurIPS conference paper, From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection, by Wang et al. The paper explores integrating event analysis into LLM-based time-series forecasting models.
Real-time text such as news articles or social media posts are underutilized in the sphere of energy forecasting due to challenges posed when integrating the knowledge of these sources into traditional forecasting models. A practical and high performance method for integrating sources such as these could have profound implications in the ability for forecasting models to capture more of the context that results in future events beyond numerical input data. Wang et al hypothesize that inclusion of text sources contribute to forecasting performance by adding knowledge-rich context to observed events.
Capturing relevant data from qualitative information
While time series forecasting is a well-established research field, the application of LLMs remains relatively novel because of the recency for which these models have been available and also the subsequent methods for applying them in non-standard applications such as forecasting. The integration of event-based features in time series forecasting has been studied in the context of traditional models where input data is mostly or completely numerical. Few practical techniques have emerged since qualitative data such as text-based sentiments are uniquely diverse and any solution to integrate the distilled knowledge likely needs domain-specific solutions when modern LLMs hadn’t yet been introduced.
A key innovation is the progress of open-source LLMs to perform reasoning within the data, and the distillation process. Reasoning is the ability for a machine to perform complex decision-making using relevance from multiple sources of information. As an example, consider how the Superbowl impacts electricity demand during the event with demand spikes usually occurring in the time before the game and after every quarter. We foresee these events occurring because of our context of how the event affects demand. Now consider integrating that knowledge into a forecasting model when this event only occurs once a year, the task becomes more difficult.
The method described in the paper takes the following data as inputs to the model:
• Numerical “supplemental” data
• Unstructured text
Supplementary data would typically be passed to a forecasting model such as weather, time of day and other structured information. The unstructured text contains information that gets distilled into pre-prompt text in a process described later in this review. The ability to contextualize historical data by pairing it with event data allows the model to learn the causality behind observation reactions which is likely missing from typical structured data.
The model begins with information retrieval, refining search queries by region, time and task-specific keywords. These are organized into raw news, which gets passed to a reasoning agent and supplementary information which bypasses the reasoning agent. To generate structured numerical outputs, the LLM is fine-tuned on various forecasting tasks to accept prompt inputs and output formatted forecasts.
To effectively filter relevant news, the reasoning agent passes text data through several iterations of reasoning whereby prompts assist the agent in distilling the most critical information. Consider this like prompting ChatGPT to summarize an article and then asking it to refine the key points of the summary and then distilling the information from that article most critical to the prediction task. Multiple iterations allow only the relevant information to pass to the final prediction prompt. Finally, the reasoned news information and supplementary information is prepared as a single prompt to perform the forecasting.
To summarize, the LLM forecasting model is akin to ChatGPT except that it has been fine-tuned to specifically produce numerical forecasts from a text-based input. The text-based input is a prompt that contains all relevant information including the news that occurred recently, and the supplemental information to guide the model’s output.
Tests performed and outcomes
The authors tested the method on several time series datasets, including load data from the Australian Energy Market Operator (AEMO), and combined the inputs with regionalized and relevant news articles and supplementary information. Four different scenarios were tested to explore the model’s viability:
1. purely numerical inputs,
2. numerical inputs structured as descriptive text
3. descriptive text with unfiltered news, and
4. descriptive text with filtered news where the news is filtered by relevance to the location.
Across all time-series tasks, descriptive text including filtered news performed best at generating numerical forecasts. The news filtering model also outperformed other LLM time-series forecasting models. Also, the performance of the models significantly increased with each iteration of the news selection evaluation agent that refined the news information passed to the model.
Benefits to energy forecasting and other applications
This is welcome news (no pun intended) in the world of time series forecasting as forecasters have long identified the value of real-time news for its richness of information. However, without a comprehensive method of selecting, reasoning and integrating this information into a forecasting model, few comprehensive and practical methods for distilling this information into a forecast have been proposed without labour-intensive manual intervention.
Outside of electricity demand forecasting, this method could have significant applications to forecasting generation and prices where real-time relevant information can be quickly applied for greater accuracy.
Conclusion
The research conducted by Wang et al is innovative and exciting in the forecasting space. Future research could explore combining this approach with reinforcement learning for real-time adaptive models or expanding the scope to financial volatility and disaster response forecasting. Introducing multi-modal input data could improve the performance of price and demand forecasting, leading to wider adoption of LLM-based forecasting and ultimately allow granular context from news to assist with forecasting the effects of events.
To access From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection:
• On NeurIPS 2024, the Thirty-Eighth Annual Conference on Neural Information Processing Systems, Vancouver, B.C.
• On ArXiv
