Application of Large Language Models and ReAct prompting in Policy Evidence Collection

Policy analysis or formulation often requires evidence-based support to ensure the scientific rigor and rationality of the policy, increase public trust, and reduce risks and uncertainties. However, manually collecting policy-related evidence is a time-consuming and tedious process, making some automated collection methods necessary. This paper presents a novel approach for automating policy evidence collection through large language models (LLMs) combined with Reasoning and Acting (ReAct) prompting. The advantages of our approach lie in its minimal data requirements, while ReAct prompting enables the LLM to call external tools, such as search engines, ensuring real-time evidence collection. Since this is a novel problem without existing methods for comparison, we relied on human experts for ground truth and baseline comparison. In 50 experiments, our method successfully collected correct policy evidence 36 times using GPT-3.5. Furthermore, with more advanced models such as GPT-4o, the improved understanding of prompts and context enhances our method's efficiency. Finally, our method using GPT-4o successfully gathered correct evidence 45 times in 50 experiments. Our results demonstrate that, using our method, policy researchers can effectively gather evidence to support policy-making.