DeLLMa: A Framework for Decision Making Under Uncertainty with Large Language Models

Author:

Ollie Liu, Deqing Fu, Dani Yogatama, Willie Neiswanger

Keyword:

Computer Science, Artificial Intelligence, Artificial Intelligence (cs.AI), Computation and Language (cs.CL), Machine Learning (cs.LG)

journal:

date:

2024-02-04 00:00:00

Abstract

Large language models (LLMs) are increasingly used across society, including in domains like business, engineering, and medicine. These fields often grapple with decision-making under uncertainty, a critical yet challenging task. In this paper, we show that directly prompting LLMs on these types of decision-making problems yields poor results, especially as the problem complexity increases. To overcome this limitation, we propose DeLLMa (Decision-making Large Language Model assistant), a framework designed to enhance decision-making accuracy in uncertain environments. DeLLMa involves a multi-step scaffolding procedure, drawing upon principles from decision theory and utility theory, to provide an optimal and human-auditable decision-making process. We validate our framework on decision-making environments involving real agriculture and finance data. Our results show that DeLLMa can significantly improve LLM decision-making performance, achieving up to a 40% increase in accuracy over competing methods.

PDF: DeLLMa: A Framework for Decision Making Under Uncertainty with Large Language Models.pdf