Abstract
Real-world human decision-making often relies on strategic planning, wherehigh-levelgoals guide the formulation of sub-goals and subsequent actions, as evidenced by domains such as healthcare, business, and urban policy. Despite notable successes in controlled settings, conventional reinforcement learning (RL) follows abottom-upparadigm, which can struggle to adapt to real-world complexities such as sparse rewards and limited exploration budgets. While methods like hierarchical RL and environment shaping provide partial solutions, they frequently rely on either ad-hoc designs (e.g. choose the set of high-level actions) or purely data-driven discovery of high-level actions that still requires significant exploration. In this paper, we introduce atop-downframework for RL that explicitly leverageshuman-like strategyto reduce sample complexity, guide exploration, and enable high-level decision-making. We first formalize theStrategy Problem, which frames policy generation as finding distributions over policies that balancespecificityandvalue. Building on this definition, we propose theStrategistagent—an iterative framework that leverages large language models (LLMs) to synthesize domain knowledge into a structured representation of actionable strategies and sub-goals. We further develop areward shaping methodologythat translates these strategies expressed in natural language into quantitative feedback for RL methods. Empirically, we demonstrate a significantly faster convergence than conventional PPO. Taken together, our findings highlight thattop-down strategic explorationopens new avenues for enhancing RL on real-world decision problems.