Knowledge-Centric Agents for Workflow Generation
Artificial Intelligence
Summary
The authors focus on improving how computer programs create workflows for visual design tools like ComfyUI. They found that just turning text into code isn't enough because it misses the deeper understanding and reasoning needed to build effective workflows. To fix this, they taught their system to understand and work with knowledge at different levels, from big-picture strategies down to detailed steps. This approach helped their system make more diverse and well-structured workflows that work better than previous methods. Overall, the authors show that thinking about knowledge itself is key to making smarter workflow generators.
Authors
Zhendong Li, Lei Sun, Ruibo Ming, He Zhang, Danda Pani Paudel, Luc Van Gool, Jinjin Gu
Abstract
Workflow generation in visual creation systems such as ComfyUI demands not only syntactic accuracy but also expert-level reasoning over modular compositions. Existing large language model (LLM) approaches often treat this as a direct text-to-JSON generation task, struggling with structural brittleness and lacking the experiential knowledge required for effective design. We argue that successful workflow generation requires modeling knowledge itself, including its structure, hierarchy, and reasoning dynamics. To this end, we propose a knowledge-centric framework that learns to invert, inject, and infer with knowledge across multiple abstraction levels. We first perform knowledge inversion to distill hierarchical representations, ranging from full pseudo-codes and skeletons to high-level strategies, from large collections of real-world workflows. We then conduct knowledge injection through supervised fine-tuning, teaching the model to reason from task descriptions to strategies and from strategies to executable structures. During inference, the model performs reversible reasoning to synthesize executable workflows, augmented by self-refinement for structural coherence. Extensive experiments demonstrate that our method produces workflows with richer node diversity, more coherent structures, and higher execution success rates than existing systems, establishing a new foundation for knowledge-driven, agentic workflow generation.