NeMo Data Designer enables easy creation of diverse synthetic multimodal datasets
NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation
Artificial IntelligenceComputation and Language
Summary
Creating synthetic data for training AI often requires many types of information like images, text, or codes, which can be tricky to produce together. The authors present NeMo Data Designer, a tool that lets users easily specify and generate complex datasets with multiple data types using a simple configuration system. It also supports testing small samples and fixing problems before creating large datasets, making data creation more efficient and reproducible. This tool can be extended with plugins to add new types of data and works with user-provided AI models during data generation.
What this means in practice
- •For enterprise data engineering teams: Generate tailored multimodal datasets to train and evaluate domain-specific AI models efficiently within production environments.
- •For ai product developers: Customize and scale synthetic datasets with mixed data types to improve model training for applications requiring text, images, and structured outputs.
Authors
Johnny Greco, Nabin Mulepati, Andre Manoel, Eric Tramel, Kirit Thadaka, Mike Knepper, Dhruv Nathawani, Dane Corneil, Yev Meyer, Alex Watson, Maarten Van Segbroeck
Abstract
We present NeMo Data Designer (NDD), an open-source, general-purpose framework for multi-modal synthetic data generation (SDG). Designed to be intuitive to use, NDD provides a declarative configuration format in which human and/or agent users define each dataset column, with column types spanning text, code, structured outputs, images, embeddings, and statistical samplers that are explicitly configured to steer dataset diversity. Additional column types and functionality can be introduced using the framework's flexible plugin system. NDD's configuration is an inspectable artifact, supporting workflow sharing and reproducibility. SDG is an inherently iterative process. NDD therefore builds a preview-and-revision loop into its core workflow, allowing users to generate and inspect a small number of records, refine the specification, and rerun generation at full scale. At runtime, NDD resolves dependencies, schedules calls to user-provided model endpoints, and retries failed requests. We describe NDD's architecture and programming model and present case studies spanning structured, agentic, multimodal, and domain-specialized tasks, including datasets used in Nemotron model development and in production enterprise deployments.