Summary
It is hard to make sure automated data engineering tasks are done correctly and can be repeated exactly. The authors tested language model programs that run on regular computers to see if they can create correct and repeatable data workflows involving tasks like data cleaning, enriching, and visualization. They found these models do much better when allowed to check and fix their own work step-by-step rather than just doing it once. Larger models produced higher success rates, and smaller, optimized versions also worked well using little memory. This shows local language model assistants can support many practical data engineering tasks if the tasks can be checked for correctness.
What this means in practice
- •For data engineering teams: Use local language model agents to produce and verify reproducible data processing pipelines without relying on cloud models.
- •For software developers: Integrate local open-weight language models for automated code generation and repair in software-intensive data workflows.
Authors
Jorge García-Carrasco, Javier Sanchis, Alejandro Reina-Reina, Alejandro Maté, Juan Trujillo
Abstract
Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasize textual responses or isolated code generation rather than the validity of complete engineering artifacts. Objectives: We evaluate whether local LLM agents can produce correct and reproducible data-engineering artifacts, quantify the effect of a closed-loop workspace condition, and examine trade-offs in model scale, architecture, quantization, runtime, tool use, and failure. Methods: We introduce a benchmark of fifteen mobility-workflow tasks covering data discovery, connectors, transport-feed processing, semantic enrichment, feature engineering, validation, visualization, and reporting. Deterministic checkers assess generated scripts, tables, structured files, figures, and reports. Ten local configurations are evaluated in one-shot and closed-loop conditions, with five repetitions per model, mode, and task, yielding 1,500 scored attempts on a consumer-grade GPU. Results: Among models larger than two billion parameters, the workspace condition increases pass rates by 26.7-52.0 percentage points over one-shot generation. The strongest configuration reaches 85.3% artifact-level success, and a quantized 9-billion-parameter model reaches 69.3% with an approximately 6.5 GB memory footprint. Gains are largest when intermediate artifacts expose errors the agent can inspect and repair. Conclusion: Local open-weight agents can support a meaningful subset of software-intensive data-engineering work, but reliability depends on model capability, task verifiability, and deterministic validation. The benchmark provides a reproducible method for evaluating complete agent configurations before adoption in engineering workflows.