Papers for

ai-assisted software developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Large language models struggle with current chart-making languages

Where LLMs Fail with Visualization DSLs

Abstract: As LLMs take up the role of authoring charts using visualization domain-specific languages (DSLs), the human constraints that shaped those languages may no longer apply, as what is easy for a person is not necessarily easy for a model. To understand how LLMs might work better with DSLs, we explore where and how they fail with current DSL designs. We evaluate 10 JSON-style visualization DSLs with 41 tasks across 3 LLMs, then assess the generated specifications with JSON and rendering checks, and qualitative coding of failed cases. Analyzing how this specification generation process fails, we identify four recurring failure patterns, link each to specific DSL features, and discuss design considerations for future DSL designs.

Thu 1 OctHuman-Computer InteractionComputation and Language
The gist
Making charts with special computer languages is often designed to be easy for humans, but these languages can be tricky for large language models (LLMs). The authors tested three popular LLMs on 10 different chart-making languages with many tasks to see where the models make mistakes. They found four common types of errors and connected these problems to certain features of the languages. This helps understand how to design better chart languages that computers can use more effectively.
Open → 2610.01873v1