RXSynth provides large Java runtime bug method pairs dataset
RXSynth: A Seed-Grounded Corpus for Studying Java Runtime-Exception Bugs
Software Engineering
Summary
Finding real buggy Java methods with runtime exceptions is hard because existing datasets are very small. The authors created RXSynth, a much bigger collection of Java method pairs made by mimicking real bug fixes in a controlled way. Their tests show the synthetic bug fixes act like real ones, making it useful to study how well different tools can detect and understand these bugs. They also found simple comparisons often reveal just superficial patterns rather than true bug understanding, suggesting more careful evaluation is needed.
What this means in practice
- •For java developers: Test and improve Java runtime-exception detection tools using a large validated synthetic method bug-fix dataset.
- •For machine learning engineers: Train and evaluate models that identify runtime exception fixes with greater realism by using RXSynth's controlled synthetic Java method pairs.
Authors
Parnian Eslamloo, Banani Roy
Abstract
Existing Java bug benchmarks offer limited method-level data for studying runtime-exception detection. Filtering Defects4J, Bugs.jar, Bears, and BugSwarm yields only 170 real buggy/fixed method pairs across seven unchecked exception types. We introduce RXSynth, a corpus of 4,275 synthetic Java method pairs generated from real bug/fix exemplars. We select the generator using structural alignment with real seeds and validate pairs using a two-stage patch-anchored procedure that identifies the labelled exception risk and checks whether the repair removes it. Independent JVM execution confirms patch-local validity for 84.2% of a stratified sample of 500 pairs. Provenance records enable evaluation without overlap between generation exemplars and test bugs. Structural analyses show good alignment within the represented size range, but compression of larger methods. We use RXSynth to compare isolated-method detection with pair discrimination when the repair is visible. Under provenance-controlled evaluation, supervised isolated-method ROC-AUC is about 0.55-0.64. Stronger zero-shot models reach about 69% accuracy but identify only 41.2% of repaired methods correctly. In contrast, fine-tuned encoders achieve 70.4-74.8% accuracy with both versions available. However, a simple token-growth rule achieves 84.7% expected accuracy, highlighting repair-direction cues. On 36 pairs without token growth, encoder accuracy falls to approximately chance, while GPT-5.5 and Gemini 3.1 Pro exceed 94%. RXSynth thus supports contrastive, patch-oriented, and context-aware studies; pairwise evaluations should control for provenance and directional surface cues before interpreting high accuracy as defect understanding.