Certifying test cases improves cobol to java migration accuracy
Certifying Repeatable Entry-Replay Witnesses for COBOL-to-Java Migration
Software EngineeringArtificial Intelligence
Summary
Testing that converted COBOL programs work the same as the original can be very hard. The authors developed a new method to create test cases that cover a lot of the original program's structure by mimicking external interactions and watching the program run. They applied this to a large COBOL program and checked the converted Java code matched the original's behavior on many test cases. While most errors like missing steps were caught, some subtle differences in how numbers were handled slipped through.
What this means in practice
- •For software migration teams: Generate thorough test cases for validating complex COBOL to Java application migrations to detect many code translation errors.
- •For enterprise software testers: Detect missing business logic steps in migrated code by using certified synthetic end-to-end tests that replay original program behavior.
Authors
Andras L. Ferenczi, Mariya Bessonov, Allwyn Ponnudurai, Chandrakanth R. Mamillapalli, Krishna K. Maddula, Krishna K. Lingamneni
Abstract
Producing synthetic data for comparison testing a migrated COBOL application is a major challenge that agentic migrations face today. Our prior paper, the Agentic Method for Deterministic Validation of Legacy Code Migration, presents a deterministic oracle to facilitate such migrations, the Locksmith Loop. We now propose a novel method for achieving high branch and paragraph coverage through generated synthetic end-to-end test cases, by means of entry-replay test-case certification by observing the code execution trace on a surrogate COBOL program that mocks external interactions via a chronological external-operation tape. We applied our method to a very large COBOL program with 33,429 lines of code to generate 849 synthetic end-to-end test cases covering 2,973/3,806 source branches (78.1%) and 989/1,033 main-AST paragraphs (95.7%). Using the Locksmith Loop's projected-agreement validation, the generated Java code matched the COBOL surrogate on the specified observations across all 849 certified synthetic end-to-end test cases. During the campaign we detected 61 defects and fixed the Java generator accordingly. To test whether the coding agent could silently introduce business logic bugs, we seeded 450 defects into the Java code, such as deleted writes, deleted paragraph calls, and altered rounding, to verify how many of these would be detected by the gating methodology. The gate detected 47.6%, including all deleted writes and paragraph calls in executed code, but was unable to detect the rounding and precision defects.