Why Generating Unit Test Cases is Important

In 1996, the Ariane 5 rocket exploded just 37 seconds after launch. The cause? A single line of unhandled code that wasn’t properly tested. Cost: $440 million and years of work, gone in an instant.

Your COBOL programs might not launch rockets, but they do something just as critical, process payrolls, handle millions of financial transactions, and manage inventory worth billions. A single untested code change can cascade into a catastrophe.

Testing isn’t a luxury. It’s a safety net. But for decades-old COBOL applications, creating and maintaining test cases has always been one of the hardest parts of modernization.

COBOL Check

In our previous post, we discussed how COBOL Check modernizes COBOL unit testing by introducing a DSL-based framework that brings test-driven development (TDD) principles to the mainframe. After experimenting with it, we were impressed by its flexibility and the way it integrates seamlessly into modern CI/CD workflows.

However, one big challenge remained – you still have to write the test cases.

Even though Cobol Check’s syntax is straightforward, becoming familiar with its DSL and manually writing .cut files still requires time and effort. Why not leverage AI to automatically generate .cut files that can be fed directly into COBOL Check?

Our Two-Step Approach to Intelligent Test Generation

We developed a two-step process that combines large language models (LLMs), automated validation, and COBOL Check execution to create a closed-loop testing workflow:

Intelligent Test Generation and Validation
Test Execution with Cobol Check

Step 1: Intelligent Test Generation and Validation

We start by providing the large language model with the source COBOL program and, if available, supporting documentation. The model then generates COBOL Check compatible unit test files.

Command to Generate Test Cases:

python Generate_tests.py --src-path /program_path/CUTGENS1.CBL \
--output-path /Output_path \
--section-name Multiplication-section \ #optional
--description /description_path/CUTGENS1.txt #optional

Console Output:

======================================================================
Generating Unit Tests for CUTGENS1
Scope: All sections
Source: /program_path/CUTGENS1.CBL
Model: gpt-4.1
Iterations: 1
======================================================================

Sub-program MAIN-SECTION written to 1_main-section.txt
Sub-program ADDITION-SECTION written to 2_addition-section.txt
Sub-program SUBTRACTION-SECTION written to 3_subtraction-section.txt
Sub-program MULTIPLICATION-SECTION written to 4_multiplication-section.txt
Syntax validation passed.
Variables validation passed.
Unit Tests saved to /Output_path/UnitTests/CUTGENS1/CUTGENS1.cut

======================================================================
✅ Test generation completed successfully!
======================================================================

COBOL programs can easily span thousands of lines across many sections. Often, when refactoring a single section, we only need to test that part. To make this scalable, we split the source program by sections (default) or by divisions, allowing the model to focus on generating tests for one logical unit at a time.

Our Source Program:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. SIMPLE-MATH.

       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01  NUM1        PIC S9(9) VALUE 50.
       01  NUM2        PIC S9(9) VALUE 50.
       01  WS-ADD      PIC S9(9).
       01  WS-SUB      PIC S9(9).
       01  WS-MUL      PIC S9(18).

       PROCEDURE DIVISION.
       MAIN-SECTION.
           PERFORM ADDITION-SECTION
           PERFORM SUBTRACTION-SECTION
           PERFORM MULTIPLICATION-SECTION
           DISPLAY "NUM1 = " NUM1 "  NUM2 = " NUM2
           DISPLAY "WS-ADD = " WS-ADD
           DISPLAY "WS-SUB = " WS-SUB
           DISPLAY "WS-MUL = " WS-MUL
           STOP RUN.

       ADDITION-SECTION.
           COMPUTE WS-ADD = NUM1 + NUM2
           EXIT.

       SUBTRACTION-SECTION.
           COMPUTE WS-SUB = NUM1 - NUM2
           EXIT.

       MULTIPLICATION-SECTION.
           COMPUTE WS-MUL = NUM1 * NUM2
           EXIT.

We don’t use the generated tests right away, they first go through our ANTLR-based COBOL Check test-suite parser, which checks for:

- Syntax errors – makes sure the test follows Cobol Check format.

- Variable references – confirms all variables in the test exist in the source program.

- Duplicates – removes repeated or redundant test cases.

If errors are found, the parser flags them, and the model regenerates the invalid tests. This process repeats through iterative refinement until all tests are valid.

Validated Test Cases

TESTSUITE 'SIMPLE-MATH multiplication tests'

BEFORE-EACH
              MOVE 50 TO NUM1
              MOVE 50 TO NUM2
              MOVE 0 TO WS-MUL
END-BEFORE

TESTCASE 'Multiply default positive values'
    PERFORM MULTIPLICATION-SECTION
    EXPECT WS-MUL TO BE 2500

TESTCASE 'Multiply positive by zero'
    MOVE 0 TO NUM1
    PERFORM MULTIPLICATION-SECTION
    EXPECT WS-MUL TO BE 0

TESTCASE 'Multiply negative by positive'
    MOVE -10 TO NUM1
    MOVE 30 TO NUM2
    PERFORM MULTIPLICATION-SECTION
    EXPECT WS-MUL TO BE -300

TESTCASE 'Multiply positive by negative'
    MOVE 22 TO NUM1
    MOVE -7 TO NUM2
    PERFORM MULTIPLICATION-SECTION
    EXPECT WS-MUL TO BE -154
...

Output Directory Layout :

Output_path/
├── split/               # Folder containing program splits
│   └── CUTGENS1/
│       ├── 1_main-section.txt
│       ├── 2_addition-section.txt
│       ├── 3_subtraction-section.txt
│       └── 4_multiplication-section.txt
├── UnitTests/           # Folder containing generated test suites
│   └── CUTGENS1/
│       └── CUTGENS1.cut

Step 2: Test Execution with Cobol Check

Once the generated .cut files pass validation, we feed them with COBOL Check using the configured properties.

Cobol Check executes the tests, compares actual and expected outputs, and produces detailed reports.

Running COBOL Check:

python Run_tests.py --src-path /program_path/CUTGENS1.CBL /
--config-file /config.properties 
--cobolcheck-path /cobol-check-0.2.19

Console Output:

======================================================================
Running Unit Tests for CUTGENS1
======================================================================
the cobolcheck path is  /cobol-check-0.2.19
📄 Test file: CUTGENS1.cut
======================================================================

⚙️  Executing CobolCheck...

======================================================================
📊 TEST RESULTS SUMMARY
======================================================================
Total Test Cases:  39
✅ Passed:         39
❌ Failed:         0
⚠️  Not Mocked:     0
======================================================================
✅ Test execution completed!
======================================================================

Conclusion

This setup makes COBOL testing faster, safer, and easier to maintain. By combining AI-generated test cases, automated validation, and COBOL Check execution, we’ve built a practical way to modernize COBOL testing without changing the codebase itself.

This approach helps teams save time, reduce human error, and bring the reliability of modern testing to long-standing COBOL systems.

Many thanks again also to Adikhan Tuichiyev, who’s master thesis set the base for this tool!

Interested in this solution? Feel free to contact us!

Transparency Note
This blog post was drafted with the support of generative AI to help structure and formulate the content. However, the technical background was thoroughly researched by our team beforehand, and we consider the topic highly relevant and worth sharing. The final content has been carefully reviewed and approved by us before publication.

CutGEN – Intelligent Unit Test Generator