A short introduction to build systems
Most developers use a build system — often without realizing it. But if it were suddenly gone, the impact would be obvious. Since this article focuses heavily on build systems, let’s take a moment to introduce them.
Generally, when working with source code and applications a compiler is required to transform the source code into an executable binary that runs on the target CPU architecture. Not all languages are compiled but for simplicity we will only talk about compiled languages in this article.
When starting out with a new language the simple Hello World sample is typically built by directly invoking the compiler: javac HelloWorld.java or gcc -o hello hello.c
As the project grows, simply calling the compiler quickly becomes messy due to added files, frameworks, and interdependencies. With the introduction of continunous integration the requirement for reproducible builds became apparent. You could call the compiler inside a shell script. I’ll leave it to Google to explain why shell scripts don’t scale and shouldn’t be used in larger projects: Bazel | What about shell scripts?
That’s where build systems come in. That’s why build systems were created. The main idea being that the build of an application (with all its details, not just invoking the compiler) is a complex process which is handled by a separate program. All kinds of different build systems have been created over the years. They have different principles but the main idea is the same: Create something executable from configuration (source code, etc.). Most languages tend to use specific build systems because the community at some point agreed on a common standard.
The number of build systems is huge but common ones are Make (often used with C), Ant, Maven, Gradle (often used with Java and Kotlin), Rake (often used with Ruby), bit (for the JavaScript world). The list goes on and on. With monorepos gaining more popularity, other build systems have emerged that support building large application complexes that are stored inside the same repository.
What build systems support COBOL?
Building COBOL is very close to the C language in terms that you have a main source file which requires some source-level includes to be compiled. The final binary may also need other binaries linked to it because external logic is required for the execution of the program.
In practice, COBOL is built in one of two ways:
- A custom build system-like application has been developed at Mainframe shops ages ago. Most of them are written in CLIST or RexX, some even in COBOL
- A commercial product is purchased from a vendor (Dependency Based Build by IBM or Endevor by Broadcome – only to name a few)
Interestingly, none of the previously mentioned build systems are used in COBOL shops. One may find a developer using Make to build COBOL but I haven’t seen this used on a large scale and isn’t typical. I think this is because the open-source and Mainframe communities have only recently started to converge.
Because I have been working with Mainframe applications written in COBOL, Assembler, C and PL/I but also applications that may run on or off the Mainframe, I have seen both worlds. It always amazed me that the Mainframe world never reused existing tools—or created a shared standard. It’s all about self-written tools or commerical software. For some years this has bugged me and I felt a growing urge to see whether I could enable an existing build system to build COBOL applications as well.
Through my previous work on Android apps back in high school and Java development I have mostly used the Gradle build system. It was built for JVM-based languages but essentially doesn’t have any restrictions on what type of software it can build. It can be called a successfor of the Maven build system but the industry isn’t even remotely close to replacing Maven with Gradle. They both exist and server their purpose. COBOL is no exception to that! Gradle’s core functionality can be extended through plugins to support all kinds of languages and tasks. Let’s see how that works.
Developing a Gradle plugin
Though I nearly became a COBOL developer, I never quite got the hang of turning business requirements into applications. I quickly turned (mainly because my first, a very successful project around migrating a legacy build system) towards being more interested in the process of building a business application. Knowing Gradle and what is necessary to build a COBOL application through my various projects I wanted to fuse that knowledge together.
Gradle provides a flexible platform for tasks, extensions, and configuration—especially for JVM-based languages. By implementing a plugin that exposes tasks and extensions to the project it is possiblel to virtually build any language. The developer will work around build scripts which are build.gradle or build.gradle.kts.
A very simple example of a build script can be the following:
tasks.register("simpleGreeting") {
doLast {
println("Hello out there!")
}
}This script registers a simpleGreeting task, which you can run with ./gradlew simpleGreeting. This task only prints something to the command line. Such a task can also be provided with inputs and outputs, it can run other tools and interact with project resources. For instance, a task can be capable of calling a specific compiler which, if successful, places the result in a special output folder. Another plugin can package these outputs into an installable archive.
Gradle provides a large set of APIs to interact with the platform itself. For example, it offers a depdency tracking and resolution mechanism that tracks changes to other components for you. It will also try to shorten build times by only rebuilding what has changed. Again, this is a common requirement for large COBOL code bases. Typically, only parts of the application should be rebuild (the parts that changed during the implementation of new requirement) and the rest should stay untouched.
I have created a simple implementation of such a plugin. The plugin exposes one task for every COBOL modules that is located in the source directory of the repository in src/cobol. If used within an IDE a button is generated for each module so that it can be built with the click of a button.
Every task has a defined set of inputs:
- The main source file
- Source code include files (like copybooks)
- Configuration data (whether a program runs inside CICS or Db2, specific compiler options)
The task also defines the outputs:
- The binary
- Logs (we call them listings, mostly)
- For Db2 programs: a DBRM
In case any of the inputs change, Gradle knows that a task is out-of-date and must be rebuilt. It internally tracks the state of these inputs and outputs and in case they don’t match anymore, it will calculate what tasks must be re-run.
While it isn’t finished yet, there is a special case which shows the power that Gradle offers: BMS maps in the Mainframe space are used to describe the terminal screen for a CICS application – basically the old-school HTML. (I have to admit: Most web developers don’t write HTML anymore but you get the point.) A BMS map is a program itself which executes and it has an input and output data structure that others programs must know. This data structure is put into a copybook and generated during build. This creates a dependency between a BMS map and a regular COBOL program. In case the BMS map was changed, all COBOL programs that serve this map must be rebuilt because the interface structure may have changed. By properly defining the output of the BMS task as an input for the COBOL task, Gradle will be able to track this dependency automatically. The same can be applied for program calls. In case a sub-program was changed, the main module must be rebuilt (in case of static calls, at least).
The plugin in action
For demonstration I am using the sample Mortgage Application from IBM.
./gradlew tasks --group ibm-cobol
> Task :tasks
------------------------------------------------------------
Tasks runnable from root project 'ibm-mortgage-application'
------------------------------------------------------------
Ibm-cobol tasks
---------------
build-epscmort
build-epscsmrd
build-epscsmrt
build-epsmlis-map
build-epsmlist
build-epsmort-map
build-epsmpmt
build-epsnbrvlTasks can be executed directly via the CLI or inside the IDE.
./gradlew build-epscmort
> Task :build-epsmort-map UP-TO-DATE
> Task :build-epscmort
PP 5655-EC6 IBM Enterprise COBOL for z/OS 6.4.0 in progress ...
IGYOS4077-I DSNH4790I DSNHPSRV DSNHDECP HAS CCSID 500 IN EFFECT
IGYOS4078-W DSNH4791I DSNHPSRV CCSID 1140 IS USED TO PROCESS SQL,
BUT DSNHDECP HAS EBCDIC CCSID 500 IN EFFECT
IGYOS4078-W DSNH527I DSNHOPTS THE PRECOMPILER OR DB2 COPROCESSOR
ATTEMPTED TO USE THE DB2-SUPPLIED DSNHDECP MODULE.
IGYOS4077-I DSNH4760I DSNHPSRV THE DB2 SQL COPROCESSOR IS USING THE
LEVEL 2 INTERFACE UNDER DB2 V13
LineID Message code Message text
IGYDS0139-W Diagnostic messages were issued during processing of
compiler options. These messages are located at the
beginning of the listing.
Messages Total Informational Warning Error Severe Terminating
Printed: 5 2 3
End of compilation 1, program EPSCMORT, highest severity 4.
Return code 4
BUILD SUCCESSFUL in 16s
2 actionable task: 1 executed, 1 up-to-dateInterestingly, Gradle found two tasks, even though we only ran one. Let’s ask Gradle about it by showing us the task dependencies:
./gradlew build-epscmort taskTree --with-inputs --with-outputs
> Task :taskTree
------------------------------------------------------------
Root project 'ibm-mortgage-application'
------------------------------------------------------------
:build-epscmort
<- /home/david/dbb-zappbuild/samples/MortgageApplication/cobol/epscmort.cbl
<- /home/david/dbb-zappbuild/samples/MortgageApplication/build/generated-dsects/epsmort.cpy
<- /home/david/dbb-zappbuild/samples/MortgageApplication/copybook/epsmtcom.cpy
<- /home/david/dbb-zappbuild/samples/MortgageApplication/copybook/epsmtinp.cpy
<- /home/david/dbb-zappbuild/samples/MortgageApplication/copybook/epsmtout.cpy
<- /home/david/dbb-zappbuild/samples/MortgageApplication/copybook/epsnbrpm.cpy
-> /home/david/dbb-zappbuild/samples/MortgageApplication/build/epscmort
\--- :build-epsmort-map
-> /home/david/dbb-zappbuild/samples/MortgageApplication/build/generated-dsects/epsmort.cpy
Task inputs are shown in red and prefixed with <-
Task outputs are shown in green and prefixed with ->Two key details stand out:
epsmtinp.cpyandepsmtout.cpyare second-level copybooks which are included throughepsmtcom.cpy.epsmort.cpydoesn’t exist in the repository because it is created by the BMS mapbms/epsmort.bms. Gradle establishes a task dependency withbuild-epsmort-mapwhich produces the copybook as an output.
If any inputs or outputs change – even bms/epsmort.bms – the task is re-run.
What’s next?
First of all, thank you for reading up until here. In case you want to know more about the plugin and its (potential) options I would love to exchange abuot this. Fell free to reach out on LinkedIn or drop me an email.
We’re considering open-sourcing a polished version of the plugin for broader use. If you are interested in using such a plugin, please let us know.
There are many more feature I’d love to implement in the plugin. Just to name a few:
- Support for creating dependencies between applications (application A uses copybooks from application B)
- Support for creating deployable distributions which can be uploaded to an artifact repository
- Support for running unit tests with COBOL Check
- Support for more languages like High Level Assembler and PL/I
- Support for more COBOL compilers (like Micro Focus/Rocket, gCOBOL)
We believe this could help teams implement CI/CD pipelines for Mainframe applications. Reach out to us if you want to evaluate whether this is an option for you.
