Heavy Metal Debugging

By Jake Labelle on 27 April, 2021

Heavy Metal Debugging; Debugging and Reversing HLASM with TSO TEST


  • zOS: IBM's mainframe operating system
  • TSO: Time Sharing Option - Interactive access to zOS
  • PDS: zOS folder
  • HLASM: High Level Assembly - Assembly on z/Architecture
  • TSO TEST: Terminal Debugger pre-installed on zOS - not really meant for reverse engineering
  • AMBLIST: Batch program to map load modules and program objects
  • JCL: Job Control Language - used to submit batch programs
  • USS: Unix SubSystem - like wsl but on zOS


Reversing Engineering on zOS has some challenges - one of the biggest is attempting to get started.

The tools available on zOS do not lend themselves to reverse engineering, and are instead created by IBM to debug applications. The application I have chosen to use was TSO TEST, due to it being free (if you have access to zOS, which I do from a licensed copy of zPDT) and installed on every IBM mainframe (even TK4- the open source pre zOS from the 1980s).

IBM documentation is fairly thorough on how to use TSO TEST and HLASM. However, there is no quick start guide, unless your willing to spend ages trawling through IBM documentation; hopefully this will give you a quick start to reversing zOS applications.

Compiling & Running a C program on zOS

First thing required is a compiled program to reverse. We will be using the below dummy C program with a fairly obvious overflow vulnerability.

#include <stdio.h>
#include <string.h>

void special()
    printf ("H4CK3D TH3 M41NFR4M3");

int main()
    char buff[15];
    int pass = 0;

    printf("\n Enter the password : \n");

    if(strcmp(buff, "fsecure"))
        printf ("\n Wrong Password \n");
        printf ("\n Correct Password \n");
        pass = 1;


    return 0;

Now we have the C source code, we need to compile it. I use USS as I find it easier than submitting batch JCLs but either will work. The object file is compiled to a MVS PDS dataset.

c89 -o "//'JAKE.TSOTEST.LOADE(OVERFLOW)'" overflow.c

The below is the job to compile, bind and run the C program from batch.

//* Compile and bind step
//* Run step

Now from TSO, you can call the program.

  Enter the password : 
  Wrong Password

To do some static analysis of this program it is possible to use AMBLIST, again you can run this in batch, but there is a USS command which is simpler.

echo " LISTLOAD OUTPUT=MAP MEMBER=(OVERFLOW)" | amblist "//'JAKE.TSOTEST.LOADE'" > /tmp/overflow_amblist

Again, the below JCL does the same thing but from batch.


It's incredibly verbose, but here is the snippet of the important information, such as what external functions are being used, and the location of the function in the compiled overflow.c binary; SPECIAL and MAIN.

-**    END OF MAP AND CROSS-REFERENCE LISTING                                                                            
1                                  *  M O D U L E   S U M M A R Y  *                                             
0    MEMBER NAME:  OVERFLOW                                                               MAIN ENTRY POINT:    00000000  
0    LIBRARY:      SYSLIB                                                                 AMODE OF MAIN ENTRY POINT: 31  
-                 CONTROL SECTION                                        ENTRY                                           
                   LMOD LOC     NAME      LENGTH  TYPE  RMODE             LMOD LOC  CSECT LOC      NAME                                
                       A0    @ST00001       348   SD    31                                                              
                                                                              138        98      SPECIAL                 
                                                                              1B8       118      MAIN
                       338    CEEMAIN         0C   SD    31                                                                        
                      15A8    gets            0A   SD    31                                                              
                                                                             15A8        00      GETS                            
0                     15B8    printf          0A   SD    31                                                              
                                                                             15B8        00      PRINTF                                                    

Now to start the actual debugging; from TSO run:


You are now in the TEST terminal. Type 'end' to leave the terminal. If you're stuck in an instruction you might need to press PA1 to cancel that instruction first.


Below is an example of what TSO test looks like.


There are a number of ways to refer to addresses in TEST:

  • 15r - address in register 15
  • OVERFLOW.MAIN - using symbols
  • +12 - bytes relative to the base address (use qualify to change the base address). Starts at the entry point
  • 1FAA12F8 - the absolute address

There are a number of TEST sub commands you should know:

  • LIST <ADDRESS> <DATA_TYPE> m(<multiple>)
    important data types i (instruction), b (binary), x (hex), c (character), use m to say how many of them you want to show
    lists PSW, allows you to see the condition flag
    set a break point at either one address or multiple address, with multiple addresses they all have to be instructions
    remove breakpoints
  • GO
    runs the program until a breakpoint
    changes base address
    outputs current location and what function you are in

Reversing Walk-through

Finding the Password

Below is what the registers in HLASM are typically used for

  • Register 1 → parameter list pointer
  • Register 13 → pointer to register save area provided by caller
  • Register 14 → return address
  • Register 15 → address of subroutine

First,  we need to set a breakpoint at the MAIN function and jump to it. Let's also set the base address to this as well. This is done with:


Let's see what instruction we are going to run next:

list +0 i
        +0    BC      15,36(,R15)

TSO TEST unhelpfully shows decimal when showing the instruction, but has the addresses in hex. But this says branch on condition, mask 1111 (always) to 0x24 + the contents of register 15. So lets set the base address +24 from the start of main, and list all the assembly instruction in main.

qualify +24
list +0 i m(200)
        +0    STM     R14,R5,12(R13)
        +4    L       R14,76(,R13)
        +8    LA      R0,208(,R14)
        +C    CL      R0,788(,R12)
       +10    BRC     2,*-32
       +14    L       R15,640(,R12)
       +18    STM     R15,R0,72(R14)
       +1C    MVI     0(R14),16
       +20    ST      R13,4(,R14)
       +24    LR      R13,R14
       +26    LARL    R3,*+210
       +2C    LARL    R5,*+216
       +32    MVHI    176(R13),0
       +38    L       R15,0(,R3)
       +3C    LA      R0,22(,R5)
       +40    LA      R1,152(,R13)
       +44    ST      R0,152(,R13)
       +48    BASR    R14,R15
       +4A    LA      R0,160(,R13)
       +4E    L       R15,4(,R3)
       +52    LA      R1,152(,R13)
       +56    ST      R0,152(,R13)
       +5A    BASR    R14,R15
       +5C    LA      R2,160(,R13)
       +60    LA      R1,48(,R5)
       +64    LA      R0,0
       +68    CLST    R2,R1
       +6C    LA      R0,0
       +70    ST      R2,180(,R13)
       +74    ST      R1,184(,R13)
       +78    ST      R0,188(,R13)
       +7C    BRC     8,*+30
       +80    L       R2,180(,R13)
       +84    L       R1,184(,R13)
       +88    LLC     R0,0(,R2)
       +8E    LLC     R1,0(,R1)
       +94    SLR     R0,R1
       +96    ST      R0,188(,R13)
       +9A    L       R0,188(,R13)
       +9E    LTR     R0,R0
       +A0    BRC     8,*+26
       +A4    L       R15,0(,R3)
       +A8    LA      R0,56(,R5)
       +AC    LA      R1,152(,R13)
       +B0    ST      R0,152(,R13)
       +B4    BASR    R14,R15
       +B6    BRC     15,*+28
       +BA    L       R15,0(,R3)
       +BE    LA      R0,76(,R5)
       +C2    LA      R1,152(,R13)
       +C6    ST      R0,152(,R13)
       +CA    BASR    R14,R15
       +CC    MVHI    176(R13),1
       +D2    L       R0,176(,R13)
       +D6    LTR     R0,R0
       +D8    BRC     8,*+10
       +DC    L       R15,8(,R3)
       +E0    BASR    R14,R15
       +E2    LA      R15,0
       +E6    LR      R0,R13
       +E8    L       R13,4(,R13)
       +EC    L       R14,12(,R13)
       +F0    LM      R2,R5,28(R13)
       +F4    BALR    R1,R14
       +F6    BCR     0,R7
       +F8    SLR     R10,R0
       +FA    LDR     FR6,FR0
       +FC    SLR     R10,R0
       +FE    LDR     FR5,FR0
      +100    SLR     R10,R0
      +102    LCR     R3,R0
      +104    LPD     R15,978(R12),964(R15)
      +10A    STH     R14,2291(R3,R12)
      +10E    STH     R13,1265(R4,R15)
      +112    CLC     2548(199,R13),1267(R13)

To understand what is happening in the program more, lets look at the calls to external functions to try and get our bearings. The code below says branch to register 15, and store the return of the function in register 14.

+48 BASR R14,R15

So lets put a breakpoint at this address, and find what is in register 15.

at +48
list 15r
 15R  1FA02860

Ok so we know where its branching to, now set a break point and run 'where' to see what function you ended up in.

at 1FA02860.
  1FA02860. LOCATED AT +0      IN OVERFLOW.printf   UNDER TCB LOCATED AT 8B9E88.

So we know this is printf, set a breakpoint one address forward to test this.

at +4A
  Enter the password :

Lets do this same process with the next function. We find that the below function calls gets.

+5A    BASR    R14,R15

The next C function called strcmp is slightly different. HLASM is somewhat CISC, and has one assembly instruction for comparing strings, "CLST", which compares to strings of characters and then sets the condition codes if they are equal. The below assembly shows the strings being compared, and then branching on condition if they are.

+68    CLST    R2,R1
+6C    LA      R0,0
+70    ST      R2,180(,R13)
+74    ST      R1,184(,R13)
+78    ST      R0,188(,R13)
+7C    BRC     8,*+30

Lets set at breakpoint on the CLST instruction, and see the two strings that are being compared.

at +68
list 1r:2r
  1R  1FA01508   2R  1FAA02E8

list 1FA01508. c m(30)
 1FA01508.  f                     
 1FA01509.  s
 1FA0150A.  e
 1FA0150B.  c
 1FA0150C.  u
 1FA0150D.  r
 1FA0150E.  e
 1FA0150F.  .

list 1FAA02E8. c m(30)
 1FAA02E8.  a               
 1FAA02E9.  b
 1FAA02EA.  c
 1FAA02EB.  d

From this we see that its comparing the input we gave it "abcd" to "fsecure".

Lets call the program again and try fsecure as the password. 

  Enter the password : 
  Correct Password 

Buffer Overflow

Now we H4CK3D TH3 M41NFR4M3 without entering the correct password. Obviously yes, but lets see how it works in HLASM.

Lets set a breakpoint on every instruction in main. Now we basically have a step over function when we press go.

at +0:+112

Notice the below instructions that are always run no matter if the password is wrong or right. MVHI is MoVe fullword from Halfword Immediate, and set the value of pass to 0, Later on if the program loads that memory into register, and runs "LTR" load and test register, which puts the contents of the second register into the first, and checks if the content was 0.

+32    MVHI    176(R13),0
+D2    L       R0,176(,R13)
+D6    LTR     R0,R0

If the password is correct it sets that variable to 1.

+CC    MVHI   176(R13),1

Lets go to D2 to find the contents of register 13.

at +D2 
list 13r
 13R  1FAA1248

Now lets add 176 to 1FAA1248 to get 1FAA12F8 and list the binary from that address. As we can see this is set to one and is when the password is entered correctly.

list 1FAA12F8. B m(4)
 1FAA12F8.  00000000    
 1FAA12F9.  00000000
 1FAA12FA.  00000000
 1FAA12FB.  00000001

 From the previous section we know that our gets data is stored at 1FAA12E8. Each character is 1 byte so to overwrite this data we need 1FAA12FB - 1FAA12E8 = 20 characters in the password.

  Enter the password : 
  Wrong Password 

Note: the absolute address may have changed throughout this walk-though

Note2: zOS compilers do not seem to implement canaries, ASLR, or NX bits, but the way virtual addresses work is zOS provides a lot of protection. For most inter address communication, users are required to be in Modeset 0, which is the most privileged a user can be. Here is further information on the virtual storage map of zOS (http://zseries.marist.edu/pdfs/ztidbitz/29%20zNibbler%20%28zOS%27%20Address%20Space%20%20-%20Virtual%20Storage%20Layout%29.pdf).

Note3: zOS doesn't have a stack. The C and other IBM compilers on zOS by convention, create a DSA (https://www.ibm.com/docs/en/zos/2.2.0?topic=conventions-language-environment-dynamic-storage-area-non-xplink), which is like having a stack per function.

Future work

To get privilege escalation, we are going to need to abuse supervisor state. On zOS this can be done through a number of methods, namely SVC, APF libraries and cross memory services. Future guides will be on how to abuse these.


The POoP (Principles of OPerations):

Introduction to Assembler Programming SHARE Boston 2013

Mainframe [z/OS] reverse engineering and exploit development