Debugging with gdb

gdb is in this image and it understands Fortran: a breakpoint on a line number, a variable printed by name, a backtrace with the source line of the crash. This lesson takes a program that faults deliberately, builds it with -g, runs a session in batch mode, and shows what the same session looks like when the debug information is missing.

A crash

The program sums four values and then writes the total through a pointer it never associated. That is a segmentation fault on the assignment, and the interesting question for a debugger is not why it crashed but what the values were when it did.

The file crash.f90:

program crash implicit none real, pointer :: p => null() real :: total, x(4) integer :: i x = [1.0, 2.0, 3.0, 4.0] total = 0.0 do i = 1, 4 total = total + x(i) end do p = total print '(a,f6.1)', 'p = ', p end program crash

gfortran -Wall -Wextra -std=f2018 -g crash.f90 -o crash
./crash

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x7b6f3d663430 in ???
#1 0x7b6f3d662345 in ???
#2 0x7b6f3da0688a in ???
  at src/signal/x86_64/restore.s:1
#3 0x5fcc7bd6f284 in crash
  at /tmp/w113/crash.f90:12
#4 0x5fcc7bd6f376 in main
  at /tmp/w113/crash.f90:14

The runtime prints a backtrace of its own, which is often enough to find the line: frame 3 is the assignment at crash.f90:12. What it does not show is a single variable, and that is what gdb is for.

The gdb session

Four commands: a breakpoint one line before the crash, a run that stops there, an inspection of the local variables, and a step into the fault. The session below is the whole output of a batch run, with addresses switched off so that the transcript does not change between machines:

gdb -batch -ex "set print address off" -ex "break crash.f90:12" -ex "run" -ex "info locals" -ex "next" -ex "bt"
    ./crash

Breakpoint 1: file crash.f90, line 12.
warning: Error disabling address space randomization: Operation not permitted

Breakpoint 1, crash () at crash.f90:12
12 p = total
i = 5
p =
total = 10
x = (1, 2, 3, 4)

Program received signal SIGSEGV, Segmentation fault.
crash () at crash.f90:12
12 p = total
#0 crash () at crash.f90:12

The four locals tell the story. total is ten, which is the sum of the four values, so the arithmetic is right; i is five, the value that ended the loop; and p is empty, which is a null pointer printed without its address. The next that follows writes through that null and the signal arrives at the same line the runtime named.

The warning about address space randomisation is this container refusing to turn ASLR off; gdb continues, and every address in the session would differ anyway. That is why the transcript above runs with set print address off: the numbers that matter are the Fortran values, and the addresses are noise in a book.

The commands

A dozen commands cover almost every session. They are the same in C, and gdb does not mind which language compiled the program.

commandwhat it does
break file:linestops at that line; break name stops at a procedure
runstarts the program, arguments after it go to the program
nextruns the current line and stays in this procedure
stepruns the current line and enters a call
print xprints a variable; print x(2) an element
info localsevery local of the current frame
btthe backtrace, one frame per line
frame 3selects a frame to inspect
continueruns to the next breakpoint or the end
listthe source around the current line
deleteremoves a breakpoint
quitleaves gdb

gdb -batch

-batch runs the commands and exits, which is what makes a session reproducible: the same four -ex arguments in a script give the same output every time, and the output can go into a bug report. A graphical front end is comfortable for exploring, and a batch session is the form a team can share.

The alternative to a breakpoint is a core file: ulimit -c unlimited before the run makes the system write one when the program dies, and gdb ./crash core opens it with the same commands. The container's limits decide whether a core is written, and this one does not write it by default, so the session above uses a breakpoint instead of a post-mortem.

Without -g

Build the same source without -g and the debugger has no source to show:

gfortran -Wall -Wextra -std=f2018 crash.f90 -o crash_nog
gdb -batch -ex "set print address off" -ex "break crash.f90:12" -ex "run" -ex "bt" ./crash_nog

No source file named crash.f90.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
warning: Error disabling address space randomization: Operation not permitted

Program received signal SIGSEGV, Segmentation fault.
MAIN__ ()
#0 MAIN__ ()
#1 main ()

The backtrace names MAIN__ and main and no line, and the breakpoint on a line was refused because gdb has no table of line numbers. The binary is smaller and faster and useless to a debugger, which is why a test build carries -g and the release build may not.

Rules for debugging

Build the program that is being debugged with -g and without a high optimisation level: -O0 keeps every variable where the source says it is, and -O2 may report a variable as optimised out. Reproduce the failure in a batch session with the breakpoint one line before it and the variables printed, so that the report contains what a reader needs. And read the frame numbers: the crash is usually in a library below the line the program shows, and the frame above it is the one to fix.

A runtime check is often faster than a debugger, and the next lesson is about the one that catches an array access before it corrupts anything. The two belong together: the check finds the line, and gdb explains why the line was reached with those values.

Exercises

  1. Add print x(1) and print total to the session and run it. Which values do you get before the crash?
  2. Set the breakpoint at line 10 in the loop and print i and total. How many times does the session stop?
  3. Build with -g -O2 and print total at the breakpoint. What does gdb say, and why?
  4. Use break crash instead of the line number on the build without -g. Does gdb find the procedure?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. Which flag puts the source line information into the binary?

    Answer

    -g

    Without it the backtrace has no lines.

  2. What does break crash.f90:12 do?

    Answer

    Stops the program before the statement on line 12

    The run stops at the breakpoint.

  3. What does this session show? info locals at the breakpoint before the crash

    Answer

    i = 5, p empty, total = 10, x = (1, 2, 3, 4)

    It is the transcript of this page.

  4. What does bt print without -g?

    Answer

    The frames with function names but no source lines

    MAIN__ () and main () are all that is left.

  5. What is -batch for?

    Answer

    Running the commands and exiting, so the session is reproducible

    The commands go on the command line with -ex.

  6. What is the difference between next and step?

    Answer

    step enters a call, next runs it and stays

    The usual stepping pair.

  7. Why build the debugged program at -O0?

    Answer

    At -O2 a variable may be optimised out or moved

    The source says where a variable is, and optimisation moves it.

  8. What does the runtime print by itself before gdb is involved?

    Answer

    A backtrace with the source line of the fault

    Frames 3 and 4 name crash.f90:12.

  9. What does set print address off change?

    Answer

    It hides the addresses so the transcript is stable

    Addresses differ on every run.

  10. When is a runtime check better than a debugger?

    Answer

    When the bug corrupts memory silently and you need the line where it happens

    The next lesson is -fcheck=all.

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev