Multi-threaded C/C++ applications that use the pthread library can be compiled with Crossbridge to run in Flash Player 11.5 (or newer). This page demonstrates how to use gdb to debug multi-threaded Crossbridge applications.

Getting Started

This page assumes that you already have your computer setup for single threaded debugging as outlined in the debugging with gdb reference section.

To get started we're going to use a very basic multi-threaded C program called sample.c from this excellent pthreads tutorial:

#include <pthread.h>
#include <stdio.h>
#define NUM_THREADS     5

void *PrintHello(void *threadid)
{
   long tid;
   tid = (long)threadid;
   printf("Hello World! It's me, thread #%ld!\n", tid);
   pthread_exit(NULL);
}

int main (int argc, char *argv[])
{
   pthread_t threads[NUM_THREADS];
   int rc;
   long t;
   for(t=0; t<NUM_THREADS; t++){
      printf("In main: creating thread %ld\n", t);
      rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t);
      if (rc){
         printf("ERROR; return code from pthread_create() is %d\n", rc);
         exit(-1);
      }
   }

   /* Last thing that main() should do */
   pthread_exit(NULL);
}

This application simply creates a few threads that print out strings to the screen. First compile the program into a debug SWF with gcc:

.../flascc/sdk/usr/bin/gcc -g -O0 -pthread sample.c -emit-swf -o sample.swf

Now launch gdb with this SWF:

.../flascc/sdk/usr/bin/gdb sample.swf

Let's put a breakpoint at PrintHello and run the application:

(gdb) break PrintHello
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (PrintHello) pending.
(gdb) run
Starting program: sample.swf
0xdddddddd in ?? ()

Notice that the PrintHello function is clearly running as you can see its output in the Flash Player window, but the debugger didn't break on it. This is because, by default, gdb only operates on the main thread. In order to interact with other threads we need to explicitly turn on multi-threading support in gdb.

To turn on multi-threading support we need to enable gdb's "non-stop mode". Non-stop mode means that when one thread in a program stops, other threads will continue running. In non-stop mode, gdb executes some commands asynchronously. When given an asynchronous, or background, command, gdb prompts you for the next command while the original command continues executing in the background. gdb will notify you when the background command finishes. Crossbridge gdb allows all threads to be controlled individually by the debugger when in this mode.

Non-stop mode is slightly different from the default mode of gdb which is called "all-stop mode". In all-stop mode gdb executes all commands synchronously, meaning that each command must complete before you can enter the next command. In this mode, Crossbridge gdb ignores all threads except the main thread. Note that this behavior is specific to Crossbridge gdb; in other gdb implementations, all-stop mode means that when one thread in a program stops, all other threads will stop as well.

Let's quit this gdb session by pressing CTRL+C, then quit, then y:

^C
Program received signal SIGTRAP, Trace/breakpoint trap.
0x00000000 in ?? ()
(gdb) q
A debugging session is active.

	Inferior 1 [Remote target] will be killed.

Quit anyway? (y or n) y

Then launch gdb again and this time we'll try using non-stop mode:

.../flascc/sdk/usr/bin/gdb sample.swf

Debugging in Non-stop Mode

TIP: If you find it annoying to type these three commands at the start of every gdb session you can define a custom gdb command in a .gdbinit file:

define myCustomCommand
    set pagination off
    set target-async on
    set non-stop on
end

This .gdbinit file is run automatically when gdb starts up. The .gdbinit file can either be placed in your home directory or the current directory. The next time you start gdb you just need to enter one command to turn on non-stop mode:

(gdb) myCustomCommand

Non-stop mode must be enabled explicitly at the beginning of your debug session. In order to turn on non-stop mode you need to call three commands:

(gdb) set pagination off 
(gdb) set target-async on
(gdb) set non-stop on

Now we're ready to start debugging this program in non-stop mode. The first thing we're going to do is put a breakpoint at main:

(gdb) break main

Then run the SWF:

(gdb) run

It may take a few seconds to load up and stop at main. You can ignore the [Worker 1]#1 stopped output that shows up and wait until you see that we are indeed stopped at main:

Breakpoint 1, 0xf0000083 in main (argc=0, argv=0x200ff0) at sample.c:18
18          for(t=0; t<NUM_THREADS; t++){
(gdb)

Up until now this should seem familiar to the default mode of gdb, but this is where things start to change. You might expect that you should be able to step through main right now, but if you try doing that you will see an error:

(gdb) step
 Cannot execute this command while the selected thread is running.

gdb reports this error because while one thread has stopped at a breakpoint, the selected thread is still running. You might wonder why there are two threads in the program at this point. After all, the program we're debugging has not called pthread_create yet. The reason is that when threading support is available, Crossbridge calls the main function on a background thread. This allows your C code to perform long running operations without blocking the UI. You can use the info threads command to see the running threads in your program:

(gdb) info threads
  Id   Target Id         Frame
  2    Worker 2          0xf0000083 in main (argc=0, argv=0x200ff0) at sample.c:18
* 1    Worker 1          (running)

You can see that thread 2 is stopped at a breakpoint on line 18 of our program. Thread 1, which is the UI thread, is currently running. In addition to managing UI events, the UI thread also services various low level calls on behalf of other threads. For these reasons, it's generally best to avoid interrupting the UI thread if you want other threads to run unimpeded.

The * signifies which thread is currently selected in gdb. In order to change which thread is currently selected use gdb's thread command and pass in the Id listed above. Here let's change to the thread that is currently broken at main:

(gdb) thread 2
[Switching to thread 2 (Worker 2)]
#0  0xf0000083 in main (argc=0, argv=0x200ff0) at sample.c:18
18          for(t=0; t<NUM_THREADS; t++){

Now that we are on the thread that is suspended at main let's step twice:

(gdb) step
19              printf("In main: creating thread %ld\n", t);
(gdb) step
20              rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t);

At this point we are about to create a thread using the pthread_create function. That thread will be calling the PrintHello function. If you look at the source code of sample.c above you will notice that we're actually going to create a bunch of threads in the for loop. Let's put a breakpoint on the PrintHello function so we have complete control over every thread created to call that function:

(gdb) break PrintHello
Breakpoint 2 at 0xf000004e: file sample.c, line 8.

Now that our breakpoint is setup lets try stepping over the pthread_create line and see what happens:

(gdb) step
21              if (rc){
(gdb) [New Worker 4]

Breakpoint 2, 0xf000004f in PrintHello (threadid=0x0) at sample.c:8
8           tid = (long)threadid;

If you looked closely what you saw might feel a little strange. What happened is you stepped over that line and got another (gdb) prompt just like normal. But meanwhile another thread was in the process of starting up to call PrintHello. As soon as it hit the PrintHello breakpoint that output was dumped onto the screen and now it doesn't look like we have a (gdb) prompt anymore. Don't worry this is all normal. You'll notice that you can still type gdb commands even though you don't see a prompt at the start of the line.

If you do ever want to get the prompt back again you can always hit CTRL+C. Try hitting CTRL+C and then call info threads to see what threads are now started:

Quit
(gdb) info threads
  Id   Target Id         Frame
  3    Worker 4          0xf000004f in PrintHello (threadid=0x0) at sample.c:8
* 2    Worker 2          0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:21
  1    Worker 1          (running)
(gdb)

Using the `step` command

The thread that was created to call PrintHello is now listed with an Id of 3 and suspended at the PrintHello function on line 8. Let's switch over to that thread and step twice:

(gdb) thread 3
[Switching to thread 3 (Worker 4)]
#0  0xf000004f in PrintHello (threadid=0x0) at sample.c:8
8           tid = (long)threadid;
(gdb) step
9           printf("Hello World! It's me, thread #%ld!\n", tid);
(gdb) step
10          pthread_exit(NULL);
(gdb)

At this point we are on the last line of the PrintHello function where the call to pthread_exit() is. If we step one more time it means that this thread will run to completion and we won't see any output:

(gdb) step

Don't worry this is again normal behavior. What happened is the step command tells gdb to run until the current thread is about to begin executing the next line of source code. However in this case since the pthread_exit function causes the calling thread to terminate without returning to the caller, there won't be a next line to arrive at (the thread has finished). So gdb patiently waits for another line that isn't ever going to come. At this point you can hit CTRL+C to interrupt gdb, hit CTRL+C again to get a prompt back, then call info threads to see what is going on in the program:

(gdb) info threads
  Id   Target Id         Frame
  2    Worker 2          0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:21
  1    Worker 1          (running)

The current thread  has terminated.  See `help thread'.
(gdb)

Notice that thread 3 (which we had selected) has been terminated and no longer shows up in the thread list. There is also no * signifying a selected thread and a message telling you that the current thread has terminated. Let's now go back to the thread that is suspended at main:

(gdb) thread 2
[Switching to thread 2 (Worker 2)]
#0  0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:21
21              if (rc){

If we step in this function four more times we will start another thread to call PrintHello:

(gdb) step
18          for(t=0; t<NUM_THREADS; t++){
(gdb) step
19              printf("In main: creating thread %ld\n", t);
(gdb) step
20              rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t);
(gdb) step
21              if (rc){
(gdb) [New Worker 5]

Breakpoint 2, 0xf000004f in PrintHello (threadid=0x1) at sample.c:8
8           tid = (long)threadid;

Now hit CTRL+C to get a prompt back and then select the new thread:

Quit
(gdb) info threads
  Id   Target Id         Frame
  4    Worker 5          0xf000004f in PrintHello (threadid=0x1) at sample.c:8
* 2    Worker 2          0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:21
  1    Worker 1          (running)
(gdb) thread 4
[Switching to thread 4 (Worker 5)]
#0  0xf000004f in PrintHello (threadid=0x1) at sample.c:8
8           tid = (long)threadid;
(gdb)

Using the `continue` command

This time instead of stepping to the end of this thread let's simply hit continue:

(gdb) continue
Continuing.

When gdb continues a thread, it waits until the program stops at a breakpoint before allowing you to enter a new command. In this case though, since thread 2 is suspended and we just allowed thread 4 to terminate, the program will not hit any more breakpoints until we resume thread 2. In order to debug our program further, we must interrupt gdb and regain control over thread 2. Hit CTRL+C once to interrupt gdb, CTRL+C again to get the prompt back, and issue the info threads command:

(gdb) info threads
  Id   Target Id         Frame
  2    Worker 2          0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:
21
  1    Worker 1          (running)

The current thread  has terminated.  See `help thread'.
(gdb)

Go back to the thread suspended at main and step 4 times again to create another thread:

(gdb) thread 2
[Switching to thread 2 (Worker 2)]
#0  0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:21
21              if (rc){
(gdb) step
18          for(t=0; t<NUM_THREADS; t++){
(gdb) step
19              printf("In main: creating thread %ld\n", t);
(gdb) step
20              rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t);
(gdb) step
21              if (rc){
(gdb) [New Worker 6]

Breakpoint 2, 0xf000004f in PrintHello (threadid=0x2) at sample.c:8
8           tid = (long)threadid;

Hit CTRL+C to get the prompt back and then switch to the thread suspended at PrintHello:

Quit
(gdb) info threads
  Id   Target Id         Frame
  5    Worker 6          0xf000004f in PrintHello (threadid=0x2) at sample.c:8
* 2    Worker 2          0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:
21
  1    Worker 1          (running)
(gdb) thread 5
[Switching to thread 5 (Worker 6)]
#0  0xf000004f in PrintHello (threadid=0x2) at sample.c:8
8           tid = (long)threadid;
(gdb)

Using the `continue&` command

Now we're going to call continue again, but this time slightly differently. Instead of issuing the continue command we are going to issue the continue& command. The ampersand at the end of this command means that the command will be issued asynchronously and then a prompt immediately shown ready to receive the next command:

(gdb) continue&
Continuing.
(gdb)

Notice that the next (gdb) prompt immediately shows up after issuing the command. This command allows us to continue long running threads, but still issue other commands or debug other threads at the same time. There is a list near the bottom of this page which shows what commands support an & at the end.

If we now issue the info threads command we'll see that thread has run to completion:

(gdb) info threads
  Id   Target Id         Frame
  2    Worker 2          0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:
21
  1    Worker 1          (running)

The current thread  has terminated.  See `help thread'.

In this case continue& was quite handy as it stopped us from needing to hit CTRL+C after the thread had been terminated.

Let's go back to the thread suspended at main and continue it to completion using continue&:

(gdb) thread 2
[Switching to thread 2 (Worker 2)]
#0  0xf00000a2 in main (argc=0, argv=0x200ff0) at sample.c:21
21              if (rc){
(gdb) continue&
Continuing.
(gdb)
Breakpoint 2, 0xf000004f in PrintHello (threadid=0x3) at sample.c:8
8           tid = (long)threadid;

Breakpoint 2, 0xf000004f in PrintHello (threadid=0x4) at sample.c:8
8           tid = (long)threadid;

If you watched closely you would again notice that the & returned the gdb prompt right away and then later on once the remaining threads were started they broke at PrintHello. Hit CTRL+C again to get a gdb prompt back.

Using the `thread apply` command

When debugging multi-threaded applications there may be many threads and you may need to switch between them often. A handy shortcut to working with multiple threads at once is the thread apply command. This command allows you to call the same command on a series of threads, for example let's get the backtrace of the two threads suspended at PrintHello:

(gdb) info threads
  Id   Target Id         Frame
  7    Worker 8          0xf000004f in PrintHello (threadid=0x4) at sample.c:8
  6    Worker 7          0xf000004f in PrintHello (threadid=0x3) at sample.c:8
  1    Worker 1          (running)

The current thread  has terminated.  See `help thread'.
(gdb) thread apply 6 7 backtrace

Thread 6 (Worker 7):
#0  0xf000004f in PrintHello (threadid=0x3) at sample.c:8
#1  0xf000ce95 in _thread_start () from remote:6.elf
#2  0xf000152e in _thread_run () from remote:2.elf
#3  0x00000000 in ?? ()

Thread 7 (Worker 8):
#0  0xf000004f in PrintHello (threadid=0x4) at sample.c:8
#1  0xf000ce95 in _thread_start () from remote:6.elf
#2  0xf000152e in _thread_run () from remote:2.elf
#3  0x00000000 in ?? ()
(gdb)

You could also apply this command to all active threads by using thread apply all:

(gdb) thread apply all backtrace

Thread 7 (Worker 8):
#0  0xf000004f in PrintHello (threadid=0x4) at sample.c:8
#1  0xf000ce95 in _thread_start () from remote:6.elf
#2  0xf000152e in _thread_run () from remote:2.elf
#3  0x00000000 in ?? ()

Thread 6 (Worker 7):
#0  0xf000004f in PrintHello (threadid=0x3) at sample.c:8
#1  0xf000ce95 in _thread_start () from remote:6.elf
#2  0xf000152e in _thread_run () from remote:2.elf
#3  0x00000000 in ?? ()

Thread 1 (Worker 1):
Target is executing.
(gdb)

You now know how to get started debugging muli-threaded Crossbridge applications with non-stop mode in gdb.

Command Summary

These commands are typically useful when when debugging multi-threaded applications:

info threads - Displays a list of threads
thread X - Sets thread with ID of X to the selected thread
thread apply X Y - Applies the command Y to a list of threads X. For example "thread apply 3 4 step"
thread apply all Y - Applies the command Y to all active threads. For example "thread apply all step"

These gdb commands can be made asynchronous with "&":

step& - Asynchronously step the program onto the next source line (going into subroutine calls)
next& - Asynchronously move the program onto the next source line (going over subroutine calls)
continue& - Asynchronously continue the current selected thread

Debugging Multi-threaded Applications with gdb

Getting Started

Debugging in Non-stop Mode

Using the `step` command

Using the `continue` command

Using the `continue&` command

Using the `thread apply` command

Command Summary

More Resources

Debugging Multi-threaded Applications with gdb

Getting Started

Debugging in Non-stop Mode

Using the step command

Using the continue command

Using the continue& command

Using the thread apply command

Command Summary

More Resources

Using the `step` command

Using the `continue` command

Using the `continue&` command

Using the `thread apply` command