Topic: GDB Command Timeout -exec-run with Docker

This topic has 7 replies, 2 voices, and was last updated 10 years, 1 month ago by cperthuis.

Viewing 8 posts - 1 through 8 (of 8 total)

Author

Posts

October 8, 2014 at 06:34 #851

Participant

Hi,

I am unable to debug an executable running in a docker instance via SSH. My target is an Ubuntu 14.04 docker instance running in an Ubuntu 14.04 VirtualBox VM.


=thread-group-added,id="i1"
-list-features
~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"
~"Copyright (C) 2014 Free Software Foundation, Inc.n"
~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law.  Type "show copying"nand "show warranty" for details.n"

~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."

~"nFor bug reporting instructions, please see:n"

~".n"

~"Find the GDB manual and other documentation resources online at:n.n"

~"For help, type "help".n"

~"Type "apropos word" to search for commands related to "word"...n"

~"Reading symbols from /home/user/app0..."

~"done.n"

^done,features=["frozen-varobjs","pending-breakpoints","thread-info","data-read-memory-bytes","breakpoint-notifications","ada-task-info","language-option","info-gdb-mi-command","undefined-command-error-code","exec-run-start-option","python"]

-gdb-set stop-on-solib-events 1

^done

-gdb-set disassembly-flavor intel

^done

-gdb-set print demangle off

^done

-gdb-version

~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"

~"Copyright (C) 2014 Free Software Foundation, Inc.n"

~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law.  Type "show copying"nand "show warranty" for details.n"

~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."

~"nFor bug reporting instructions, please see:n"

~".n"

~"Find the GDB manual and other documentation resources online at:n.n"

~"For help, type "help".n"

~"Type "apropos word" to search for commands related to "word".n"

^done

-break-insert -f main

....



-exec-run

=thread-group-started,id="i1",pid="7085"

=thread-created,id="1",group-id="i1"

( Note: I changed the description: gdb doesn’t report thread-exited or any error until I chose Abort operation in the Timeout dialog box )

The program starts and keeps running fine after VisualGDB displays the Command timeout dialog box.
If I run the exact same list of commands manually on the machine everything works fine.
I can debug that program with VisualGDB if I target an Ubuntu VM instead of an Ubuntu docker instance.
It also runs fine in the docker instance under GDB or standalone.

How can I diagnostic this issue?
How does VisualGDB determines that the command has timed out, what is it checking? what is it waiting on?

Thanks!

October 8, 2014 at 18:05 #3236

cperthuis

Participant

Here’s more info to reproduce the issue.
I am using VirtualBox 4.3.16, I forwarded 2222 onto 2222 and 22 onto 22.
I installed Ubuntu 14.04 server in the VM.

In the VM:


sudo -i
apt-get update
apt-get install vim openSSH-server docker.io
ln -sf /usr/bin/docker.io /usr/local/bin/docker
sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker.io

in a folder create those 2 files:

Dockerfile


# FROM ubuntu:14.04
# fix for PAM issue:
FROM sequenceiq/pam:ubuntu-14.04
RUN /home/user/install.sh
EXPOSE 2222

install.sh


#!/bin/sh

dpkg-divert --local --rename --add /sbin/initctl
ln -nfs /bin/true /sbin/initctl
apt-get update
apt-get -y install openSSH-server gdbserver build-essential gdb vim
apt-get clean
rm -rf /var/lib/apt/lists/*

# we allow the root user to connect via SSH to allow debugging with VisualGDB
# setup SSH to allow root user to connect and change port
sed -i 's/Port 22/Port 2222/g' /etc/ssh/sshd_config
sed -i -e"s/^PermitRootLogin/#PermitRootLogin/" /etc/ssh/sshd_config
echo "nPermitRootLogin yesn" >> /etc/ssh/sshd_config



echo 'root:root' | chpasswd



useradd -d /home/user -m -s /bin/bash user

echo 'user:user' | chpasswd

echo 'user ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers.d/user

chmod 0440 /etc/sudoers.d/user



chown -R user /home/user

chgrp -R user /home/user



rm /home/user/install.sh

To build:
docker build –rm -t gdbimage

To run:
docker run -d -v /home/user/src:/home/user/src –net=host -p 2222:2222 gdbimage

To stop:
CONTAINER_ID=$(docker ps | grep gdbimage | awk ‘{print $1}’)
docker stop -t 1 $CONTAINER_ID

Once the docker instance is running, setup a VisualGDB project with root:password @ 127.0.0.1:2222 as connection info.
use /home/user/src/projectfolder as destination for the source.

I used a linux makefile project with the files hosted on the windows host.
The code will compile fine, and when trying to debug, gdb will fails during the -exec-run step with:


-exec-run
=thread-group-started,id="i1",pid="7085"
=thread-created,id="1",group-id="i1"

and if you hit Abort in the timeout dialog box, it then prints:


=thread-exited,id="1",group-id="i1"
=thread-group-exited,id="i1"
^error,msg="During startup program exited with code 1."

October 8, 2014 at 18:32 #3231

support

Keymaster

Hi,

Based on your log it looks like the application you’re debugging with gdb fails to initialize. That usually happens when the application cannot locate the libraries it depends on due to missing directories in LD_LIBRARY_PATH. This can happen because VisualGDB starts gdb using a special ‘command’ mode of SSH that can bypass some startup commands (like motd) depending on your sshd configuration. Please try explicitly specifying the full LD_LIBRARY_PATH to GDB command environment in VisualGDB settings.

If this does not help, we need to understand the virtualization scheme used by docker in order to help you pinpoint the problem. As we have not used docker ourselves, we would need some clarification from you. Does it run a separate virtual kernel, a set of processes under the same kernel, or some layer on top of a single process?

October 8, 2014 at 18:52 #3232

cperthuis

Participant

But the application doesn’t fail initializing, GDB is actually starting it correctly even though VisualGDB fails to acknowledge it.
As long as I don’t press Abort in the window, I am getting all the output I am supposed and the app is fully functional. it’s a server, and I can interact with it fine.

October 8, 2014 at 18:58 #3233

cperthuis

Participant

I am not sure that breakpoints work in this config.
If I use VisualGDB to create a new Hello World project. If I put a breakpoint in the main. it won’t stop on the breakpoint. I saw that there supposed to be a breakpoint inserted at the beginning of main by VisualGDB too, I don’t think it’s hit either.

October 8, 2014 at 19:05 #3234

support

Keymaster

OK, than looks like either VisualGDB misses the “*running” notification send by gdb, or gdb itself misses some signal from the app.
Does it work normally when you run “gdb –interpreter mi” via SSH console and run the exact commands VisualGDB issues? Does the “*running” notification appear in GDB output?

October 8, 2014 at 23:11 #3235

cperthuis

Participant

The program works if I use the SSH console, but it doesn’t stop at any breakpoint either, which is the real issue. It probably has nothing to do with VisualGDB.
As soon as I type run (or -exec-run), the program starts and stays up. it goes way beyond the breakpoint inserted in main. There’s no running message. I can’t really redirect the program output to pts/1 when I run manually, so what I see in GDB is the program output, I am not sure how to check the GDB logs past that point.

Here’s what I get for the simple HelloWorld sample. The program goes through the breakpoint set in main as if it wasn’t there.

VisualGDB is licensed to XXXXXX
gdb --interpreter mi --args "/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1"
=thread-group-added,id="i1"
-list-features
~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"
~"Copyright (C) 2014 Free Software Foundation, Inc.n"
~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law.  Type "show copying"nand "show warranty" for details.n"

~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."

~"nFor bug reporting instructions, please see:n"

~".n"

~"Find the GDB manual and other documentation resources online at:n.n"

~"For help, type "help".n"

~"Type "apropos word" to search for commands related to "word"...n"

~"Reading symbols from /tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1..."

~"done.n"

^done,features=["frozen-varobjs","pending-breakpoints","thread-info","data-read-memory-bytes","breakpoint-notifications","ada-task-info","language-option","info-gdb-mi-command","undefined-command-error-code","exec-run-start-option","python"]

-gdb-set stop-on-solib-events 1

^done

-gdb-set disassembly-flavor intel

^done

-gdb-set print demangle off

^done

-gdb-version

~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"

~"Copyright (C) 2014 Free Software Foundation, Inc.n"

~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law.  Type "show copying"nand "show warranty" for details.n"

~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."

~"nFor bug reporting instructions, please see:n"

~".n"

~"Find the GDB manual and other documentation resources online at:n.n"

~"For help, type "help".n"

~"Type "apropos word" to search for commands related to "word".n"

^done

-break-insert -f main

^done,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x00000000004008dc",func="main(int, char**)",file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp",line="6",thread-groups=["i1"],times="0",original-location="main"}

-inferior-tty-set /dev/pts/2

^done

info target

&"info targetn"

~"Symbols from "/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1".n"

~"Local exec file:n"

~"t`/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1', file type elf64-x86-64.n"

~"tEntry point: 0x4007e0n"

~"t0x0000000000400238 - 0x0000000000400254 is .interpn"

~"t0x0000000000400254 - 0x0000000000400274 is .note.ABI-tagn"

~"t0x0000000000400274 - 0x0000000000400298 is .note.gnu.build-idn"

~"t0x0000000000400298 - 0x00000000004002c8 is .gnu.hashn"

~"t0x00000000004002c8 - 0x0000000000400418 is .dynsymn"

~"t0x0000000000400418 - 0x00000000004005a3 is .dynstrn"

~"t0x00000000004005a4 - 0x00000000004005c0 is .gnu.versionn"

~"t0x00000000004005c0 - 0x0000000000400610 is .gnu.version_rn"

~"t0x0000000000400610 - 0x0000000000400640 is .rela.dynn"

~"t0x0000000000400640 - 0x0000000000400718 is .rela.pltn"

~"t0x0000000000400718 - 0x0000000000400732 is .initn"

~"t0x0000000000400740 - 0x00000000004007e0 is .pltn"

~"t0x00000000004007e0 - 0x0000000000400a12 is .textn"

~"t0x0000000000400a14 - 0x0000000000400a1d is .finin"

~"t0x0000000000400a20 - 0x0000000000400a5c is .eh_frame_hdrn"

~"t0x0000000000400a60 - 0x0000000000400bac is .eh_framen"

~"t0x0000000000600de8 - 0x0000000000600df8 is .init_arrayn"

~"t0x0000000000600df8 - 0x0000000000600e00 is .fini_arrayn"

~"t0x0000000000600e00 - 0x0000000000600e08 is .jcrn"

~"t0x0000000000600e08 - 0x0000000000600ff8 is .dynamicn"

~"t0x0000000000600ff8 - 0x0000000000601000 is .gotn"

~"t0x0000000000601000 - 0x0000000000601060 is .got.pltn"

~"t0x0000000000601060 - 0x0000000000601068 is .datan"

~"t0x0000000000601080 - 0x0000000000601198 is .bssn"

^done

-break-insert -f /tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp:7

^done,bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x00000000004008eb",func="main(int, char**)",file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp",line="7",thread-groups=["i1"],times="0",original-location="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp:7"}

-break-info 2

^done,BreakpointTable={nr_rows="1",nr_cols="6",hdr=[{width="7",alignment="-1",col_name="number",colhdr="Num"},{width="14",alignment="-1",col_name="type",colhdr="Type"},{width="4",alignment="-1",col_name="disp",colhdr="Disp"},{width="3",alignment="-1",col_name="enabled",colhdr="Enb"},{width="18",alignment="-1",col_name="addr",colhdr="Address"},{width="40",alignment="2",col_name="what",colhdr="What"}],body=[bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x00000000004008eb",func="main(int, char**)",file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp",line="7",thread-groups=["i1"],times="0",original-location="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp:7"}]}

-break-after 2 0

^done

info symbol 0x4008eb

&"info symbol 0x4008ebn"

~"main + 30 in section .textn"

^done

info line *0x4008eb

&"info line *0x4008ebn"

~"Line 7 of "LinuxProject1.cpp" starts at address 0x4008eb  and ends at 0x400906 .n"

^done

-file-list-exec-source-files

^done,files=[{file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp"},{file="/usr/include/wctype.h",fullname="/usr/include/wctype.h"},{file="/usr/include/x86_64-linux-gnu/c++/4.8/bits/atomic_word.h",fullname="/usr/include/x86_64-linux-gnu/c++/4.8/bits/atomic_word.h"},{file="/usr/include/x86_64-linux-gnu/bits/types.h",fullname="/usr/include/x86_64-linux-gnu/bits/types.h"},{file="/usr/include/locale.h",fullname="/usr/include/locale.h"},{file="/usr/include/c++/4.8/debug/debug.h",fullname="/usr/include/c++/4.8/debug/debug.h"},{file="/usr/include/c++/4.8/ext/numeric_traits.h",fullname="/usr/include/c++/4.8/ext/numeric_traits.h"},{file="/usr/include/c++/4.8/ext/new_allocator.h",fullname="/usr/include/c++/4.8/ext/new_allocator.h"},{file="/usr/include/time.h",fullname="/usr/include/time.h"},{file="/usr/include/c++/4.8/iosfwd",fullname="/usr/include/c++/4.8/iosfwd"},{file="/usr/include/c++/4.8/cwctype",fullname="/usr/include/c++/4.8/cwctype"},{file="/usr/include/c++/4.8/bits/ios_base.h",fullname="/usr/include/c++/4.8/bits/ios_base.h"},{file="/usr/include/c++/4.8/clocale",fullname="/usr/include/c++/4.8/clocale"},{file="/usr/include/x86_64-linux-gnu/c++/4.8/bits/c++config.h",fullname="/usr/include/x86_64-linux-gnu/c++/4.8/bits/c++config.h"},{file="/usr/include/c++/4.8/bits/char_traits.h",fullname="/usr/include/c++/4.8/bits/char_traits.h"},{file="/usr/include/c++/4.8/cwchar",fullname="/usr/include/c++/4.8/cwchar"},{file="/usr/include/wchar.h",fullname="/usr/include/wchar.h"},{file="/usr/lib/gcc/x86_64-linux-gnu/4.8/include/stddef.h",fullname="/usr/lib/gcc/x86_64-linux-gnu/4.8/include/stddef.h"},{file="/usr/include/stdio.h",fullname="/usr/include/stdio.h"},{file="",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/"},{file="/usr/include/c++/4.8/iostream",fullname="/usr/include/c++/4.8/iostream"},{file="/usr/include/c++/4.8/iostream",fullname="/usr/include/c++/4.8/iostream"}]

-exec-run

=thread-group-started,id="i1",pid="1428"

=thread-created,id="1",group-id="i1"

=thread-exited,id="1",group-id="i1"

=thread-group-exited,id="i1"

^error,msg="During startup program exited normally."

-stack-list-frames --thread 1

^error,msg="Invalid thread id: 1"

-stack-list-frames

^error,msg="No registers."

-stack-list-frames --thread 1 0 0

^error,msg="Invalid thread id: 1"

-stack-list-frames 0 0

^error,msg="No registers."

-data-evaluate-expression --thread 1 "$pc"

^error,msg="Invalid thread id: 1"

-data-evaluate-expression "$pc"

^error,msg="No registers."

-thread-info

^done,threads=[]

-stack-list-frames --thread 1

^error,msg="Invalid thread id: 1"

-stack-list-frames

^error,msg="No registers."

-stack-list-frames --thread 1 0 0

^error,msg="Invalid thread id: 1"

-stack-list-frames 0 0

^error,msg="No registers."

-data-evaluate-expression --thread 1 "$pc"

^error,msg="Invalid thread id: 1"

-data-evaluate-expression "$pc"

^error,msg="No registers."

-stack-list-frames --thread 1

^error,msg="Invalid thread id: 1"

-stack-list-frames

^error,msg="No registers."

-stack-list-frames --thread 1 0 0

^error,msg="Invalid thread id: 1"

-stack-list-frames 0 0

^error,msg="No registers."

-data-evaluate-expression --thread 1 "$pc"

^error,msg="Invalid thread id: 1"

-data-evaluate-expression "$pc"

^error,msg="No registers."

-stack-list-frames --thread 1

^error,msg="Invalid thread id: 1"

-stack-list-frames

^error,msg="No registers."

-stack-list-frames --thread 1 0 0

^error,msg="Invalid thread id: 1"

-stack-list-frames 0 0

^error,msg="No registers."

-data-evaluate-expression --thread 1 "$pc"

^error,msg="Invalid thread id: 1"

-data-evaluate-expression "$pc"

^error,msg="No registers."

-thread-info

^done,threads=[]

October 12, 2014 at 18:52 #3230

support

Keymaster

OK, that rules out SSH communication problems. So it looks like GDB cannot receive signals from your program (are they handled by some Docker component instead?). Does it only happen in the “–interpreter MI” mode or in the normal command-line GDB mode as well? If you can make a repro case consisting of just a “hello, world” program and a GDB failing to stop at a breakpoint, the Docker support might be able to provide more help as they know Docker internals.

Author

Posts

Viewing 8 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.