GDB Command Timeout -exec-run with Docker

Sysprogs forums Forums VisualGDB GDB Command Timeout -exec-run with Docker

Viewing 8 posts - 1 through 8 (of 8 total)
  • Author
    Posts
  • #851
    cperthuis
    Participant

    Hi,

    I am unable to debug an executable running in a docker instance via SSH. My target is an Ubuntu 14.04 docker instance running in an Ubuntu 14.04 VirtualBox VM.

    
    =thread-group-added,id="i1"
    -list-features
    ~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"
    ~"Copyright (C) 2014 Free Software Foundation, Inc.n"
    ~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law.  Type "show copying"nand "show warranty" for details.n"
    ~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."
    ~"nFor bug reporting instructions, please see:n"
    ~".n"
    ~"Find the GDB manual and other documentation resources online at:n.n"
    ~"For help, type "help".n"
    ~"Type "apropos word" to search for commands related to "word"...n"
    ~"Reading symbols from /home/user/app0..."
    ~"done.n"
    ^done,features=["frozen-varobjs","pending-breakpoints","thread-info","data-read-memory-bytes","breakpoint-notifications","ada-task-info","language-option","info-gdb-mi-command","undefined-command-error-code","exec-run-start-option","python"]
    -gdb-set stop-on-solib-events 1
    ^done
    -gdb-set disassembly-flavor intel
    ^done
    -gdb-set print demangle off
    ^done
    -gdb-version
    ~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"
    ~"Copyright (C) 2014 Free Software Foundation, Inc.n"
    ~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law. Type "show copying"nand "show warranty" for details.n"
    ~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."
    ~"nFor bug reporting instructions, please see:n"
    ~".n"
    ~"Find the GDB manual and other documentation resources online at:n.n"
    ~"For help, type "help".n"
    ~"Type "apropos word" to search for commands related to "word".n"
    ^done
    -break-insert -f main
    ....

    -exec-run
    =thread-group-started,id="i1",pid="7085"
    =thread-created,id="1",group-id="i1"

    ( Note: I changed the description: gdb doesn’t report thread-exited or any error until I chose Abort operation in the Timeout dialog box )

    The program starts and keeps running fine after VisualGDB displays the Command timeout dialog box.
    If I run the exact same list of commands manually on the machine everything works fine.
    I can debug that program with VisualGDB if I target an Ubuntu VM instead of an Ubuntu docker instance.
    It also runs fine in the docker instance under GDB or standalone.

    How can I diagnostic this issue?
    How does VisualGDB determines that the command has timed out, what is it checking? what is it waiting on?

    Thanks!

    #3236
    cperthuis
    Participant

    Here’s more info to reproduce the issue.
    I am using VirtualBox 4.3.16, I forwarded 2222 onto 2222 and 22 onto 22.
    I installed Ubuntu 14.04 server in the VM.

    In the VM:

    
    sudo -i
    apt-get update
    apt-get install vim openSSH-server docker.io
    ln -sf /usr/bin/docker.io /usr/local/bin/docker
    sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker.io
    

    in a folder create those 2 files:

    Dockerfile

    
    # FROM ubuntu:14.04
    # fix for PAM issue:
    FROM sequenceiq/pam:ubuntu-14.04
    RUN /home/user/install.sh
    EXPOSE 2222
    

    install.sh

    
    #!/bin/sh
    
    dpkg-divert --local --rename --add /sbin/initctl
    ln -nfs /bin/true /sbin/initctl
    apt-get update
    apt-get -y install openSSH-server gdbserver build-essential gdb vim
    apt-get clean
    rm -rf /var/lib/apt/lists/*
    
    # we allow the root user to connect via SSH to allow debugging with VisualGDB
    # setup SSH to allow root user to connect and change port
    sed -i 's/Port 22/Port 2222/g' /etc/ssh/sshd_config
    sed -i -e"s/^PermitRootLogin/#PermitRootLogin/" /etc/ssh/sshd_config
    echo "nPermitRootLogin yesn" >> /etc/ssh/sshd_config

    echo 'root:root' | chpasswd

    useradd -d /home/user -m -s /bin/bash user
    echo 'user:user' | chpasswd
    echo 'user ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers.d/user
    chmod 0440 /etc/sudoers.d/user

    chown -R user /home/user
    chgrp -R user /home/user

    rm /home/user/install.sh

    To build:
    docker build –rm -t gdbimage

    To run:
    docker run -d -v /home/user/src:/home/user/src –net=host -p 2222:2222 gdbimage

    To stop:
    CONTAINER_ID=$(docker ps | grep gdbimage | awk ‘{print $1}’)
    docker stop -t 1 $CONTAINER_ID

    Once the docker instance is running, setup a VisualGDB project with root:password @ 127.0.0.1:2222 as connection info.
    use /home/user/src/projectfolder as destination for the source.

    I used a linux makefile project with the files hosted on the windows host.
    The code will compile fine, and when trying to debug, gdb will fails during the -exec-run step with:

    
    -exec-run
    =thread-group-started,id="i1",pid="7085"
    =thread-created,id="1",group-id="i1"
    

    and if you hit Abort in the timeout dialog box, it then prints:

    
    =thread-exited,id="1",group-id="i1"
    =thread-group-exited,id="i1"
    ^error,msg="During startup program exited with code 1."
    
    #3231
    support
    Keymaster

    Hi,

    Based on your log it looks like the application you’re debugging with gdb fails to initialize. That usually happens when the application cannot locate the libraries it depends on due to missing directories in LD_LIBRARY_PATH. This can happen because VisualGDB starts gdb using a special ‘command’ mode of SSH that can bypass some startup commands (like motd) depending on your sshd configuration. Please try explicitly specifying the full LD_LIBRARY_PATH to GDB command environment in VisualGDB settings.

    If this does not help, we need to understand the virtualization scheme used by docker in order to help you pinpoint the problem. As we have not used docker ourselves, we would need some clarification from you. Does it run a separate virtual kernel, a set of processes under the same kernel, or some layer on top of a single process?

    #3232
    cperthuis
    Participant

    But the application doesn’t fail initializing, GDB is actually starting it correctly even though VisualGDB fails to acknowledge it.
    As long as I don’t press Abort in the window, I am getting all the output I am supposed and the app is fully functional. it’s a server, and I can interact with it fine.

    #3233
    cperthuis
    Participant

    I am not sure that breakpoints work in this config.
    If I use VisualGDB to create a new Hello World project. If I put a breakpoint in the main. it won’t stop on the breakpoint. I saw that there supposed to be a breakpoint inserted at the beginning of main by VisualGDB too, I don’t think it’s hit either.

    #3234
    support
    Keymaster

    OK, than looks like either VisualGDB misses the “*running” notification send by gdb, or gdb itself misses some signal from the app.
    Does it work normally when you run “gdb –interpreter mi” via SSH console and run the exact commands VisualGDB issues? Does the “*running” notification appear in GDB output?

    #3235
    cperthuis
    Participant

    The program works if I use the SSH console, but it doesn’t stop at any breakpoint either, which is the real issue. It probably has nothing to do with VisualGDB.
    As soon as I type run (or -exec-run), the program starts and stays up. it goes way beyond the breakpoint inserted in main. There’s no running message. I can’t really redirect the program output to pts/1 when I run manually, so what I see in GDB is the program output, I am not sure how to check the GDB logs past that point.

    Here’s what I get for the simple HelloWorld sample. The program goes through the breakpoint set in main as if it wasn’t there.

    VisualGDB is licensed to XXXXXX
    gdb --interpreter mi --args "/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1"
    =thread-group-added,id="i1"
    -list-features
    ~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"
    ~"Copyright (C) 2014 Free Software Foundation, Inc.n"
    ~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law.  Type "show copying"nand "show warranty" for details.n"
    ~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."
    ~"nFor bug reporting instructions, please see:n"
    ~".n"
    ~"Find the GDB manual and other documentation resources online at:n.n"
    ~"For help, type "help".n"
    ~"Type "apropos word" to search for commands related to "word"...n"
    ~"Reading symbols from /tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1..."
    ~"done.n"
    ^done,features=["frozen-varobjs","pending-breakpoints","thread-info","data-read-memory-bytes","breakpoint-notifications","ada-task-info","language-option","info-gdb-mi-command","undefined-command-error-code","exec-run-start-option","python"]
    -gdb-set stop-on-solib-events 1
    ^done
    -gdb-set disassembly-flavor intel
    ^done
    -gdb-set print demangle off
    ^done
    -gdb-version
    ~"GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7n"
    ~"Copyright (C) 2014 Free Software Foundation, Inc.n"
    ~"License GPLv3+: GNU GPL version 3 or later nThis is free software: you are free to change and redistribute it.nThere is NO WARRANTY, to the extent permitted by law. Type "show copying"nand "show warranty" for details.n"
    ~"This GDB was configured as "x86_64-linux-gnu".nType "show configuration" for configuration details."
    ~"nFor bug reporting instructions, please see:n"
    ~".n"
    ~"Find the GDB manual and other documentation resources online at:n.n"
    ~"For help, type "help".n"
    ~"Type "apropos word" to search for commands related to "word".n"
    ^done
    -break-insert -f main
    ^done,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x00000000004008dc",func="main(int, char**)",file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp",line="6",thread-groups=["i1"],times="0",original-location="main"}
    -inferior-tty-set /dev/pts/2
    ^done
    info target
    &"info targetn"
    ~"Symbols from "/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1".n"
    ~"Local exec file:n"
    ~"t`/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/Debug/LinuxProject1', file type elf64-x86-64.n"
    ~"tEntry point: 0x4007e0n"
    ~"t0x0000000000400238 - 0x0000000000400254 is .interpn"
    ~"t0x0000000000400254 - 0x0000000000400274 is .note.ABI-tagn"
    ~"t0x0000000000400274 - 0x0000000000400298 is .note.gnu.build-idn"
    ~"t0x0000000000400298 - 0x00000000004002c8 is .gnu.hashn"
    ~"t0x00000000004002c8 - 0x0000000000400418 is .dynsymn"
    ~"t0x0000000000400418 - 0x00000000004005a3 is .dynstrn"
    ~"t0x00000000004005a4 - 0x00000000004005c0 is .gnu.versionn"
    ~"t0x00000000004005c0 - 0x0000000000400610 is .gnu.version_rn"
    ~"t0x0000000000400610 - 0x0000000000400640 is .rela.dynn"
    ~"t0x0000000000400640 - 0x0000000000400718 is .rela.pltn"
    ~"t0x0000000000400718 - 0x0000000000400732 is .initn"
    ~"t0x0000000000400740 - 0x00000000004007e0 is .pltn"
    ~"t0x00000000004007e0 - 0x0000000000400a12 is .textn"
    ~"t0x0000000000400a14 - 0x0000000000400a1d is .finin"
    ~"t0x0000000000400a20 - 0x0000000000400a5c is .eh_frame_hdrn"
    ~"t0x0000000000400a60 - 0x0000000000400bac is .eh_framen"
    ~"t0x0000000000600de8 - 0x0000000000600df8 is .init_arrayn"
    ~"t0x0000000000600df8 - 0x0000000000600e00 is .fini_arrayn"
    ~"t0x0000000000600e00 - 0x0000000000600e08 is .jcrn"
    ~"t0x0000000000600e08 - 0x0000000000600ff8 is .dynamicn"
    ~"t0x0000000000600ff8 - 0x0000000000601000 is .gotn"
    ~"t0x0000000000601000 - 0x0000000000601060 is .got.pltn"
    ~"t0x0000000000601060 - 0x0000000000601068 is .datan"
    ~"t0x0000000000601080 - 0x0000000000601198 is .bssn"
    ^done
    -break-insert -f /tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp:7
    ^done,bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x00000000004008eb",func="main(int, char**)",file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp",line="7",thread-groups=["i1"],times="0",original-location="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp:7"}
    -break-info 2
    ^done,BreakpointTable={nr_rows="1",nr_cols="6",hdr=[{width="7",alignment="-1",col_name="number",colhdr="Num"},{width="14",alignment="-1",col_name="type",colhdr="Type"},{width="4",alignment="-1",col_name="disp",colhdr="Disp"},{width="3",alignment="-1",col_name="enabled",colhdr="Enb"},{width="18",alignment="-1",col_name="addr",colhdr="Address"},{width="40",alignment="2",col_name="what",colhdr="What"}],body=[bkpt={number="2",type="breakpoint",disp="keep",enabled="y",addr="0x00000000004008eb",func="main(int, char**)",file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp",line="7",thread-groups=["i1"],times="0",original-location="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp:7"}]}
    -break-after 2 0
    ^done
    info symbol 0x4008eb
    &"info symbol 0x4008ebn"
    ~"main + 30 in section .textn"
    ^done
    info line *0x4008eb
    &"info line *0x4008ebn"
    ~"Line 7 of "LinuxProject1.cpp" starts at address 0x4008eb and ends at 0x400906 .n"
    ^done
    -file-list-exec-source-files
    ^done,files=[{file="LinuxProject1.cpp",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/LinuxProject1.cpp"},{file="/usr/include/wctype.h",fullname="/usr/include/wctype.h"},{file="/usr/include/x86_64-linux-gnu/c++/4.8/bits/atomic_word.h",fullname="/usr/include/x86_64-linux-gnu/c++/4.8/bits/atomic_word.h"},{file="/usr/include/x86_64-linux-gnu/bits/types.h",fullname="/usr/include/x86_64-linux-gnu/bits/types.h"},{file="/usr/include/locale.h",fullname="/usr/include/locale.h"},{file="/usr/include/c++/4.8/debug/debug.h",fullname="/usr/include/c++/4.8/debug/debug.h"},{file="/usr/include/c++/4.8/ext/numeric_traits.h",fullname="/usr/include/c++/4.8/ext/numeric_traits.h"},{file="/usr/include/c++/4.8/ext/new_allocator.h",fullname="/usr/include/c++/4.8/ext/new_allocator.h"},{file="/usr/include/time.h",fullname="/usr/include/time.h"},{file="/usr/include/c++/4.8/iosfwd",fullname="/usr/include/c++/4.8/iosfwd"},{file="/usr/include/c++/4.8/cwctype",fullname="/usr/include/c++/4.8/cwctype"},{file="/usr/include/c++/4.8/bits/ios_base.h",fullname="/usr/include/c++/4.8/bits/ios_base.h"},{file="/usr/include/c++/4.8/clocale",fullname="/usr/include/c++/4.8/clocale"},{file="/usr/include/x86_64-linux-gnu/c++/4.8/bits/c++config.h",fullname="/usr/include/x86_64-linux-gnu/c++/4.8/bits/c++config.h"},{file="/usr/include/c++/4.8/bits/char_traits.h",fullname="/usr/include/c++/4.8/bits/char_traits.h"},{file="/usr/include/c++/4.8/cwchar",fullname="/usr/include/c++/4.8/cwchar"},{file="/usr/include/wchar.h",fullname="/usr/include/wchar.h"},{file="/usr/lib/gcc/x86_64-linux-gnu/4.8/include/stddef.h",fullname="/usr/lib/gcc/x86_64-linux-gnu/4.8/include/stddef.h"},{file="/usr/include/stdio.h",fullname="/usr/include/stdio.h"},{file="",fullname="/tmp/VisualGDB/z/test/LinuxProject1/LinuxProject1/"},{file="/usr/include/c++/4.8/iostream",fullname="/usr/include/c++/4.8/iostream"},{file="/usr/include/c++/4.8/iostream",fullname="/usr/include/c++/4.8/iostream"}]
    -exec-run
    =thread-group-started,id="i1",pid="1428"
    =thread-created,id="1",group-id="i1"
    =thread-exited,id="1",group-id="i1"
    =thread-group-exited,id="i1"
    ^error,msg="During startup program exited normally."
    -stack-list-frames --thread 1
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames
    ^error,msg="No registers."
    -stack-list-frames --thread 1 0 0
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames 0 0
    ^error,msg="No registers."
    -data-evaluate-expression --thread 1 "$pc"
    ^error,msg="Invalid thread id: 1"
    -data-evaluate-expression "$pc"
    ^error,msg="No registers."
    -thread-info
    ^done,threads=[]
    -stack-list-frames --thread 1
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames
    ^error,msg="No registers."
    -stack-list-frames --thread 1 0 0
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames 0 0
    ^error,msg="No registers."
    -data-evaluate-expression --thread 1 "$pc"
    ^error,msg="Invalid thread id: 1"
    -data-evaluate-expression "$pc"
    ^error,msg="No registers."
    -stack-list-frames --thread 1
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames
    ^error,msg="No registers."
    -stack-list-frames --thread 1 0 0
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames 0 0
    ^error,msg="No registers."
    -data-evaluate-expression --thread 1 "$pc"
    ^error,msg="Invalid thread id: 1"
    -data-evaluate-expression "$pc"
    ^error,msg="No registers."
    -stack-list-frames --thread 1
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames
    ^error,msg="No registers."
    -stack-list-frames --thread 1 0 0
    ^error,msg="Invalid thread id: 1"
    -stack-list-frames 0 0
    ^error,msg="No registers."
    -data-evaluate-expression --thread 1 "$pc"
    ^error,msg="Invalid thread id: 1"
    -data-evaluate-expression "$pc"
    ^error,msg="No registers."
    -thread-info
    ^done,threads=[]
    #3230
    support
    Keymaster

    OK, that rules out SSH communication problems. So it looks like GDB cannot receive signals from your program (are they handled by some Docker component instead?). Does it only happen in the “–interpreter MI” mode or in the normal command-line GDB mode as well? If you can make a repro case consisting of just a “hello, world” program and a GDB failing to stop at a breakpoint, the Docker support might be able to provide more help as they know Docker internals.

Viewing 8 posts - 1 through 8 (of 8 total)
  • You must be logged in to reply to this topic.