- This topic has 12 replies, 2 voices, and was last updated 2 months, 1 week ago by
.
-
Posts
-
VisualGDB version 6.0R3 5191, VS2022.
I have a CMake integration project that builds both managed and native code, and targets an ARM64 Linux embedded target. Through CMake conditional compile, VisualGDB sees and handles building only the native code. To debug .so code, I want to launch mono against my managed executable, which will eventually load the .so. I have a ‘debug only’ VisualGDB target that runs /usr/bin/mono with appropriate arguments, but when I start this up it errors out for some reason that I don’t understand. I don’t need to debug managed code in this scenario, though if that is possible, that would certainly be useful. For now, just being able to debug native code would suffice. I’m attaching the detailed GDB log.
It seems to go bad about here:
[ 5157 ms] &”Cannot insert breakpoint -1.\n”
[ 5157 ms] &”Cannot access memory at address 0x2cf80\n”
[ 5157 ms] &”\n”
[ 5157 ms] ^error,msg=”Command aborted.”I see this output whether or not I have any breakpoints set in my .so. I have disabled ‘Stepping into new instance (F10) will stop in: ___’ in advanced gdb settings. I have disabled stopping on signals specified by the mono documentation (SIGXCPU SIG33 SIG35 SIG36 SIG37 SIG38 SIGPWR)
Thanks for any help on how to get this working.
Hi,
This is somewhat of a known issue, however it could be non-trivial to untangle. The most important lines here are these:
[ 5113 ms] ~"Reading /lib64/ld-linux-x86-64.so.2 from remote target...\n" [ 5114 ms] &"warning: File transfers from remote targets can be slow. Use \"set sysroot\" to access files locally instead.\n" [ 5114 ms] &"warning: Unable to find dynamic linker breakpoint function.\nGDB will be unable to debug shared library initializers\nand track explicitly loaded dynamic code."
This means that GDB cannot figure out how to detect when shared libraries are loaded (i.e. the .so file you are trying to debug), so it won’t be able to set breakpoints there.
To fix this, you need to find the unstripped file corresponding to /lib64/ld-linux-x86-64.so.2, and make sure gdb can load symbols from it. If you are using yocto, the build process might already produce the symbols and put them into a separate sysroot directory. If this is the case, all you need to do is issue a set sysroot command to gdb, pointing it to the correct sysroot with the symbols. If not, you would need to locate the symbol file manually, and either recreate a sysroot directory on the gdb machine, or just put it to some location and add it to solib-search-path.
As for debugging the managed code, VisualGDB doesn’t have any special support for it, however you can still use the gdb debugging techniques from Mono documentation. We can also give you a quote for properly supporting Mono debugging as a custom feature, however it would be rather non-trivial and may require additional maintenance to support future Mono versions.
Thank you. I will look into symbols for the loader library.
In the mean time, I notice that I gdb works if I just run it manually. I can attach to the remote, set a breakpoint in my .so, run to the breakpoint, and then continue. Seems like that should not work according to what you have said above.
On the target:
ffox-fluyt-npi03:/tmp/test$ gdbserver :2201 /usr/bin/mono /tmp/test/ffox2.exe --run-native Process /usr/bin/mono created; pid = 32464 Listening on port 2201
On the build machine:
dwstark@earthtux:~$ $GDB GNU gdb (GDB) 11.2 Copyright (C) 2022 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-keysight-linux --target=aarch64-keysight-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word". (gdb) target remote ffox-fluyt-npi03:2201 Remote debugging using ffox-fluyt-npi03:2201 Reading /usr/bin/mono-sgen from remote target... warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead. Reading /usr/bin/mono-sgen from remote target... Reading symbols from target:/usr/bin/mono-sgen... Reading /usr/bin/.debug/mono-sgen from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//usr/bin/mono-sgen from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/usr/bin//mono-sgen from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/usr/bin//mono-sgen from remote target... (No debugging symbols found in target:/usr/bin/mono-sgen) Reading /lib/ld-linux-aarch64.so.1 from remote target... Reading /lib/ld-linux-aarch64.so.1 from remote target... Reading symbols from target:/lib/ld-linux-aarch64.so.1... Reading /lib/.debug/ld-linux-aarch64.so.1 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//lib/ld-linux-aarch64.so.1 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//ld-linux-aarch64.so.1 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//ld-linux-aarch64.so.1 from remote target... (No debugging symbols found in target:/lib/ld-linux-aarch64.so.1) 0x0000fffff7fd9cc0 in ?? () from target:/lib/ld-linux-aarch64.so.1 (gdb) b MSDriver_CreateDriver Function "MSDriver_CreateDriver" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (MSDriver_CreateDriver) pending. (gdb) c Continuing. Reading /lib/libz.so.1 from remote target... Reading /lib/libm.so.6 from remote target... Reading /lib/libc.so.6 from remote target... Reading /lib/libz.so.1.2.11 from remote target... Reading /lib/.debug/libz.so.1.2.11 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//lib/libz.so.1.2.11 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libz.so.1.2.11 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libz.so.1.2.11 from remote target... Reading /lib/.debug/libm.so.6 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//lib/libm.so.6 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libm.so.6 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libm.so.6 from remote target... Reading /lib/.debug/libc.so.6 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//lib/libc.so.6 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libc.so.6 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libc.so.6 from remote target... Reading /usr/lib/../lib/libmono-native.so from remote target... Reading /usr/lib/../lib/libmono-native.so.0.0.0 from remote target... Reading /usr/lib/../lib/.debug/libmono-native.so.0.0.0 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//usr/lib/../lib/libmono-native.so.0.0.0 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/usr/lib/../lib//libmono-native.so.0.0.0 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/usr/lib/../lib//libmono-native.so.0.0.0 from remote target... Reading /lib/libpthread.so.0 from remote target... Reading /lib/.debug/libpthread.so.0 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//lib/libpthread.so.0 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libpthread.so.0 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libpthread.so.0 from remote target... Reading /tmp/test/libmsdriver.so from remote target... Reading /usr/lib/libstdc++.so.6 from remote target... Reading /lib/libgcc_s.so.1 from remote target... Reading /usr/lib/libstdc++.so.6.0.29 from remote target... Reading /usr/lib/.debug/libstdc++.so.6.0.29 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//usr/lib/libstdc++.so.6.0.29 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/usr/lib//libstdc++.so.6.0.29 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/usr/lib//libstdc++.so.6.0.29 from remote target... Reading /lib/.debug/libgcc_s.so.1 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug//lib/libgcc_s.so.1 from remote target... Reading /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libgcc_s.so.1 from remote target... Reading target:/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-keysight-linux/usr/lib/aarch64-keysight-linux/debug/lib//libgcc_s.so.1 from remote target... [New Thread 32464.32470] [New Thread 32464.32471] Thread 1 "mono" hit Breakpoint 1, MSDriver_CreateDriver () at /home/dwstark/VisualGDB/ffox-2/native/msdriver/msdriver.cpp:18 18 msdriver::MSDriver* p = new msdriver::MSDriver(); (gdb) c Continuing. warning: Temporarily disabling breakpoints for unloaded shared library "target:/tmp/test/libmsdriver.so" [Inferior 1 (process 32464) exited normally] (gdb)
Note that the same SDK is configured in the CMake tab of VisualGDB project properties as is sourced in the above command line example that works.
dwstark@earthtux:~$ export | grep SYSROOT declare -x OECORE_NATIVE_SYSROOT="/opt/ffox/qjet/ffox-instrument/4.0-2.0.0/sysroots/x86_64-keysight-linux" declare -x OECORE_TARGET_SYSROOT="/opt/ffox/qjet/ffox-instrument/4.0-2.0.0/sysroots/cortexa53-keysight-linux" declare -x PKG_CONFIG_SYSROOT_DIR="/opt/ffox/qjet/ffox-instrument/4.0-2.0.0/sysroots/cortexa53-keysight-linux" declare -x SDKTARGETSYSROOT="/opt/ffox/qjet/ffox-instrument/4.0-2.0.0/sysroots/cortexa53-keysight-linux" dwstark@earthtux:~$ ls -la /opt/ffox/qjet/ffox-instrument/current lrwxrwxrwx 1 root root 9 Aug 15 15:19 /opt/ffox/qjet/ffox-instrument/current -> 4.0-2.0.0/ dwstark@earthtux:~$
Hi,
Then the sysroot directory should be already present and it should be fairly easy to get it working:
- Try running the exact gdb command line shown by VisualGDB in the log file.
- Try manually issuing the gdb commands issued by VisualGDB. You can dump them into a text file and load it via the “source” command. You can filter out “-gdb-show” and other commands that are unlikely to change anything.
- Confirm that the problem persists and that it works when running gdb without any extra commands.
- Try removing half of the commands copied from VisualGDB and checking whether the problem persists.
Once you narrow down to a specific command that causes the problem, we can help you find the setting to adjust it.
You can also quickly try checking the values of sysroot and solib-search-path in the working session, and forcing them via VisualGDB Project Properties -> Additional GDB Commands -> after selecting a target. But if it doesn’t help, doing the side-by-side comparison as shown above should work.
OK, in order to make the command line version fail, all I need to do is pass the gdb argument –args “/usr/bin/mono”. The –interpreter argument doesn’t matter.
dwstark@earthtux:~$ $GDB --args "/usr/bin/mono" GNU gdb (GDB) 11.2 Copyright (C) 2022 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-keysight-linux --target=aarch64-keysight-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/mono... (No debugging symbols found in /usr/bin/mono) (gdb) target remote ffox-fluyt-npi03:2201 Remote debugging using ffox-fluyt-npi03:2201 warning: Build ID mismatch between current exec-file /usr/bin/mono and automatically determined exec-file /usr/bin/mono-sgen exec-file-mismatch handling is currently "ask" Load new symbol table from "/usr/bin/mono-sgen"? (y or n) y Reading symbols from /usr/bin/mono-sgen... (No debugging symbols found in /usr/bin/mono-sgen) Reading /lib64/ld-linux-x86-64.so.2 from remote target... warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead. warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. 0x0000fffff7fd9cc0 in ?? () (gdb) b MSDriver_CreateDriver Function "MSDriver_CreateDriver" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (MSDriver_CreateDriver) pending. (gdb) c Continuing. Warning: Cannot insert breakpoint -1. Cannot access memory at address 0x2cf80 Command aborted. (gdb)
Hi,
Thanks, this is very strange, but easy to work around. You can clear the VisualGDB Project Properties -> Debug Settings -> Common Settings -> Debugged Executable field, or outright switch the debug mode to “fully custom” and specify the command line explicitly (the –interpreter mi part should still be present).
Clearing the ‘debugged’ executable field causes a dialog to come up that says:
“The project settings are not configured to debug the startup or selected target. Do you want to fix the debug settings?”
Answering yes puts you back where you started, and No works, but you get prompted every time you try to debug. Can we get that to stop happening?
I can also put the full path to mono in the SDK sysroot (the mono that actually on the target), and this also seems to work, so it clearly doesn’t like pointing to the /usr/bin/mono on the build machine.
I can work around getting that dialog coming up each time with the ‘full custom’ setup.
Unfortunately, to get that to work I have to fill out the ‘Target selection command:”, and the macro for my host alias “TargetMachine” does not seem to be accessible, so I have to hardcode my debug target name in there. That won’t work well for a group of developers, I don’t think.
I have noticed that host aliases also don’t seem to be available when writing temporary scripts in custom shortcuts or post-build rules, etc. That would be useful.
Hi,
No problem, we have made the $(TargetPath) check optional in this build: VisualGDB-6.0.103.5206.msi. It also fixes the BuildHost/DeployHost variable behavior when using aliases.
That said, using the mono path inside sysroot should be the way to go. If the gdb machine has a different architecture than the target machine, the /usr/bin/mono executable could give GDB all kinds of incorrect hints, resulting in weird glitches across the board.
Thanks. I can see that DeployHost is properly populated. However, if I change the value of DeployHost to a new host (using the host aliases GUI), and then re-run my custom shortcut script, it continues to expand to the old value until I unload and reload the project. Is this the expected behavior?
I also see that the $(DeployDir) when used in my custom shortcuts seems fixed at /tmp and does not change with Debug Settings -> Deployment -> Base Deployment Directory. I don’t see where /tmp is set anywhere else, so I’m assuming $(DeployDir) is tied to that field, but maybe not? I don’t see $(DeployDir) mentioned here: (https://visualgdb.com/documentation/variables/)
Hi,
This is somewhat of a known trade-off. Visual Studio requests a lot of background information from VisualGDB quite often, so it somewhat limits how much details about the project it normally collects, and tries to cache as much as possible.
That said, detecting changes to host aliases and using the correct deployment directory for custom shortcuts is doable without too much slowdown. Feel free to try this build: VisualGDB-6.0.103.5211.msi.
- You must be logged in to reply to this topic.