Path mapping and breakpoint issues

[BitFlipper]

I use VGDB in a somewhat unusual way, mainly just for debugging. It does not take part in building or deploying the project. Because of the way our product source code works, we build it on a dedicated build farm and then I copy the binaries over to a Linux test machine. So it goes something like this:

1. Project Setup
A custom tool is used to configure the VS project as well as VGDB’s settings, including the remote debug host settings for VGDB. This is all automated so developers don’t need to open the VGDB configuration. This is done mainly to simplify setup and prevent misconfiguration since our project is quite complex.

2. Initiate Build
The custom tool is used to sync edited files from the Windows machine to the Linux build machine. The build is then initiated using the custom tool. At this point VGDB is not involved.

3. Copy Build to Test Machine
The custom tool executes commands on the build machine to copy the binaries over to the test machine (but source files are not copied). Still no VGDB involvement.

4. Start Debugging
VGDB is used for the debugging session.

So essentially I only want to use the debugging features of VGDB. It mostly works, but I have some weird breakpoint issues that is probably related to incorrect path mapping. Issues are as follows:

1. Sometimes I can set a breakpoint, however both VS and VGDB shows it as unresolved, yet the breakpoint is hit successfully. In this case VGDB says “The currently loaded GDB symbols have no record of any file called “somefile.c”. Please ensure that your library is loaded, contains debugging symbols and the symbols are loaded by GDB.”

Yet I’m able to step through the source file successfully and everything seems to work other than the breakpoint is shown as unresolved.

2. Other times breakpoints are just not hit. I followed the various troubleshooting steps but no luck.

Maybe I should ask how the mapping should be set up. Let’s say this is my environment:

Windows machine:
– Source code root: C:ProjectCode
– Note the VS project itself is an completely different path

Linux build machine:
– Source code root: /usr/project/code

Linux debug machine:
– Binary files root: /usr/project/buildoutput
– No source files on this machine

I tried a mapping like this:
/usr/project/code -> C:ProjectCode
…but it still doesn’t map it correctly. What would the correct way be?

[support]

Hi,

Your mapping seems to be set correctly and should work. Please follow the instructions below to diagnose it:
1. In the GDB Session window select ‘all GDB interaction’ so that VisualGDB shows the commands it sends to GDB automatically.
2. Set a breakpoint in a problematic file. VisualGDB will issue a -break-insert command. GDB will report whether it has been resolved.
3. If the breakpoint shows as unresolved, please run the following commands:
* info shared – this will show the loaded shared libraries. Ensure that the library that contains your code is loaded.
* info sources – this will list all source files known to GDB

If your library is not listed in the output of ‘info shared’ it has not been loaded yet, so please ensure it gets loaded and then repeat setting the breakpoint. If the library is shown and has the symbols loaded, but the ‘info sources’ does not list your source file among the source files, there is some problem with your symbols (either GCC is a lot newer or older than GDB or you are missing the -ggdb flag or you are stripping some symbols during build). If the file is shown, please compare the exact path of the file reported by ‘info sources’ with the path that VisualGDB used in the ‘-break-insert’ command. If they don’t match, you need to adjust the path mapping until they match.

[BitFlipper]

Note for the build machine path I changed “project” to “buildmachineproject” to make things clearer.

@bazis wrote:

2. Set a breakpoint in a problematic file. VisualGDB will issue a -break-insert command. GDB will report whether it has been resolved.

When I set a breakpoint, it takes a long time (~15 seconds), then the output appears to show success. It shows the correct path to the file as it was on the build machine, ie:

/usr/buildmachineproject/code/somefile.c

However VS shows the breakpoint as unresolved. When I click on “Diagnose breakpoint problems”, VGDB lists the file as problematic and the error message shows “The currently loaded GDB symbols have no record of any file called …”.

This breakpoint actually gets hit successfully though.

* info shared – this will show the loaded shared libraries. Ensure that the library that contains your code is loaded.

All libraries appear to be loaded successfully and all show “Yes” for “Syms Read”.

* info sources – this will list all source files known to GDB

This shows only one source file (a different one), and the full path is of the build machine pov. It also shows one other file for which symbols will be loaded on demand.

So the question is: Does the Linux debug machine need to have copies of the source files as well? Some files can have breakpoints set successfully so it seems it should not be required.

Something else that may be relevant… When I click on “Show a list of all source files”, I get a list that looks like this:

/usr/buildmachineproject/code/somefolder/source.c -> C:ProjectCodeusrbuildmachineprojectcodesomefoldersource.c

So it is appending the full build machine path to the local path.

I did the following tests… I cleared all manually added path mappings, then in VGDB Project settings under “File synchronization”, if I uncheck “Use project directory” and clear the path there, then the path mapping is reported as:

/usr/buildmachineproject/code/somefolder/source.c -> usrbuildmachineprojectcodesomefoldersource.c

So it is still wrong.

Next, whatever local path I put into the path in File synchronization, it gets prepended to the reported local path. So if I put “C:ProjectDummyPathCode” in there, the reported mapping looks like:

/usr/buildmachineproject/code/somefolder/source.c -> C:ProjectDummyPathCodeusrbuildmachineprojectcodesomefoldersource.c

No matter which path mapping I enter manually, they don’t seem to be honored, instead it always seem to use the above logic. So I’m unsure how to resolve the path mapping for my use case. I did check the gdb version, it is 7.3-0.6.1 which is pretty old I think. It might cause the issue, not sure.

Any more ideas?

Another question I have… Is it only necessary to add a path mapping to root paths, or do you need to add a mapping for each and every folder?

[support]

Hi,

The sources don’t need to be copied the the debug machine. Does the breakpoint reply contain the address it was resolved to or is it marked as ‘pending’? Can you send us the full GDB log? This will help us diagnose the problem further.
The manual mapping should work with any path prefix (try restarting debug session if it’s not updated immediately). E.g. if you want to map /usr/buildmachineproject/code/somefolder to c:projectcodesomefolder, you need to create the following mapping:

Linux path: /usr/buildmachineproject/code
Windows path: c:projectcode

If this still does not work, please send us the screenshots of your source file mapping window and the window showing incorrectly mapped paths.

[BitFlipper]

I have to make a correction to one of my previous comments… I said only one file was listed when I did “info sources”. I did not realize it listed all files on the same line. After looking at all files listed, I can see that it shows every file where I’m setting breakpoints. So the error shown by VGDB:

“The currently loaded GDB symbols have no record of any file called…”

…is confusing since “info sources” does show the file in question. Here is another weird thing: For one source file, the breakpoint is resolved successfully and shown as resolved. For another file in the same executable, it shows the breakpoint as unresolved but the breakpoint is hit. Both files are listed in “info sources”.

@bazis wrote:

Does the breakpoint reply contain the address it was resolved to or is it marked as ‘pending’? Can you send us the full GDB log? This will help us diagnose the problem further.

My company has strict policies regarding source code so I can’t give the exact file name paths, function names etc. So I will find/replace source code paths and function names to make it generic. But you should still be able to see the issue below.

So when I put a breakpoint on the first file (source1.c) where it resolves it properly (although it can take up to 30 seconds to place the breakpoint), the GDB output is as follows:

^done,bkpt={number=”9″,type=”breakpoint”,disp=”keep”,enabled=”y”,addr=”0x00007ffff50613d8″,func=”api1_connect”,file=”/usr/buildmachineproject/projectname/sources1/source1.c”,line=”4026″,times=”0″,original-location=”/usr/buildmachineproject/projectname/sources1/source1.c:4026″}
-break-info 9
^done,BreakpointTable={nr_rows=”1″,nr_cols=”6″,hdr=[{width=”7″,alignment=”-1″,col_name=”number”,colhdr=”Num”},{width=”14″,alignment=”-1″,col_name=”type”,colhdr=”Type”},{width=”4″,alignment=”-1″,col_name=”disp”,colhdr=”Disp”},{width=”3″,alignment=”-1″,col_name=”enabled”,colhdr=”Enb”},{width=”18″,alignment=”-1″,col_name=”addr”,colhdr=”Address”},{width=”40″,alignment=”2″,col_name=”what”,colhdr=”What”}],body=[bkpt={number=”9″,type=”breakpoint”,disp=”keep”,enabled=”y”,addr=”0x00007ffff50613d8″,func=”api1_connect”,file=”/usr/buildmachineproject/projectname/sources1/source1.c”,line=”4026″,times=”0″,original-location=”/usr/buildmachineproject/projectname/sources1/source1.c:4026″}]}
-break-after 9 0
OK
info symbol 0x7ffff50613d8
&”info symbol 0x7ffff50613d8n”
api1_connect + 30 in section .text of /usr/lib/company-foo-apilib/lib64/libApi.so
OK
info line *0x7ffff50613d8
&”info line *0x7ffff50613d8n”
Line 4026 of “/usr/buildmachineproject/projectname/sources1/source1.c” starts at address 0x7ffff50613d8 and ends at 0x7ffff50613e0 .
OK

Now when I put a breakpoint on a line in the second file (source2.c) where it says the breakpoint is unresolved, yet it still hits the breakpoint, the GDB output looks like:

-break-insert -f /usr/buildmachineproject/projectname/lib/apilib/source2.c:539
^done,bkpt={number=”7″,type=”breakpoint”,disp=”keep”,enabled=”y”,addr=”“,times=”0″,original-location=”/usr/buildmachineproject/projectname/lib/apilib/source2.c:539”}
-break-info 7
^done,BreakpointTable={nr_rows=”1″,nr_cols=”6″,hdr=[{width=”7″,alignment=”-1″,col_name=”number”,colhdr=”Num”},{width=”14″,alignment=”-1″,col_name=”type”,colhdr=”Type”},{width=”4″,alignment=”-1″,col_name=”disp”,colhdr=”Disp”},{width=”3″,alignment=”-1″,col_name=”enabled”,colhdr=”Enb”},{width=”18″,alignment=”-1″,col_name=”addr”,colhdr=”Address”},{width=”40″,alignment=”2″,col_name=”what”,colhdr=”What”}],body=[bkpt={number=”7″,type=”breakpoint”,disp=”keep”,enabled=”y”,addr=”“,times=”0″,original-location=”/usr/buildmachineproject/projectname/lib/apilib/source2.c:539”}]}
-break-after 7 0
OK

The manual mapping should work with any path prefix (try restarting debug session if it’s not updated immediately). E.g. if you want to map /usr/buildmachineproject/code/somefolder to c:projectcodesomefolder, you need to create the following mapping:

Linux path: /usr/buildmachineproject/code
Windows path: c:projectcode

If this still does not work, please send us the screenshots of your source file mapping window and the window showing incorrectly mapped paths.

That is pretty much how I set up the mapping. Using the same path structure as in the above GDB output, it looks like:

Linux path: /usr/buildmachineproject/projectname
Windows path: c:projectname

Then the reported mapping looks like:

Linux path: /usr/buildmachineproject/projectname/sources1/source1.c
Windows path: usrbuildmachineprojectprojectnamesources1source1.c

So it ignores the specified Windows path. However note I unchecked the “Use project directory” option under “File synchronization” and also cleared the “Local directory” path. Whatever I put in there will simply be prepended to the Windows path listed above, so it will always contain the “usrbuildmachineprojectprojectname” part.

I will try to provide as much info as required, but I might not be able to provide VGDB logs due to my company’s strict source code policies. Let me know what other info you require and I will see what I can do.

[support]

Hi,

It is always very hard to diagnose such problems without seeing the full GDB log because in most of the cases the root cause is some tiniest difference between different representations of a path. Can you reproduce the problem on a ‘hello, world’ project on the same machine? The only thing we can do is show you the GDB logs on our side demonstrating normal VisualGDB behavior so that you can compare them with yours and search for abnormalities.

Regarding the breakpoint that is shown as unresolved, looks like it corresponds to multiple addresses, but GDB does not report them in a way VisualGDB expects it. Here’s a sample response from setting a breakpoint in a template function that has 2 instances:

^done,bkpt={number="4",type="breakpoint",disp="keep",enabled="y",addr="",times="0",original-location="/tmp/VisualGDB/e/PROJECTS/TEMP/LinuxProject10/LinuxProject10/LinuxProject10.cpp:7"},{number="4.1",enabled="y",addr="0x080486fd",func="f<1>()",file="LinuxProject10.cpp",fullname="/tmp/VisualGDB/e/PROJECTS/TEMP/LinuxProject10/LinuxProject10/LinuxProject10.cpp",line="7"},{number="4.2",enabled="y",addr="0x08048707",func="f<2>()",file="LinuxProject10.cpp",fullname="/tmp/VisualGDB/e/PROJECTS/TEMP/LinuxProject10/LinuxProject10/LinuxProject10.cpp",line="7"}

Your output looks different. Which platform/GDB version are you using?

Regarding path mapping, here’s a sample path mapping that maps /tmp/test to c:something:
[attachment=1:1xtzhfzt]pathmapping.png[/attachment:1xtzhfzt]
Here’s how the /tmp/test/test.h is mapped to c:somethingtest.h:
[attachment=0:1xtzhfzt]sourcelist.png[/attachment:1xtzhfzt]

Does it work for you with exactly the same paths? Do you have any symbolic links involved (e.g. /tmp is mapped to /system/tmp and while you set /tmp, GDB reports it as /system/tmp)?

If a breakpoint shows as resolved but is not hit, does the code at the address reported by GDB match the expected function body (e.g. is it equivalent to the disassembly of your ELF file)? If yes, does placing a manual breakpoint there (asm (“int3”); if you are using GCC) work? Can the memory be write-protected so that GDB fails to set the breakpoint?

Are you running GDB on your target machine or are you using a gdbserver? Are you using a .vgdbsettings file or quick debug?

[BitFlipper]

OK I will try to send you the VGDB log later today. Where can I send it to as I don’t want to post it here publicly?

[support]

Please use support at sysprogs dot com.

[BitFlipper]

I wonder if this could be related to the fact that VGDB is used in a non-standard way in our case. We change the build, rebuild and clean command line settings in the VS NMake project to that of our own tool. So VGDB does not partake in any of the build phases so maybe it is causing some issues. VGDB does complain that the VS project settings are inconsistent but up until this issue we have not seen any other side effects that interfere with the debugging functionality.

I will try to repro this first with a hello world application since it will be easier to share the logs.

EDIT: In the meantime here is some relevant info…

Windows 7 Enterprise SP1 x64
VS Ultimate 2012 Update 4
VisualGDB 4.1
Debug OS: SLES 11, Kernel 3.0.13-0.27-default
GDB 7.3-0.6.1

[BitFlipper]

I believe I was able to resolve the file mapping issue. The main problem here was my misunderstanding of what some of the settings under “File synchronization” means. I was under the impression that this was only needed when you want to sync files between systems, which I don’t need. However it seems that both “Local directory” and “Remote directory” actually play a role in how file mapping is resolved at debug time, regardless of what manual mappings you add.

For instance, if you leave “Remote directory” blank, then you can reproduce the issue where the local Windows paths always have the remote path appended at the end.

I am still not able to resolve the issue with some breakpoint that show as unresolved but that are hit correctly. My efforts to reproduce this with the hello world application did not succeed. I will do some more testing tomorrow and hope to provide some more info then.

[Author]

Posts