Problem debugging custom kernel

Sysprogs forums Forums VisualKernel Problem debugging custom kernel

This topic contains 13 replies, has 2 voices, and was last updated by  support 8 months ago.

Viewing 14 posts - 1 through 14 (of 14 total)
  • Author
    Posts
  • #27065

    Mischo
    Participant

    Hello,

    I’m trying to debug over Ethernet custom kernel for i.MX7 from fsl-community-bsp BSP with sources and configuration generated with Yocto.
    I had copied kernel sources to other folder and setup VisualKernel advanced kernel project. For build I’m using toolchain from Yocto generated SDK for target image. Build finished correctly without errors.
    Next I deployed image to running device originally flashed with Yocto result image by replacing zImage in boot partition. Device boots correctly and uname -v displays correct build time and version.
    When I attach debugger and try to unload module with breakpoint set in print_modules() as displayed in tutorial , I get Received a SIGTRAP: Trace/breakpoint trap error from Visual Studio 2015 Enterprise.

    Output from debug console:

    Same thing happened, when I tried to debug image build directly from Yocto loading its symbols using VisualKernel Quick Debug Linux Kernel option, probably when it should hit automatic breakpoint after attach.

    What could be a problem?
    Thanks in advance.

     

    • This topic was modified 8 months, 1 week ago by  Mischo.
    #27071

    support
    Keymaster

    Hi,

    This might indicate that the KGDBoE module (debugging over Ethernet) is not fully compatible with your target, or it could indicate a symbol problem.

    If you could attach a gdb log from the debug session, we should be able to tell what is going on. Please also consider debugging the target via JTAG, as it is generally less fragile.

    #27077

    Mischo
    Participant

    Thanks for quick response. Log and .config file is in attachment. KGDBoE shows no error on build and looks like it is installed successfully. In meantime I’m trying to get debug running using LPC-Link 2 probe JTAG with CMSIS-DAP firmware.

    Attachments:
    You must be logged in to view attached files.
    #27079

    Mischo
    Participant

    .config file

    • This reply was modified 8 months ago by  Mischo.
    Attachments:
    You must be logged in to view attached files.
    #27082

    Mischo
    Participant

    Kernel module project running on Kernel build from VisualKernel and set “Kernel type” to it (so symbols should match) throws same error. Output from debug UART on device:

    Log is in attachment.

    Attachments:
    You must be logged in to view attached files.
    #27084

    Mischo
    Participant

    I have tried to debug kernel module project (I assume it is easier for debugger first setup than Kernel debugging) with J-LINK using OpenOCD method and it fails with “Failed to connect to debug stub” error. Test connection passes correctly for J-LINK. GDB stub log and VisualKernel settings image are attached.

    Attachments:
    You must be logged in to view attached files.
    #27095

    support
    Keymaster

    Hi,

    Thanks for providing the detailed description.

    It looks like the KGDBoE-based debug session works. The stop happens inside the entry-common.S file and is likely by design:

    Normally, VisualKernel would open the entry-common.S file in Visual Studio once that stop happened, however depending on how you imported the kernel into it, it may not know where to locate it. If this is the case, you can setup a manual mapping between the paths reported by gdb (e.g. /home/build/fsl-community-bsp/buildOutput/tmp/work-shared/imx7dsabresd) and the paths on the Windows machine via VisualKernel Project Properties -> Path Mapping.

    Regarding JTAG, most likely your firewall is blocking the connection (gdb running on the build machine needs to connect to OpenOCD running on the Windows machine), or the build machine is not able to resolve the Windows machine‘s host name due to missing DNS entries. Please double-check the firewall settings and the gdb log (search the gdb log for “remote” to find out the host/port used by VisualKernel). You can override the “remote” command via VisualKernel Project Properties -> Debug Settings -> Advanced.

    #27105

    Mischo
    Participant

    Thanks for response.

    I forgot to mention, that debugger shows entry-common.S file in editor on correct line so symbols are probably working correctly. Problem is that I cannot continue because it stops immediately on the same line (284) when I try to continue (I have no breakpoint there). Code looks like:

    It breaks on
    b ret_slow_syscall

    I’m attaching the whole file (renamed because does not allow to upload it with original name).

    For JTAG I will try to play with network settings on Windows machine.

    • This reply was modified 8 months ago by  Mischo.
    Attachments:
    You must be logged in to view attached files.
    #27109

    support
    Keymaster

    Thanks for the clarification. It looks like some patches or configuration in the kernel you are using make it incompatible with KGDBoE.

    Generally KGDBoE is less reliable than other debug methods as it relies on several assumptions about the network driver implementation that don’t always hold.

    If your board has JTAG pins available, using it instead of KGDBoE should result in much more consistent and reliable experience. Let us know if you need help understanding the connectivity issues between gdb and OpenOCD.

    #27114

    Mischo
    Participant

    Thanks.

    I managed to get through GDB error for JTAG connection (was GDB side problem on build machine). Now VisualKernel attaches to kernel module without any error notifications, but it does not hit breakpoint in LinuxKernelModule1_init() function and “LinuxKernelModule1: Hello, world!” is printed correctly to debug UART. Breakpoint changes to transparent as when no symbol files are loaded when normally debugging in Visual Studio, but it would be really strange to VisualKernel would not be able to load symbols for module it just built and deployed to device. I’m attaching log, GDB printed line 487 as last until I killed debug session.

    Attachments:
    You must be logged in to view attached files.
    #27118

    Mischo
    Participant

    I have tried to change “Obtain module information via:” to “Optimized helper module”. Looks like it at least detect module in “Modules” tab and symbols tab points to correct object, but now I get error in UART debug console and module does not run. Without Optimized helper module I cannot see module in “Modules” tab (lsmod displays it correctly on device) so it is probably cause of why debugging symbols are not loaded in my previous post.

    #27120

    support
    Keymaster

    Hi,

    It looks like the instead of passing the breakpoint event to the JTAG debugger, the kernel tries handling it directly. This might also be the cause for kgdboe error you encountered before.

    Please try adding the following startup command to VisualKernel Project Properties -> Startup Commands -> Before Connecting to Target:

    This will force OpenOCD to use hardware breakpoints instead of software breakpoints. Depending on the way exception handling is implemented in this kernel port, it may resolve the issue.

    #27149

    Mischo
    Participant

    Hi,

    I have tried to change to hardware breakpoints and it now breaks with SIGTRAP on first line in

    However I was able to start debugging with workaround correctly using J-LINK with official driver and J-Link GDB server. Next I setup Custom kernel connection on Host/Port provided by GDB server and select Before debugging, target is: Crashed/frozen in Kernel session tweaking category, because GDB server puts MCU to halted state automatically. Kernel module is not loaded automatically this way, so It must be loaded manually over SSH or from GDB session tab.

    Thanks for all support.

    #27173

    support
    Keymaster

    No problem. Most likely, the chip you are using handles the exceptions slightly differently from the way OpenOCD would expect them and hence prevents the breakpoints from being handled correctly. Normally, this should be fixed in one of the upcoming OpenOCD updates.

    Either way, if J-Link software works better, you can use the following workaround to avoid the “Attach to crashed/frozen target”. Try creating a gdb script with the following contents:

    It instructs gdb to connect to the J-Link gdb stub, resume the target and disconnect. You can run it via command line as shown below:

    Or alternatively, add it to VisualKernel Project Properties -> Custom Debug Steps -> Before Debugging. This will allow VisualKernel to connect to the target via SSH and handle deployment/module enumeration as if it does with regular debug sessions.

Viewing 14 posts - 1 through 14 (of 14 total)

You must be logged in to reply to this topic.