I’m trying to debug a module for a custom PCIe video capture card. I’ve tried Ubuntu 14.04, 16.04 and have even rebuilt the kernels. Everything is up-to-date. I’ve followed the tutorials and the target machine completely hangs with both network debugging and KDB over a null modem cable. It’s a physical DELL quad core with stock everything except the video card and it hangs no matter where breakpoints are set. It will break into the debugger when the first breakpoint is hit but won’t resume or single step. Please advise.
The Linux kernel is unfortunately not as well-designed in terms of debugging as the Windows one, so unless you are using JTAG or VMWare that runs on top of the kernel, random hanging could happen. Our best advice would be to try disabling all CPU cores except #0. This should reduce the chance of deadlocks in the kernel.
Also if this is possible, we would advise getting a special machine with a JTAG connection. Although this is slower than debugging over serial port, this will be 100% reliable as the debugging logic will run on top of the kernel without sharing any resources with it.
I can appreciate the complications with a process debugging itself but it seems the claims of the product’s capabilities are a bit disingenuous. Has the JTAG debugging on x86 been tested? If so, what setup was used?
Sorry about the confusion. We have only tested VisualKernel with JTAG on ARM. Regarding product capabilities, VisualKernel automates and simplifies the kernel debugging for configurations supported by the Linux kernel. The only exception is added support for debugging over Ethernet (KGDBoE), however it is not very reliable due to the limitations of the Linux Kernel.