[MST]

Hi,

we made a slight update of our kernel version (5.10.174 -> 5.10.175) and now the live tracing fails to build. I tested build 2367 and 2372, both show the same error message:

Run "make -C "/dev/host/kernelsrc/linux-5.10.175" M="/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb" modules DISABLE_TRACE_ENGINE=1" in directory "" on local computer
--------------------------
make: Entering directory '/dev/host/kernelsrc/linux-5.10.175'
CC [M] /tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/AppendableBuffer.o
CC [M] /tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/LinuxKernelDebugHelper_main.o
CC [M] /tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/SequentialModuleInfo.o
CC [M] /tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/LiveMemoryEngine.o
CC [M] /tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/TraceEngine.o
make: Leaving directory '/dev/host/kernelsrc/linux-5.10.175'
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/LiveMemoryEngine.c: In function 'LiveMemoryEngineThread':
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/LiveMemoryEngine.c:178:11: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
178 | if (!traceEngine)
| ^~
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/LiveMemoryEngine.c:181:5: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
181 | iov.iov_len = TraceEngineProcessMessage(traceEngine, &packet, &remoteAddr, done);
| ^~~
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/TraceEngine.c:1171:109: warning: 'struct ftrace_regs' declared inside parameter list will not be visible outside of this definition or declaration
1171 | static notrace void FTraceHookImpl(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *regs)
| ^~~~~~~~~~~
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/TraceEngine.c: In function 'ApplyTracepointState':
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/TraceEngine.c:1253:34: error: assignment to 'ftrace_func_t' {aka 'void (*)(long unsigned int, long unsigned int, struct ftrace_ops *, struct pt_regs *)'} from incompatible pointer type 'void (*)(long unsigned int, long unsigned int, struct ftrace_ops *, struct ftrace_regs *)' [-Werror=incompatible-pointer-types]
1253 | tp->AddressBased.FTrace.func = &FTraceHookImpl;
| ^
At top level:
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/TraceEngine.c:776:44: warning: always_inline function might not be inlinable [-Wattributes]
776 | static bool __attribute__((always_inline)) TraceEngineRollbackWrite(struct TraceTransaction *transaction, size_t size)
| ^~~~~~~~~~~~~~~~~~~~~~~~
/tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/TraceEngine.c:763:46: warning: always_inline function might not be inlinable [-Wattributes]
763 | static void * __attribute__((always_inline)) TraceEnginePrepareForWrite(struct TraceTransaction *transaction, size_t size)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:286: /tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb/TraceEngine.o] Error 1
make: *** [Makefile:1826: /tmp/LinuxKernelDebugHelper-acf78497-911a-4b6b-8f13-53c16f2060bb] Error 2
--------------------------
Command exited with code 2

 

[MST]

(Cannot edit my previous post…)
Ok, I was too quick. I switched back to the previous build (2367) because the “Trace Data” windows is completely empty in your latest build (2371) (at least for me). No data is shown for any event. No options are shown when I try to edit an existing tracepoint. The options bar at the top is also not shown. It’s really an empty window.

Furtunately tracing still works in the “old” build via the tracingpoint bar 🙂

[MST]

Hi,

just a first quick feedback: Tracing the hid_irq_in works with the new build 🙂 I couldn’t set tracepoints via the tracepoint bar before. This also works now and is a very nice feature.

[MST]

Hi,

I tested the tracing some more and it’s a really nice feature. But I had some issues during my tests that I want to let you know:

1. The tracing does not work in the “three machine setup”. It seems like the “tracer” is looking for the vmlinux file on the windows pc but tries to use the linux path. You can see the screenshot of the error message in the attachment. I switched to the “two machine setup” for further testing.
2. I tested the debugging with the usbhid driver and stopped the system by pressing a random key and a breakpoint within the “hid_irq_in” function (hid-core.c). I tried to trace the same function but the option to add a trace point to the function (the red dot) is not available (see Tracepoint.png). I checked some more functions in “hid-core.c” and recognized that I can add tracepoints to some functions (e.g. “hid_reset”, “usbhid_mark_busy”, …) but not all of them. Do I have to (somehow) update the database of the traceable functions or what is happening here?

Attachments:

You must be logged in to view attached files.

[MST]

hi,

unfortunately I am not allowed to hand out any debug information. But we do not have any modifications to the default kernel code. The error didn’t occur anymore. Maybe it “magically” dissapeared with the recent build (or I make a mistake before ^^).

I still have some issues starting the debugging at the beginning and I don’t really know why it happens. The commands by visual kernel and my commands look exactly the same as far as I can see it in the diagnostics console. Could you maybe add an option that removesthe logging of the “stat” commands. They are really spammy and slow down visual studio due to all the output.
My recipe right now to start the debugging:

1. Start debugging with “accessible via SSH” –> this will build/copy all files (if required) and start gdb. This usually hangs when gdb tries to connect “-target-select remote /dev/pts/5”.
2. Stop debugging, change options to “Crashed/frozen” and restart debugging. In more than 90 % I am able to debug now. From this point on I can also also restart the debugging when the debugee is accessible via SSH (and of course setting this option).

This is a valid solution for me because I can reproduce it more or less all the time (if the machine is not killing itself due to some timeouts in the stopped kernel threads ^^).

[MST]

Hi again,

I did a testing session with your new version today. Unfortunately, unsuccessful. Some things seems to have gotten worse with this new version.

1) I like the prompt that you included when compiling the HelperModule when there is an issue during compilation of the TraceEngine. My kernel (5.10.174) seems to be uncompatible with your code (still the same error “/tmp/LinuxKernelDebugHelper-444e4d9f-85c5-4f74-a164-a74130863526/TraceEngine.c:1172:61: error: dereferencing pointer to incomplete type ‘struct ftrace_ops’
1172 | struct TracepointObject *tp = (struct TracepointObject *)op->private;” as before) but then I can skip this part (“continue without tracing”) and still use the HelperModule. Since the trace engine is required for the live tracing – just recognized the “error message” “System.Exception: The LinuxKernelDebugHelper module was built without tracing support. Tracing will be disabled until the target is restarted.” – I am not able to test this feature right now. Nevertheless I tried scanning the list of traceable functions and got stuck at 99 % at 4372 of 4374 files repeatingly (the files varies – strange). I once even waited for more than an hour but no more progress happens (takes ~ 12 minutes to get to this point).

2) Considering the manual switch to kgdb: I can enable this feature in the project file and the window appears and waits for my input to continue. Unfortunately kdb does not switch to kgdb when I enter “kgdb” in the console but returns to its bash command prompt (= leaves the debug mode) and consequently the debug session crashes. This does not happend when I call tha sysrq-g manually, but I enters kgdb state then. Maybe you are doing some more commands that my system does not like. I will try to fix all this KDB/KGDB issue on a “brute force way”: I asked my colleague for a new kernel version without kdb. I hope to get this tomorrow and see if this fixing my issues in this sector.

greets,
MST

[MST]

Hi,

thanks for the help again. It would be create if you could make the KDB->KGDB trigger send manually because I’m still stuck quite often in it. A had a short glimpse at the diagnostic log (love the feature) but it is spammed by synchronization commands. Will further look at it next week.
I couldn’t test your other remarks yet.

This new feature sounds nice. How can I test it and see if it’s feasible with our machines?

greets,
MST

[MST]

Hi,
I tried your advice but unfortunately I am not successful.
Let me explain: My setup for this is a “three machine setup” so I am using a linuxVM to debug. The connection via quick debug works and I reach the breakpoint in “kgdb_breakpoint”. So in principle debugging works.  As you see in the attached KernelSymbols.png the kernel symbols are cached on the VM. I get the message, that the “selected symbols match the running kernel”. But how can I access the source code that is connected to my actual debug run? In visual studio there is only “debug_core.c” open and when I see that I is cached on my windows PC from the VM. But it is the only source file in the cache directory (see CacheDirectory.png)! How can I cache the rest of the source code? Or is there another option? Can I copy it manually or will visual kernel not recognize it?

On your second remark: I tried adding and existing project but the dialog won’t show me .vkrnlrelease files (see ProjectChoices.png). How can I add them?

greets,
MST

Attachments:

You must be logged in to view attached files.

[MST]

Hi,

the forum “ate” my first version of my reply. I’ll hope not to forgot anything in this second try …

1) Changing the include works but there are compilation errors after that. I attached the error log. There seems to be some issues with the live tracing in the TraceEngine. I’m using kernel 5.10.174 without any modification to the source code.

2) Concerning KDB -> KGDB: I am testing two setup: including a VM to build an debug (“three machine mode”) and direct debugging on windows and building the modules on the debugee (“two machine mode”).
2.1) In the “three machine mode” I am able to multiplex the serial port via KDMX and hence I am able to access the serial console even while gdb is running. This allows me to manually enter “kgdb” if required (it does not happen all the time) – your echo commands works too 🙂
2.2) I have more issues with the “two machine mode” because here I can only monitor the serial port (via “serial monitor”) and not access it once gdb is running. Disconnecting gdb kills my debuggee quite often (or at least kills the networking…) which requires to restart the debuggee. I tried adding the echo as a “custom debug step” “before launching the debugger” but this seems to be executed prior to the sysrq-g and hence before the debuggee enters KDB. In some cases the debugger is able to switch to kgdb but far from all the time. After that the debuggee is most of the time rather unstable or at least frozen. In this state there is no ssh connection possible. Usually (in the “three machine setup”) I can recover from this state by changing the debug setting “before debugging, the target is” to “Crashed/frozen” but in the “two machine setup” there is another issue at this point: After changing the setting visual kernel tries to synchronize its setting to the debuggee via ssh – which is not available! This means that I cannot do it that way and that means that I have to restart the debugee.
All in all the “three machine setup” seems to be more reliable and stable for me. Will it be quicker when using the HelperModule? I am fighting to improve the performance/reliability right now.

greets,
MST

Attachments:

You must be logged in to view attached files.

[MST]

Hi,

a second question with the new build (can’t edit my previous post). There is an option to “Explicitly switch from KDB to KGDB” in the debug options. This is a great idea imo because I’m often getting stuck in KDB when debugging –  especially after restarting my debuggee. Usually I have to start debugging, stop it and restart it with the “crashed/frozen” setting, stop the debugging again. At this point my machine can usually be accessed via ssh and I can start a normal debug session.
Should this option improve this interaction? Unfortunately I don’t see any effect of checking or unchecking it yet.

greets,
MST

[MST]

Hi,

I tried your new build and I like the updates in the interfaces.

But I get a compilation error when starting the debugging with the helper module option switched on (stdarg.h not found, in TraceEngine.c). I am pretty sure it’s related the the changes described e.g. in this link https://github.com/bondagit/aes67-linux-daemon/issues/64 . My systems are kernel 5.10.173 (debugge, custom config) and kernel 5.19.0-38 (debugger, ubuntu 22.4 lts).

greets,
MST

[MST]

Hi,

thank you very much for these hints. Because of these it could fix the problem.
In my case the issue was that I named the kgdboc port in the options gui (this is the same as the <KgdbocArguments> element in the .vkrnlproj file) “/dev/ttyS4” and this is somehow not permitted by the “echo <args>” > /sys/module/kgdboc/parameters/kgdboc” command. Skipping the path (“/dev/ttyS4” -> “ttyS4”) fixed the kgdboc registration errors! Maybe you should add this as a hint to the ui/documentation or check for the path while running the bash scripts and remove it?

I tested this also manually by running
echo /dev/ttyS4,115200 > /sys/module/kgdboc/parameters/kgdboc
which returns
-bash: echo: write error: No such device

This is the error message when adding the path to the tty. At the same time the kgdboc interface is unregistered.
Removing the path from the command (echo ttyS4,115200 > /sys/module/kgdboc/parameters/kgdboc) gives no message at all but one can se the unregistration and reregistration of the kgdboc interface in dmesg.

thank you very much again, 🙂

MST

• This reply was modified 1 year ago by MST.

[MST]

Hi,

thanks for your support but the problem still remains after increasing the time out time (even to 100).

Why and at what point does VK try to reset the kgdboc interface? Because this is the issue as I can observe it in the gdb log (“Cannot configure KGDB. Please try configuring it manually by writing to /sys/module/kgdboc/parameters/kgdboc.”) and in the dmesg

[21107.877453] KGDB: Unregistered I/O driver kgdboc, debugger disabled
         +[21108.350362] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-
memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z)

and at this moment the sysrq-g command is disabled and hence the machine cannot enter debug mode. I also tried to build some workaround by making these calls manually by adding calls prior to the start of the debugger (via the visual kernel options menu) – but unfortunately without success. The kgdboc interface is always killed by visual kernel as soon as it enters gdb. You can see my commands in the visual kernel launcher output (LauncherOutput.log). The “SetKGDBOC.sh” scripts simply runs “echo ttyS4,115200 > /sys/module/kgdboc/parameters/kgdboc” and enables the kgdoc interface. “SendSYSRQ-g.sh” is also a self describing one line script and a leftover of my tests (just ignore the line, its not executed 😉 ).
With this setup dmesg looks like this:

   [24847.242012] KGDB: Registered I/O driver kgdboc     
         [24853.202471] KGDB: Unregistered I/O driver kgdboc, debugger disabled
        +[24853.671324] sysrq: HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-
         memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) force-fb(v) show-blocked-tasks(w) dump-ftrace-buffer(z)

The first line of this dmesg log (activation of kgdboc) is done by “SetKGDBOC.sh”. The unregistration after that is caused be visual kernel (as explained at the top). The sysrq error is caused by the missing kgdboc.

I hope this helps you to have a further look at this issue. Could it be an option to keep the kgdboc interface untouched by visual kernel (or make it optional?).

greetings,
MST

Attachments:

You must be logged in to view attached files.

[MST]

Hi,

I cannot edit my previous post a second time.
I found another issue with the new build when running (debugging) my Pi3B KernelModule project.
The error message given is:
“Debugging failed: Value cannot be null. Parameter name: pane”

Logs for more details are attached.

Attachments:

You must be logged in to view attached files.

[MST]

Hi,

there seems to be another glitch in the updated version. It’s happening at the same line:

KERNEL=kernel7l ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make olddefconfig
/*edit*/
It’s likely an issue of an incorrect directory where the make command is called than a wrong command.
/*end edit*/

Full log is attached.

• This reply was modified 1 year, 2 months ago by MST.

Attachments:

You must be logged in to view attached files.

[Author]

Posts