![\"\"](\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/trace-1.png\")
<\/a>Unlike the regular breakpoints that require a non-trivial debugging setup (e.g. JTAG), tracepoints work over a regular network connection and do not interfere with the normal kernel operation. Each time a tracepoint is hit, it quickly records the pre-selected data into an internal memory buffer, and immediately resumes the execution.<\/p>\nSupported Targets<\/h2>\n
Live Tracing is supported on all kernel targets that have a usable network connection and are built with kprobe<\/strong> or ftrace<\/strong> support (most stock kernels do). It can be used within a regular debugging session<\/a> (e.g. using KGDBoE or JTAG), or independently<\/a> without having to setup a debug connection. The second option is useful on targets where the regular means of debugging are not available.<\/p>\n Tracepoints can be created during a tracing session, or in advance. There are 3 different ways to create them:<\/p>\n Each tracepoint can capture the values of arbitrary global or local variables. It can descend into pointers (automatically validating them), handle arrays, and capture NULL-terminated strings: Dynamic arrays (plain byte arrays or arrays of structs) can be captured by selecting “Dynamic Array” in the “Show As” column, and then drag-and-dropping another variable in the “Number of elements” field below: You can set multipliers (e.g. capture 2x[count] elements) and maximum counts to avoid depleting the trace buffer space.<\/p>\n You can view the events produced by one or more tracepoints by selecting them in the Tracepoints window, or switching to the Events<\/strong> view to view all events: You can also view events by clicking the tracepoint annotations directly in the code: (if enabled via the Tracepoints<\/strong> window): You can quickly make sense of large amounts of traced data using the table and graph views. Views can be created via the button in the Trace Data<\/strong> window when recording a trace, or viewing an existing trace file: Graph views work similarly to table views – they plot the values of selected variables as they were captured by the tracer. The X axis can show the event number, or an arbitrary variable (e.g. jiffies<\/strong> for the system clock value): Tracepoints can be grouped in virtual folders. Selecting a folder will automatically show events from all tracepoints within it (including subfolders): If you are tracing a kernel module project, it will automatically recognize when your module is loaded, and will place the tracepoints inside it. You can manually load\/unload the main module via the button on the GDB Session window (when tracing during regular debug sessions) or the Tracepoints window (when doing trace-only sessions):Creating Tracepoints<\/h2>\n
\n
<\/a><\/li>\n<\/a><\/li>\n<\/a><\/li>\n<\/ol>\nSelecting Traced Data<\/h2>\n
<\/a>Generally, if a variable value would be visible from a gdb breakpoint, it will be traceable if you set a tracepoint in the same location.<\/p>\n<\/a><\/p>\nViewing Trace Data<\/h2>\n
<\/a><\/p>\n<\/a><\/p>\nData Views<\/h2>\n
<\/a>Once a table view is created, you can select drag items from the Trace Data<\/strong> window into the bottom part of the table view, and they will be automatically converted to data columns. Each data column can refer to one or more data items from one or more tracepoints. E.g. creating a “Name” column capturing the file name from sys_openat2()<\/strong> and a ‘fd’ column creating the ‘fd’ variable from open<\/strong>, read<\/strong>, write,<\/strong> and close<\/strong> syscalls can give an instant overview of all read\/write oprations going along in the kernel. The screenshot below shows the system calls involved in copying \/etc\/fstab<\/strong> to \/root\/fstab<\/strong>:<\/a>You can sort the table rows by clicking any column header, or simply export the entire table into a CSV file via the export button.<\/p>\n<\/a><\/p>\nGrouping\/Exporting Tracepoints<\/h2>\n
<\/a>You can also export the tracepoints (along with the data configuration) into XML files, and import them to other projects.<\/p>\nKernel Modules<\/h2>\n
<\/a>You can also place tracepoints in other modules, as long as you specify their names via VisualKernel Project Properties -> Live Tracing -> Traced in-tree modules<\/strong>:<\/a><\/p>\nTutorials<\/h2>\n