{"id":746,"date":"2023-05-29T20:24:12","date_gmt":"2023-05-30T03:24:12","guid":{"rendered":"https:\/\/sysprogs.com\/tutorials\/?p=746"},"modified":"2023-05-29T20:24:12","modified_gmt":"2023-05-30T03:24:12","slug":"tracing-linux-kernel-code-without-a-debug-connection","status":"publish","type":"post","link":"https:\/\/sysprogs.com\/VisualKernel\/tutorials\/tracing\/nodebug\/","title":{"rendered":"Tracing Linux Kernel Code without a Debug Connection"},"content":{"rendered":"<p>This tutorial shows how to trace the Linux Kernel code (i.e. record variable values at arbitrary code locations) without having to setup regular debugging connection. This allows tracing targets that would otherwise be hard to debug (e.g. embedded devices without JTAG, desktops\/laptops incompatible with KGDBoE, etc).<\/p>\n<p>In this tutorial we will create a kernel module implementing a RAM disk, will run it on a non-debuggable VM, and will show how to use tracing to see what sectors are being read and written. Before you begin, install VisualKernel 4.1 or later.<\/p>\n<ol>\n<li>Start Visual Studio and open the VisualKernel Linux Kernel Module wizard:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/01-new.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-747\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/01-new.png\" alt=\"\" width=\"1014\" height=\"675\" \/><\/a><\/li>\n<li>Enter the name and location for your project: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/02-name.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-748\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/02-name.png\" alt=\"\" width=\"1014\" height=\"675\" \/><\/a><\/li>\n<li>Select the &#8220;Ramdisk&#8221; module template and click &#8220;Next&#8221;: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/03-ramdisk.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-749\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/03-ramdisk.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>Select the target where you would like to run the module. In this tutorial we will use a VMWare virtual machine that is itself running under Hyper-V, making it not debuggable using the regular means: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/04-host.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-750\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/04-host.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>Proceed with the default source code access settings:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/05-src.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-751\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/05-src.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>The <strong>Debug Settings<\/strong> page of the wizard normally allows selecting a debug method (e.g. JTAG). However, starting from VisualKernel 4.1, you can also choose &#8220;Use Live Tracing&#8221; to completely bypass it and use a regular network connection instead: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/06-traceonly.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-752\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/06-traceonly.png\" alt=\"\" width=\"886\" height=\"693\" \/><\/a><\/li>\n<li>Press &#8220;Finish&#8221; to create the project, then build the entire solution. Once it is built, press F5 to start a tracing session. When it launches, click on the tracepoint icon near the <strong>AllocateBasicRamDisk()<\/strong> function: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/07-start.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-753\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/07-start.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a><\/li>\n<li>Check both function arguments, set <strong>pDiskName<\/strong> type to <strong>NULL-terminated string<\/strong>, and click &#8220;OK&#8221; to create a tracepoint:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/config.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-764\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/config.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a>You can also use the <strong>Debug-&gt;Windows-&gt;Tracepoints<\/strong> command to trace any function in your module (or the kernel itself) by name.<\/li>\n<li>Use the green bulb icon on top of the <strong>Tracepoints<\/strong> window to reload the module. VisualKernel will show that the <strong>AllocateBasicRamDisk()<\/strong> function got called, and will display the values of the arguments:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/08-event.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-754\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/08-event.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a>Note that unlike breakpoints, that stop the target, tracepoints immediately capture the selected data in an internal memory buffer, and resume it. Hence, tracepoints do not interfere with the regular kernel operation, and do not require any special debugging setup other than a regulra network connection.<\/li>\n<li>Now we will set another tracepoint using the tracepoint bar. Hover the mouse to the left of the <strong>is_write<\/strong> check in <strong>BasicRamDisk_Transfer()<\/strong> until you see the gray lightning icon. Click on it to begin creating a tracepoint:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/09-side-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-766\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/09-side-2.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a><\/li>\n<li>Select the data you would like to trace and press OK: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/10-data.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-756\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/10-data.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a><\/li>\n<li>On the target, run the &#8220;mkfs&#8221; command to initialize the RAM disk we created: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/11-mkfs.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-757\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/11-mkfs.png\" alt=\"\" width=\"905\" height=\"523\" \/><\/a><\/li>\n<li>The tracepoint we created will immediately report multiple events. You can click through them to see the captured data in each case. You can flag individual events as favorite, rename them to provide meaningful descriptions, or rename the individual tracepoints to more meaningful names:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/13-rename.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-759\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/13-rename.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a><\/li>\n<li>Navigating multiple events producing loads of data could be challenging. To make it easier, VisualKernel allows creating table and graph views to make better sense of them. Go to the Trace Data window, click the Views button and select &#8220;Create a new table view&#8221;:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/12-events.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-758\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/12-events.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a><\/li>\n<li>Drag the <strong>sectorNumber<\/strong>, <strong>len<\/strong> and <strong>is_write<\/strong> variables from the <strong>Trace Data<\/strong> window into the bottom part of the view. This will create a 3-column table showing all events that captured these variables:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/14-table.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-760\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/14-table.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a>Each column can have multiple variables from different events (e.g. file handle from both <strong>read()<\/strong> and <strong>write()<\/strong> calls).<\/li>\n<li>Click the <strong>sectorNumber<\/strong> column to sort the table by sector number. You can then set the custom event description to easily find it later:<a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/15-rename.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-761\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/15-rename.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a><\/li>\n<li>Click the &#8220;Export to CSV&#8221; button next to the Graph\/Table view switch. VisualKernel will export the event data into a CSV file that can be used to analyze the data further: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/16-csv.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-762\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/16-csv.png\" alt=\"\" width=\"1241\" height=\"485\" \/><\/a><\/li>\n<li>Once you are done tracing, press Shift-F5 to end debugging session. VisualKernel will save a trace report that can be loaded and replayed later. For convenience, you can rename the reports, or provide additional descriptions to them: <a href=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/17-session.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-763\" src=\"https:\/\/sysprogs.com\/tutorials\/wp-content\/uploads\/2023\/05\/17-session.png\" alt=\"\" width=\"1391\" height=\"837\" \/><\/a><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>This tutorial shows how to trace the Linux Kernel code (i.e. record variable values at arbitrary code locations) without having<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[59],"tags":[62],"_links":{"self":[{"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/posts\/746"}],"collection":[{"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/comments?post=746"}],"version-history":[{"count":2,"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/posts\/746\/revisions"}],"predecessor-version":[{"id":768,"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/posts\/746\/revisions\/768"}],"wp:attachment":[{"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/media?parent=746"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/categories?post=746"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysprogs.com\/tutorials\/wp-json\/wp\/v2\/tags?post=746"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}