{"id":9551,"date":"2016-11-22T23:12:16","date_gmt":"2016-11-22T22:12:16","guid":{"rendered":"https:\/\/sysprogs.com\/w\/?p=9551"},"modified":"2016-11-22T23:38:31","modified_gmt":"2016-11-22T22:38:31","slug":"limitations-of-the-esp32-debugging","status":"publish","type":"post","link":"https:\/\/sysprogs.com\/w\/limitations-of-the-esp32-debugging\/","title":{"rendered":"Limitations of the ESP32 debugging"},"content":{"rendered":"<p>We have just added support for the new ESP32 chip to VisualGDB. While VisualGDB allows building projects, setting breakpoints and debugging your ESP32\u00a0code out-of-the-box, the ESP32 tools come with a few known limitations that may interfere with debugging.\u00a0In this post I will summarize them and show known workarounds.<\/p>\n<p><!--more--><\/p>\n<h2>Reset behavior<\/h2>\n<p>The biggest problem of the ESP32 chip is that it cannot be reliably reset into a predictable\u00a0initial state. Programming the SPI FLASH requires putting the chip in a state where the background tasks are stopped, FLASH is initialized properly and the interrupts are disabled.\u00a0Unlike ARM\u00a0devices that enter this state immediately after reset, ESP32 does not allow\u00a0doing it easily.\u00a0Resetting just the CPU leaves the peripherals in an unpredictable state that often results in a crash and resetting the entire chip immediately starts the CPU, so by the time VisualGDB can stop it, it has already begun\u00a0running some code.<\/p>\n<p>VisualGDB works around this by\u00a0doing a chip reset and halting both CPUs some time after it, but this\u00a0does not work 100% reliably.<\/p>\n<p>If you encounter a\u00a0situation where your FLASH memory cannot be programmed, try the following steps to troubleshoot it:<\/p>\n<ol>\n<li>Put the chip in the bootloader mode (hold the bootloader button, press the reset button).<\/li>\n<li>Start JTAG debugging. The memory should get programmed, but the\u00a0chip will not leave the bootloader mode without\u00a0a physical reset.<\/li>\n<li>Reset the chip by pressing the Reset button.<\/li>\n<li>Select &#8220;do not program FLASH&#8221; in VisualGDB Project Properties and start JTAG debugging again<\/li>\n<\/ol>\n<h2>Breakpoints in FLASH<\/h2>\n<p>The ESP32 chip has only 2 hardware breakpoints. This means that you cannot set more than 2 simultaneous breakpoints in the code that\u00a0resides in the SPI FLASH.\u00a0To work around this you can move some of your functions from FLASH to RAM:<\/p>\n<pre class=\"\">void __attribute__((section(\".iram1\"))) ramfunc()\r\n{\r\n    \/\/...\r\n}<\/pre>\n<p>VisualGDB will automatically detect the code that is placed in RAM and will use software breakpoints instead of hardware ones for it.<\/p>\n<h2>Breakpoints in RAM<\/h2>\n<p>Unlike ESP8266 where software breakpoints in RAM just work, the ESP32 makes things a bit trickier due to\u00a0having 2 cores. \u00a0They will work flawlessly if they are triggered on the same core\u00a0that was active when setting them. If a different core triggers a\u00a0software breakpoint, OpenOCD will not handle it correctly will display it as\u00a0a random stop in a random place:<a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/multicore.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-9553\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/multicore.png\" alt=\"multicore\" width=\"1022\" height=\"645\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/multicore.png 1022w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/multicore-300x189.png 300w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/multicore-768x485.png 768w\" sizes=\"(max-width: 1022px) 100vw, 1022px\" \/><\/a><\/p>\n<p>Although this is\u00a0annoying, it does not mean the end of the world. You can switch to the other core via the Threads window in Visual Studio and see the code that triggered the breakpoint:<a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/core2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-9554\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/core2.png\" alt=\"core2\" width=\"1021\" height=\"460\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/core2.png 1021w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/core2-300x135.png 300w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/core2-768x346.png 768w\" sizes=\"(max-width: 1021px) 100vw, 1021px\" \/><\/a><\/p>\n<p>Viewing local variables and call stack will work, but trying to continue\u00a0debugging will result in a random effect from stopping at\u00a0the same place in code to crashing gdb. The workaround would be to disable the breakpoint, optionally set another on on the next line, resume execution with F5 and reenable the original breakpoint once the next breakpoint is hit.<\/p>\n<p>We have experimented with modifying OpenOCD to report the correct core in the &#8216;session stopped&#8217; event, but this often results in crashing the firmware, so as of November 2016,\u00a0the core that triggered the breakpoint needs to be selected manually.<\/p>\n<h2>Stepping into functions does not work<\/h2>\n<p>This is another unfortunate limitation of the ESP32 GDB. Trying to step into and over function calls often\u00a0ends up with just resuming the execution. The workarounds that work are:<\/p>\n<ul>\n<li>Set a breakpoint on the next line and press F5 instead of trying step over (beware of the breakpoint count limit)<\/li>\n<li>Switch to the Disassembly View and step instruction-by-instruction if you want to step into a function<\/li>\n<\/ul>\n<h2>The firmware sometimes crashes randomly<\/h2>\n<p>VisualGDB does a lot of background work to keep debugging as usable as possible. Stopping watchdog timers, disabling interrupts during\u00a0stepping,\u00a0distinguishing RAM and FLASH functions&#8230; Despite that, debugging sometimes does interfere with\u00a0some ESP32 internals, causing the firmware to throw one exception after another.\u00a0The fastest shortcut out of this state is to use the &#8220;Reset Device&#8221; button in the GDB Session window:<a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/reset.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-9555\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/reset.png\" alt=\"reset\" width=\"562\" height=\"304\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/reset.png 562w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/reset-300x162.png 300w\" sizes=\"(max-width: 562px) 100vw, 562px\" \/><\/a><\/p>\n<h2>Some CPU registers are shown incorrectly<\/h2>\n<p>Yet another bug of the ESP32 gdb\/OpenOCD is incorrect reporting of\u00a0some of the CPU registers. If you see invalid values shown in Visual Studio, use the &#8220;mon reg &lt;register name&gt;&#8221; command to query the correct one directly from OpenOCD:<a href=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/regs.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-9556\" src=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/regs.png\" alt=\"regs\" width=\"563\" height=\"305\" srcset=\"https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/regs.png 563w, https:\/\/sysprogs.com\/w\/wp-content\/uploads\/2016\/11\/regs-300x163.png 300w\" sizes=\"(max-width: 563px) 100vw, 563px\" \/><\/a><\/p>\n<h2>Stepping through code sometimes crashes gdb<\/h2>\n<p>Stepping through code with inline\u00a0functions may\u00a0crash gdb with the following message:<\/p>\n<pre class=\"\">..\/..\/..\/..\/src\/gdb-7.10\/gdb\/inline-frame.c:167: internal-error: inline_frame_this_id: Assertion `frame_id_p (*this_id)' failed.<\/pre>\n<p>The suggested workaround is to disable function\u00a0inlining for the source files you are debugging.<\/p>\n<h2>Next steps<\/h2>\n<p>Despite the few shortcomings shown here, Espressif has done a great job open-sourcing most of the ESP32 libraries in the ESP-IDF framework and providing an OpenOCD version that can\u00a0provide basic\u00a0debugging functionality. \u00a0The version shipped with VisualGDB contains a few stability improvements over the original one and can be found in our <a href=\"https:\/\/github.com\/sysprogs\/openocd-esp32\">Github repository<\/a>. We will be\u00a0contacting Espressif regarding the issues\u00a0in the ESP32 tools and will post updates here once better\u00a0workarounds are available.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We have just added support for the new ESP32 chip to VisualGDB. While VisualGDB allows building projects, setting breakpoints and debugging your ESP32\u00a0code out-of-the-box, the ESP32 tools come with a few known limitations that may interfere with debugging.\u00a0In this post I will summarize them and show known workarounds.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/9551"}],"collection":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/comments?post=9551"}],"version-history":[{"count":6,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/9551\/revisions"}],"predecessor-version":[{"id":9561,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/9551\/revisions\/9561"}],"wp:attachment":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/media?parent=9551"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/categories?post=9551"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/tags?post=9551"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}