ESP-IDF strange CMake problem

[dedvalson]

Hi,

I recently converted my project to CMake. Wow, it is amazing how much faster it is. But I have run into a problem that I have spent a couple days on now with no success. I finally decided to create brand new projects to make sure it was really an issue with my project. It turns out that it is not.

To test, I created two brand new projects using Visual Studio and the VisualGDB wizard. The two projects are identical except that one is GNU make and the other is cmake. They are both based on the simple “hello world” sample. With no code modifications at all I brought in only one file, my sdkconfig file. So I built the hello world once under GNU and the other under cmake with brand new projects and my sdkconfig.

The GNU based project works great. The CMake based project crashes before it reaches my code, but only if there are any JTAG breakpoints enabled. This is the same behavior I saw on my project but I was surprised that I could see it on Hello World.  Here is the stack trace from when it crashes:

rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:6620
load:0x40078000,len:13952
load:0x40080400,len:4040
entry 0x40080680
I (68) boot: Chip Revision: 1
I (69) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (38) boot: ESP-IDF v4.0-beta2-174-g99fb9a3f7 2nd stage bootloader
I (39) boot: compile time 17:43:06
I (41) boot: Enabling RNG early entropy source...
I (45) boot: SPI Speed : 40MHz
I (49) boot: SPI Mode : DIO
I (53) boot: SPI Flash Size : 4MB
I (57) boot: Partition Table:
I (61) boot: ## Label Usage Type ST Offset Length
I (68) boot: 0 nvs WiFi data 01 02 00009000 00004000
I (75) boot: 1 otadata OTA data 01 00 0000d000 00002000
I (83) boot: 2 phy_init RF data 01 01 0000f000 00001000
I (90) boot: 3 fat_fs Unknown data 01 81 00010000 00030000
I (98) boot: 4 ota_0 OTA app 00 10 00040000 001e0000
I (105) boot: 5 ota_1 OTA app 00 11 00220000 001e0000
I (113) boot: End of partition table
I (117) boot: No factory image, trying OTA 0
I (122) boot_comm: chip revision: 1, min. application chip revision: 0
I (129) esp_image: segment 0: paddr=0x00040020 vaddr=0x3f400020 size=0x0835c ( 33628) map
I (152) esp_image: segment 1: paddr=0x00048384 vaddr=0x3ffbdb60 size=0x02100 ( 8448) load
I (156) esp_image: segment 2: paddr=0x0004a48c vaddr=0x40080000 size=0x00400 ( 1024) load
I (159) esp_image: segment 3: paddr=0x0004a894 vaddr=0x40080400 size=0x0577c ( 22396) load
I (178) esp_image: segment 4: paddr=0x00050018 vaddr=0x400d0018 size=0x17480 ( 95360) map
I (217) esp_image: segment 5: paddr=0x000674a0 vaddr=0x40085b7c size=0x06cf4 ( 27892) load
I (238) boot: Loaded app from partition at offset 0x40000
I (273) boot: Set actual ota_seq=1 in otadata[0]
I (273) boot: Disabling RNG early entropy source...
I (273) psram: This chip is ESP32-D0WD
I (277) spiram: Found 64MBit SPI RAM device
I (282) spiram: SPI RAM mode: flash 40m sram 40m
I (287) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
I (294) cpu_start: Pro cpu up.
I (298) cpu_start: Application information:
I (303) cpu_start: Project name: hello-world
I (308) cpu_start: App version: 1
I (312) cpu_start: Compile time: Feb 2 2020 17:44:04
I (318) cpu_start: ELF file SHA256: 1b48fee3aeae9987...
I (324) cpu_start: ESP-IDF: v4.0-beta2-174-g99fb9a3f7
I (331) cpu_start: Starting app cpu, entry point is 0x4008132c
I (318) cpu_start: App cpu up.
I (1219) spiram: SPI SRAM memory test OK
I (1219) heap_init: Initializing. RAM available for dynamic allocation:
I (1220) heap_init: At 3FFAE6E0 len 0000F480 (61 KiB): DRAM
I (1227) heap_init: At 3FFC0D48 len 0001F2B8 (124 KiB): DRAM
I (1232) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (1239) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1248) heap_init: At 4008C870 len 00013790 (77 KiB): IRAM
I (1251) cpu_start: Pro cpu start user code
I (1256) spiram: Adding pool of 4096K of external SPI memory to heap allocator
I (1841) spi_flash: detected chip: generic
I (1842) spi_flash: flash io: dio
I (1842) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (1850) spiram: Reserving pool of 32K of internal memory for DMA/internal allocations
Guru Meditation Error: Core 0 panic'ed (InstrFetchProhibited) at pc=ffffffff. Setting bp and returning..


Since there is no stack trace when debugging, I am unable to provide that, it doesn’t fail unless a debugger is attached. However, I have been able to determine that crash occurs at line 368 of heap_caps.c as shown here:

SLIST_FOREACH(heap, &registered_heaps, next) {

I noticed one interesting thing. The file sizes vary quite a bit between the two projects. In GNU Make, the elf file is 2.4 MB and the .bin file is 147K. In CMake the elf is 3.4 MB and the .bin file is 185K. So something is a lot different in the configuration but I am trying to figure out how to determine where the differences are. These are both debug builds.

I am using ESP32 toolchain 8.2.0/8.1.0/r2 and ESP-IDF release 4.0. However the same thing happens with 5.2.0/7.10/R16 and ESP-IDF release 3.2. My VisualGDB is version 5.5(Preview 3, build 3458).

The problem is easily repeatable as follows:

1. Create a new advanced ESP-IDF project based on “Hello World” and a JTAG debugger.
2. Copy the attached sdkconfig file into it.
3. Build it.
4. Insert 2 breakpoints anywhere in the hello_world_main.c file.
5. Run it under the VisualGDB debugger within Visual Studio.

I am sure there are things I could try to produce better diagnostics or figure out what is going on but I thought I would check in with you for suggestions.

I zipped the whole project (after cleaning it) and attached it here.

Thanks,

Don

Attachments:

You must be logged in to view attached files.

[support]

Hi,

Please try reproducing the problem with the command-line gdb and OpenOCD tools. If the problem persists, please contact Espressif for help.

If the problem only happens when using VisualGDB, we can help you compare the setups and replicate the results of the command-line session with VisualGDB.

[dedvalson]

Hi,

I was able to track this down to a configuration problem after changing to CMake. Previously I had the number of breakpoints limited to 2 (which is how many hardware breakpoints the ESP-32 can support). I failed to transfer that setting and it appears that the VisualGDB code was inserting more than 2 breakpoints even if I only  had one user specified breakpoint.  Flash breakpoints are not currently supported on the ESP-32. Putting that limitation back in solved the problem. The ESP-IDF code still shouldn’t crash under these circumstances but that does seem to be the issue.

 

Thanks,

Don

 

[support]

Good to know it works. In our tests, hardware breakpoints do work when using the latest OpenOCD (2019-10-24), however each subsequent breakpoint reduces the overall debugging performance and may indeed cause weird errors for large projects. You might be able to work around of some issues by adding the “set breakpoint always-inserted 1” command to the gdb startup commands to reduce the amount of times gdb sets the breakpoints.

Also if you can reliably reproduce ESP-IDF crashing when using software breakpoints outside VisualGDB, feel free to report it to Espressif. They might be able to fix it eventually.

[Author]

Posts