Partition Table Errors in ESP-ADFv2.x

[Aristarchos]

Using ESP-ADF v2.x examples, specifically those that have a Custom partition table, are showing error in the partition table.

Although the partition.csv file is shown in the environment the build process cannot handle it thus it is not utilizing the partition scheme and show an error in solution explorer.

I am attaching the screenshots.

From what I see example projects of ESP-IDF that have custom partition, do work ok, also example projects of ESP-ADFv2.x that do not have custom partition also work ok. The issue is with example ESP-ADFv2.x projects that happen to have custom partition table.

So if the partition table is not used in these examples then the actual executable and flashing of memory is not proper.

 

Is this a bug or am I missing something obvious?

 

Attachments:

You must be logged in to view attached files.

[support]

Hi,

This looks like a bug in the ESP-ADF itself. Please try building the project outside VisualGDB as described here. If the problem persists, it is not specific to VisualGDB and likely requires a patch to ESP-IDF or ESP-ADF. If the problem only happens when using VisualGDB, please let us know more details and we will help you configure VisualGDB to replicate the results of the manual build.

EDIT: We have updated our ESP32 Documentation explaining the relations between ESP-ADF and ESP-IDF, and how they could cause this type or problems.

• This reply was modified 2 years, 5 months ago by support. Reason: added link to the updated documentation

[Aristarchos]

Hi,

Thanks for the reply, though, there is no “Clean and Reconfigure CMake Project” menu item in the VisualGDBv5.6 .. so the relevant guide you point might not help or it will be inconclusive since we cannot follow the same steps. You can see the attached image screenshot were the aforementioned menu item does not exist.

Regarding the reproduction of the problem, it can easily be done if someone select the “record_while_play” example to build in VisualGDB from the ESP-ADFv2.x. This is not an issue of a user-developed solution, it is in any of the ESP-ADFv2.x examples which have custom partition table. So after the build everyone can see it in the Partition Table item in project explorer.

 

As an added info, I also got ESP32 Eclipse environment running ok with the same projects eg “record_while_play” which has a custom partition table definition, where I can build and run ok on a LyraT, so if it was a plain ESP-IDF issue by itself it should had an impact there too right?

Specifically, the build without VisualGDB as shown in Eclipse log is (up to the point where the partition table gets built ok) attached as a file.

 

Attachments:

You must be logged in to view attached files.

[support]

No problem. If the project has loaded successfully, the command would be “Reload CMake project”.

The Eclipse-based environment might be using a different version of ESP-IDF, or a different toolchain, that would indeed produce different results. Please try reconfiguring and building the project via the command line exported from VisualGDB. If it works differently from the build from Eclipse, you can try comparing the 2 command lines, or the SDK versions, so see what exact part is causing the difference.

[Aristarchos]

Well, I performed the outside of VisualGDB build and it shows no fail in any stage, it concludes ok, bear in mind that also in the GUI VisualGDB it was reporting successful build too. The problem is that in the Project Explorer the Partition Table visual item shows the error that I mentioned in the first post and screenshot.

It seems to me now that I’ve seen the actual VisualGDB logs (Configure/Build) that it is a VisualGDB GUI issue, the Partition Table item in Project Explorer misbehaves when specifically a ESP-ADFv2.x custom partition table project is loaded.

Partition Table binary gets built ok but the relevant GUI item fails to read properly what its in the project (and we cannot edit that table if we want to).

Attachments:

You must be logged in to view attached files.

[support]

Please try checking whether the project built via command line actually uses the correct partition table file (i.e. whether it flashes successfully via idf.py flash). If the file works just fine (e.g. you can edit it manually and the changes are applied) and the only issue is that the “Partition table” item in Solution Explorer is not working), we should be able to fix it. If the file doesn’t work when the project is built outside VisualGDB, the issue is not on the VisualGDB side, and VisualGDB cannot fix it automatically.

[Aristarchos]

In the system that I have for VisualGDB, I have -only- VisualGDB and nothing else eg esp-idf and/or eclipse, in order to have it as clean as it can be from any conflicts.

 

So, since I do not have the ‘standard’ esp-idf how can I use the “idf.py -p COMx flash” working? is there a way to do that in a system with only VisualGDB? (please excuse me for such questions but I’m quite new in this tool)

[Aristarchos]

Tried quite a few times to see and understand what happens.

 

the current situation is this: using the record_while_play example project, VisualGDB (v5.6) with the older VisualGDB provided toolchain (gcc5.2.0 gdb7.10 r17) and the ESP-ADF v2.3, can load and configure successfully the project, then it builds it ok (according to the logs), in both the environments, command line via batch export and VisualGDB GUI.

I could not find a way to execute in command line the idf.py flash, so I used ony the GUI’s method of selecting “Program FLASH Memory” in the project properties at solution explorer.

With that method, the operation failed whereas the system reported partition overlaps, I’m attaching the error screenshot.

So, as it seems although the build succeeds in both batch and GUI VisualGDB, the produced binaries cannot be used for flashing.

 

As I mentioned earlier, I do have in another system, ‘regular’ esp-idf installed for building in command line and also in eclipse. In these environments (idf-cmdline & eclipse) the same process compiles ok, flashes eith idf.py flash and also runs ok viewing the monitor. I can provide full output logs if needed from all stages (environment details, build, flash, monitor)

By checking the various parameters in contrast, seen that in VisualGDB latest toolchain that I use is the 2021r1 whereas in the espressif’s environment (idf-installer:master build by me) the toolchain is 2021r2, do not know though if that is the reason for the different results.

For sure, as I see it, VisualGDB still has an issue with ESP-ADFv2.x specifically with custom partition table projects, it also looks that it is more than a simple GUI element change, most likely something deeper, or it could be espressif’s work moving too fast relying in newer toolschains.

Attachments:

You must be logged in to view attached files.

[support]

Hi,

idf.py is normally included in the toolchain (e.g. sysgcc\esp32\esp-idf\v4.3.1\tools\idf.py and sysgcc\esp32\tools\idf-exe\1.0.1\idf.py.exe). The easiest way to run it would be to dump the project configuration command to a batch file and replace the actual “cmake” command with cmd.exe. This will run the command prompt window with the same environment that VisualGDB would use for cmake (that should be sufficient for most tools). You can then try running idf.py from that window.

Based on the description you provided, it looks like a bug in ESP-ADF that might be triggered by a specific toolchain. If you can confirm that it works just fine in the Eclipse environment (i.e. the generated file can be FLASHed successfully), please try downloading the same toolchain version from our download page and installing ESP-ADF there.

You can also simply replace the contents of the esp-adf directory inside the toolchain with the version that works from Eclipse. If it doesn’t help, you may need to also replace the tools in the tools subdirectory, and update the paths to them inside toolchain.xml.

[Aristarchos]

Ok, thanks for the explanation on the cmdline for idf.py in VisualGDB.

And yes, I can confirm that the other standard esp-idf/adf and eclipse work just fine all the way from build, flash up to monitor checking.

 

Regarding toolchains, I cannot find the same toolchain in your downloads, the latest one that resides in your toolchain downloads gives this regarding its version

C:\SysGCC\esp32\tools\xtensa-esp32-elf\esp-2021r1-8.4.0\xtensa-esp32-elf\bin>xtensa-esp32-elf-cc –version
xtensa-esp32-elf-cc (crosstool-NG esp-2021r1) 8.4.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

 

Whereas the one that is used with ‘standard’ espressif’s idf-install is this

C:\esp\.espressif\tools\xtensa-esp32-elf\esp-2021r2-8.4.0\xtensa-esp32-elf\bin>xtensa-esp32-elf-gcc.exe –version
xtensa-esp32-elf-gcc.exe (crosstool-NG esp-2021r2) 8.4.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

 

Whereas the difference is 2021r1 vs 2021r2

From what I see, there is no such toolchain (esp-2021r1-8.4.0) in your toolchain downloads, do I miss something?

 

[support]

Hi,

The ESP32 toolchains are provided by Espressif. We merely pick up the stable releases, bundle it together with the latest stable ESP-IDF and run a few basic tests. If you would like to know the difference between 2021r1 and 2021r2, please contact Espressif.

You can download an equivalent of the 2021r1 toolchain from Espressif here: esp32-gcc8.4.0-r3.exe

[Aristarchos]

Ok, thank you for explaining but I kinda knew that.

Anyway, I do not want to know what the differences are between toolchain versions, just want to have environment that works with the things I’m looking at now.

These are the latest ESP-ADF with custom partition tables and the ability to use the IDF Component Manager. For both it looks that I need to have 2021r2 toolchain which from what I understand it is not available yet from VisualGDB, I hope in due time will be.

[support]

Based on our experience with ESP32, not all combinations of toolchains and SDKs work together well. Espressif frequently releases updates to the toolchains, ESP-IDF and ESP-ADF, where we pick them up, run a few basic tests and release ready-to-use packages.

Sometimes, these releases contain bugs that are addressed later (e.g. the latest ESP-IDF 4.3.1 breaks projects using mbedtls, and the workaround is to cherry-pick a specific commit from another branch). As Espressif typically fixes the issues relatively fast, we leave it up to them and stick to shipping ready-to-use snapshots of their most recent public releases.

We have designed VisualGDB to be very flexible, so that it can be used with any combination of the tools that works otherwise. If you have a specific version of ESP-ADF and toolchain that work together, you can merge them into our toolchain by replacing the contents of the tools and esp-adf directories, and updating the paths in toolchain.xml. VisualGDB will pick up these paths and will produce exactly the same results as your Eclipse-based environment does, because it will run exactly the same tools with exactly the same arguments. This might require some minor troubleshooting, but since VisualGDB allows dumping the exact command line it uses for building, you can very easily troubleshoot issues by comparing it against another command line that works just fine.

[Aristarchos]

After some checking, I think I found the cause for the partition table visual error and why the idf.py reports a overlap error when flashing.

There are two problems.

At first the contents of the example partition.csv file need to change a bit as they do not have the commas at the end of two lines, so I guess the VisualGDB GUI partition element cannot load and view it correctly

The original partitions.csv file had these contents

# Name, Type, SubType, Offset, Size
nvs, data, nvs, 0x9000, 0x6000
phy_init, data, phy, 0xf000, 0x1000
factory, app, factory, 0x10000, 3M,


Whereas it should have these, the comas at the end of lines

# Name, Type, SubType, Offset, Size
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 3M,


After that change the GUI Partition element is working ok.

It looks though that the build process do not care about the commas and it succeeds at building the partition.bin file.

Now, the other problem, where the idf.py flash is reporting an overlap error, as I see it, its not due t the partition.bin, but rather on the bootloader.bin file.

From what I know, the bootloader should be flashed at 0x1000, the partition table at 0x8000 and the application at 0x10000.

The bootloader.bin has to be loaded at address 0x1000 and then the partition.bin to be loaded at 0x8000, that means there is a space of 28672 bytes for the bootloader to be flashed.

Well, the produced bootloader.bin file (of a esp-adf example with custom partition table) exceeds that size limit and it is 29552 bytes, so eventually it does not fit in.

In later toochains (2021r1, 2021r2), there is a flag “CONFIG_BOOTLOADER_COMPILER_OPTIMIZATION_SIZE=y” so the bootloader.bin remains small in size even to a esp-adf example with custom partition tables.

I tried to apply manually this flag adding it to the sdkconfig of the ‘record_while_play’ esp-adf example that has the custom partition table, but to no avail, it got rejected and removed from the sdkconfig file when build the project (with the only old toolchain GCC520GDB710r17 that is known so far that works).

So to recap, the visual aspect of partition table error can be get fixed by strictly enforcing csv rules to the partition.csv file by adding the missing commas.

But I do not know how the excessive size of the bootloader.bin can be reduced with the old toolchain GCC520GDB710r17.

 

[support]

Hi,

No problem. The issue with the partition editor GUI is within our control, so we have updated VisualGDB to support the syntax used by ESP-ADF. Please try this build: VisualGDB-5.6.101.4473.msi

The bootloader issue is somewhat trickier. When you install ESP-ADF via VisualGDB, it simply clones the corresponding tag from the https://github.com/espressif/esp-adf repository, that references ESP-IDF is a submodule. Due to the reasons unknown to us, even the latest ESP-ADF releases reference the old ESP-IDF 3.3.2, although we have heard of other users being able to build it with newer ESP-IDF versions.

You can try checking out other ESP-IDF versions for use by ESP-ADF by manually going to sysgcc\esp32\esp-adf\v2.3\esp-idf and running “git tag” followed by “git checkout <tag name>“. That said, this is something to do entirely at your own risk. Each IDF/ADF combination may work differently, or may not work at all. Also, the workarounds you may discover for ESP-ADF v2.3 may no longer work for the next ESP-ADF release. Finding out the versions used in your Eclipse-based environment (via “git branch”) and checking out the same versions into our toolchain could be a good starting point.

[Author]

Posts