The Arduino-based projects are different from the regular ESP-IDF projects. They rely on the Arduino framework (using the rules in the ESP32 Arduino core) to build and program the project. Programming FLASH for Arduino projects is done via the COM port (same when using the Arduino IDE).
If the regular ESP-IDF projects work, you can use VisualGDB to develop them via the ESP-IDF project wizard (not the Arduino project wizard).
If you are getting different behavior between the regular ESP-IDF projects and Arduino projects, it is likely caused by differences between the ESP-IDF framework and the Arduino framework (that internally uses a different version of ESP-IDF and a completely different set of rules to build the code). Using VisualGDB on top of either ESP-IDF or Arduino will not change the way the underlying framework operates and will not eliminate the differences between these 2 frameworks.