The exact answer depends on the debug method you are using. Generally each method provides a setting that allows selecting whether you want to program the firmware or assume it’s already programmed.
If you are not sure, let us know which debug method you are using so that we could help you locate the settings.
Currently this is a bit not straight-forward with Segger. Please edit the %LOCALAPPDATA%\VisualGDB\EmbeddedDebugPackages\com.sysprogs.arm.segger-dmsp\edp.xml file by removing the “load” command from GDBStartupCommands. We will add a configurable option to disable it in one of the next releases of the debugger package.