Normally both options should work the same way. Please try double-checking whether the LEDs start blinking after you power cycle the board after it has been programmed.
It still doesn’t work after power cycling the board
And if I program the board with the “Program and start without debugging” option and then attach to it using the “Attach to running embedded firmware” (even after a power cycle) I can see in the stack trace that it received some signal handler that sends the board to the hard fault handler, that’s showed in the attached picture
This looks more like a broken board than anything else. You can try erasing the FLASH memory and programming the image using the Nordic JTAG tools to see if there is any difference.
If it still doesn’t work, please try modifying one of the Nordic samples from their SDK to work with your board, and then build it using the original Make scripts and program it outside VisualGDB. If that works and an equivalent project built with VisualGDB doesn’t, please try comparing their memory layout via Memory Explorer. Most likely, some fine setting, like memory size or some peripheral configuration, is off.