Hi,
VisualGDB uses randomized ports for J-Link and OpenOCD in order to support multiple debugging sessions at once. Normally it should choose a port randomly and configure both gdb stub and gdb to use it. Perhaps your firewall is interfering with it? If this is the case, please try adding JLinkGDBServerCL.exe to the list of exceptions.
If not, please check the command line used by VisualGDB to run segger gdb server (shown in the corresponding window in VS) and the ‘target’ command in the gdb session window. If the port numbers there don’t match, please let us know. If they do, please try running the same commands manually and see if it works and if there could be any other components (e.g. antivirus) blocking the connection.