If it is an external project managed by an external Makefile, the linker is likely re-invoked because the Makefile doesn’t handle dependencies properly.
You could try re-creating the project as an MSBuild-based project to let VisualGDB handle dependencies, although it might be non-trivial if the project has complex build settings.
If you are using the VisualGDB MSBuild backend, you can see why the linker is invoked by enabling verbose output:
Tools->Options->Projects and Solutions->Build and Run->MSBuild build output verbosity -> Detailed.
Then build the project and check for the “Re-linking <…> due to <…> message”. If you are not sure, simply attach the detailed build log here and we can help you understand it.