Hi,
No problem. Although this is not officially supported (i.e. may not work in some circumstances/may require inconvenient custom setup), you should be able to achieve this.
In order to use VisualGDB without VisualStudio (MSBuild only), please copy its directory (from Program Files (x86)\Sysprogs) to an arbitrary location on the server, then run VisualGDB.exe /about to register it. Then set %VISUALGDB_DIR% to point to the VisualGDB location and register the “VisualGDB” platform with MSBuild by copying the contents of the <MSBuild Platforms>\VisualGDB folder from a machine with Visual Studio.
As long as MSBuild can find the VisualGDB platform and the platform files can find VisualGDB itself (via %VISUALGDB_DIR), you should be able to build your projects even if VS is not installed.
If you encounter any difficulties, please let us know and we will help you get it to work.