The exact answer depends on the project type you are using. You can find more information about different project types here: https://visualgdb.com/documentation/projects/overview/
Generally, ignoring the directories with the binary files and the .visualgdb directory should be sufficient (please update to VisualGDB 5.5 Preview 7, as earlier versions used several more temporary directories dependent on the project type).
If the documentation doesn’t clarify it for your project type, please let us know which one it is and we will update it.