Hi,
The Source Cache Manager is used to cache the remote source/header files (e.g. in /usr/src) that are not a part of the project but somehow participated in build, hence they are only transferred one way.
The sources from the project directory should be normally auto-transferred on each build (see your VisualGDB Project Properties for file transfer settings).
Sometimes when the target directories are symlinked, gdb reports the path that is different from the one used by VisualGDB and VisualGDB assumes those files are out-of-project files.
Can you identify one particular file that does not get transferred, check its local path, Linux path reported by GDB (see the GDB Session window) and check if your settings are configured to upload the contents of that directory?