Importing a very large project – Help!!

[BitFlipper]

I’m currently evaluating the trial version of VisualGDB and I’m very interested to use this product. I work on a cross-platform project and this would make life much easier. BTW I’m working for a LARGE company and many of my dev coworkers could also be interested if I can get this to work.

Anyway, the way that our source code is set up is that we have a whole lot of projects within the same tree. I’m working on one of these projects and it compiles various of the other project files into itself. The problem is that the whole source code tree is large with ~200,000 files at 10GB total just for source files alone (this excludes the build folder). So when I tried to set up a VisualGDB project, it gets stuck at the “Creating project ‘MyProject’… phase right after I close the wizard. It’s been sitting there for over two hours now and I’m not sure it will be able to complete. VS memory usage is hovering around 1GB but fluctuating, so it’s doing something I guess. I’m guessing it’s trying to add each and every header and source file into the VS project?

The question is, what would be the best way to deal with this? Is there a way to tell VisualGDB to exclude some folders? This will need to happen before the wizard is closed. Should I instead configure the project manually (and hopefully have a way to exclude whole folders)?

EDIT: I should maybe add that I’m importing the project from existing makefiles. In order to make a specific product in the source code tree, one needs to specify that product name as a parameter when running make. Make succeeded when I started the build from the wizard.

Thanks,
Any help appreciated.

[support]

Hi,

What you are describing looks like a limitation of Visual Studio. Adding and indexing lots of files may take a great deal of time. You can improve this by only adding the files you want to work with.

When you are importing a project using existing makefiles, you can select “Do not import sources to Solution Explorer” on the “Source Code Location” page. VisualGDB will not add any files to Solution Explorer, but building and debugging will work nonetheless. In order to make IntelliSense work you will need to add the files you are going to edit manually and specify include paths in VisualGDB settings.

As you are using existing makefiles, there the files in Solution Explorer are handled separately from the files participating in build, as the build is fully handled by your makefile system without any interference from VisualGDB. When you start debugging, VisualGDB will get the list of files from the symbols produced by the build system using a different mechanism that should not reduce the performance.

Please let us know if you need further help.

[BitFlipper]

Agreed this is most likely a limitation in VS. Even so, it finally completed and all the files were in the project. However everything was quite sluggish so I deleted the whole project. I’m now starting from scratch with the suggestion to not import sources into Solution Explorer.

So while the source files will now not be added automatically, I assume it should not be too difficult to add sub-branches from the source code into Solution Explorer, right? It would still be useful to use Solution Explorer to browse the sections of the source files that I am interested in.

Thanks for your help!

[Nox Metus]

Probably we work in the same company. I ended up manually creating a solution only for one project I’m working on. Well, almost manually: I tracked the files compiled when I run make, then added only those files that belong to my project to the solution. I chose “makefile” type of the solution configuration and added all include directories to the NMake section of configuration for IntelliSense. Also I use plink to initiate build: it supports authentication through pageant in contrast to VisualGDB. And finally you might want to specify another location for IntelliSense database, since most likely you project is located on a network share. You can do it in Tools|Options|Text Editor|C/C++|Advanced|Fallback location.

[BitFlipper]

@Nox Metus wrote:

Probably we work in the same company. I ended up manually creating a solution only for one project I’m working on. Well, almost manually: I tracked the files compiled when I run make, then added only those files that belong to my project to the solution. I chose “makefile” type of the solution configuration and added all include directories to the NMake section of configuration for IntelliSense. Also I use plink to initiate build: it supports authentication through pageant in contrast to VisualGDB. And finally you might want to specify another location for IntelliSense database, since most likely you project is located on a network share. You can do it in Tools|Options|Text Editor|C/C++|Advanced|Fallback location.

Thanks for the info. Unfortunately the project I need to work on pulls in many files all over the place so it would be a nightmare to try and do it manually in VS. It takes over an hour to build from scratch just to give you an idea of the number of files involved.

I can now successfully build the project from VS. However I’m a bit stuck on how to set everything else up. I build the project on one machine, then deploy on another for debugging. Right now it is deploying just the main binary I specified instead of all the files in the build folder. I’m guessing instead of specifying just the one file I need to use wildcards. Trial and error is getting me closer…

But back to the issue with performance issues with large projects… It would be good to be able to do the following (maybe there is already a way to do some of there):

1. Have multiple source file sync options. For instance, even if nothing changes, it still takes a long time for VisualGDB to get past the sync stage since it is checking all the files. What would help is to have multiple sync groups. I can have an “All” sync group, in which case all files are synced after I got new sources for my whole tree. Then I can also have a “Debug” sync group that only contains files/folders that I know I will be working on. This way, when VisualGDB tries to sync the files, it will be much faster since it only checks the sub-group of files in my “Debug” list.

2. Can I start debugging without going through a build stage? Right now if I press F5, it syncs the files (see above), then initiates the remote build. However that build is really slow, even if everything is up to date (~10 minutes). So this is a real productivity killer if all I want to do is start debugging but I know the project is already up to date. I guess I can select to attach to an already running project. I’ll need to play with all the options to see what will work.

Anyway I have not gotten to the point of actual debugging but if it works anything like it says on the tin then that would be great. Hopefully the sheer size of the project won’t make things too slow during debugging. Keeping my fingers crossed…

[Nox Metus]

@BitFlipper wrote:

Unfortunately the project I need to work on pulls in many files all over the place so it would be a nightmare to try and do it manually in VS.

Write a simple Python script that parses .d files and creates the correspondent “” and “Source Files” strings for a chosen subset of your project.

[BitFlipper]

@Nox Metus wrote:

@BitFlipper wrote:

Unfortunately the project I need to work on pulls in many files all over the place so it would be a nightmare to try and do it manually in VS.

Write a simple Python script that parses .d files and creates the correspondent “” and “Source Files” strings for a chosen subset of your project.

Hmmm, interesting. I’ll look into this. Unfortunately right now I’m struggling to figure out how to copy all the output files from the build machine to the debug machine.

[Nox Metus]

Why not to run debug from a network share?

[ket]

1. Have multiple source file sync options. For instance, even if nothing changes, it still takes a long time for VisualGDB to get past the sync stage since it is checking all the files. What would help is to have multiple sync groups. I can have an “All” sync group, in which case all files are synced after I got new sources for my whole tree. Then I can also have a “Debug” sync group that only contains files/folders that I know I will be working on. This way, when VisualGDB tries to sync the files, it will be much faster since it only checks the sub-group of files in my “Debug” list.

That could be considered as a future feature. The usual solution to long file syncing times is to use shared folders to avoid syncing all together. That is why in the Linux wizard there are several options on where to store source files, VisualGDB can set up both Windows and Linux shares for you. In Tools->Shared Folder Manager you can also make and manage shared folders with VisualGDB at any time.

2. Can I start debugging without going through a build stage? Right now if I press F5, it syncs the files (see above), then initiates the remote build. However that build is really slow, even if everything is up to date (~10 minutes). So this is a real productivity killer if all I want to do is start debugging but I know the project is already up to date. I guess I can select to attach to an already running project. I’ll need to play with all the options to see what will work.

Trying to build before running happens due to a general Visual Studio setting. In Visual Studio go to Tools->Options->Projects and Solutions->Build and Run to change what to do on run. On normal Visual Studio projects Visual Studio is able to track whether a project is up to date very easily making this option very fast, but with make-based projects Visual Studio needs to poll make whether building is required and on large projects this can take a while.

[Author]

Posts