Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build using multiple cores #40

Open
GoogleCodeExporter opened this issue Mar 15, 2015 · 15 comments
Open

Build using multiple cores #40

GoogleCodeExporter opened this issue Mar 15, 2015 · 15 comments

Comments

@GoogleCodeExporter
Copy link

With Visual Studio c++ compiler rebuilds my project (ARM,WM) in 1 minute. With 
vs-android I have near 8 minutes to build such project.
Looks like it's because native VS compiler utilizes all 4 cores of my CPU, and 
vs-android uses only one.

Feature Request: add support for multiple cores.

Original issue reported on code.google.com by izx...@gmail.com on 31 Oct 2011 at 6:57

@GoogleCodeExporter
Copy link
Author

Original comment by gavin.dj.pugh on 31 Oct 2011 at 4:43

  • Changed state: Accepted
  • Added labels: Type-Enhancement, Priority-Low
  • Removed labels: Type-Defect, Priority-Medium

@GoogleCodeExporter
Copy link
Author

This improvement would be really welcome - I'm trying to use vs-android with a 
source code base of over 1000 c++ files. Do you have any pointers for how to go 
about it - I might be able to find some time to look into making the required 
changes. From my limited knowledge of MSBuild it parallelises the building of 
independent projects but not of individual build tasks within a project. So 
presumably the collection of source files to build is being treated as a single 
project and each individual build task is happening in sequence. Since VS 
itself manages to parallelise compilation there must be some way around this, 
perhaps treating each build task as its own MSBuild project?

Original comment by mikeheal...@gmail.com on 14 Feb 2012 at 1:02

@GoogleCodeExporter
Copy link
Author

I believe Microsoft's CL compiler handles the multithreaded builds within the 
C# compilation task.

vs-android has it's own compilation task written in C#, where this could 
potentially be implemented. It'd need to create a thread for each instance of 
the g++ compiler.

The biggest gotcha though is with the MSBuild scripting. For vs-android I sent 
each compilation unit one-by-one to the C# task. For Microsoft CL, they send 
them as a big batch, such that the C# task can farm them out to multiple 
threads.

So it'd require pretty fundamental changes to the MSBuild script. The part 
where it gets trickier is where you can potentially override individual .c/cpp 
files with specific compiler options. Dealing with that in the mix, makes it a 
difficult problem to solve.

Unity builds might be a short-term solution (hack) to your compile times:
http://cheind.wordpress.com/2009/12/10/reducing-compilation-time-unity-builds/

I've used them extensively in the past for big game projects, but they do have 
their downsides.

Original comment by gavin.dj.pugh on 14 Feb 2012 at 5:48

@GoogleCodeExporter
Copy link
Author

Issue 67 has been merged into this issue.

Original comment by gavin.dj.pugh on 17 Feb 2012 at 10:42

@GoogleCodeExporter
Copy link
Author

Here's my hacky first attempt at enabling this. I changed the foreach in  
CompileWithGCC to a parallel task, and fixed up the non-thread-safe bits (at 
least, the ones that I spotted). It appears to do the job, maxing out all my 
cores, but unfortunately it's screwed up dependency checking in some cases 
(specifically, if a files fails to compile, it won't attempt to recompile that 
file next time around). There's also a minor issue with spurious newlines being 
inserted into the output.

Does this at least look like the right direction to take? I'll continue to look 
at the dependency issue myself.

Original comment by kim.rand...@gmail.com on 29 Feb 2012 at 12:33

Attachments:

@GoogleCodeExporter
Copy link
Author

By the way, I've also made a small modification to install.cmd which kills any 
MSBuild processes if the plugin is already installed, as otherwise the DLL 
cannot be over-written.

Original comment by kim.rand...@gmail.com on 29 Feb 2012 at 12:36

Attachments:

@GoogleCodeExporter
Copy link
Author

Fixed the spurious newline issue by changing line 244 to:

string[] lines = output.Split(new Char[] { '\n', '\r' }, 
StringSplitOptions.RemoveEmptyEntries);

Original comment by kim.rand...@gmail.com on 29 Feb 2012 at 12:54

@GoogleCodeExporter
Copy link
Author

Apologies for the continual spamming, but I want to keep this up-to-date...

I think that the dependencies issue is present in the release version. You can 
test it quite easily thus:

1) Build the san-angeles sample
2) Add #error to the top of importgl.h
3) Build again. You should see something like the following output:

1>  demo.c
1>  C:\work\vs-android_samples\san-angeles\CppSource\demo.c(30): includes this 
header: 
1>C:\work\vs-android_samples\san-angeles\CppSource\importgl.h(28,2): error : 
#error
1>  File C:\WORK\VS-ANDROID_SAMPLES\SAN-ANGELES\CPPSOURCE\IMPORTGL.C is missing 
it's .d file: C:\work\vs-android_samples\san-angeles\Android\Debug\importgl.d

4) Build again. The dependency check will be incorrect and you will just get 
this:

1>------ Build started: Project: san-angeles, Configuration: Debug Android 
------
========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

I'm going to see if I can fix this myself, although I am a complete C# and 
MSBuild noob, so I'm not very confident. I *think* it's just a case of 
correcting the tlog files when the .d is missing or empty.

Original comment by kim.rand...@gmail.com on 29 Feb 2012 at 3:00

@GoogleCodeExporter
Copy link
Author

Okay, I think I've fixed the dependency issues. It now tracks the result of 
each compilation task, and only adds them to the tlog if they have succeeded.

I've also fixed up the parallelisation to use the correct way of breaking out 
of a parallel foreach, and locked the output to avoid interleaving of error 
messages.

I suppose we ought to make the parallelisation optional, but aside from that I 
think this satisfies all the requirements.

Original comment by kim.rand...@gmail.com on 2 Mar 2012 at 1:02

Attachments:

@GoogleCodeExporter
Copy link
Author

Bah, looks like my implementation hangs if there are too many error lines. A 
quick fix is to do p.StandardError.ReadToEnd() before 
p.StandardOutput.ReadToEnd(), since it's much more likely that it'll fill up 
stderr first. However, I probably ought to be doing:
http://stackoverflow.com/a/7608823

Original comment by kim.rand...@gmail.com on 2 Mar 2012 at 2:55

@GoogleCodeExporter
Copy link
Author

Okay, here's a version with the stderr/stdout overflow handling fixed.

Original comment by kim.rand...@gmail.com on 2 Mar 2012 at 3:30

Attachments:

@GoogleCodeExporter
Copy link
Author

Thanks for looking into this, sounds very promising!

Yeah, when I integrate this (eventually, it'll be at least a couple of weeks - 
it's GDC time), I'll put the parallelisation on a BoolProperty.

Thanks again!

Original comment by gavin.dj.pugh on 2 Mar 2012 at 6:24

@GoogleCodeExporter
Copy link
Author

I am not sure if somebody else tested this updated GCCCompile.cs code, but just 
for the record, I just tested and worked perfectly!

Thanks a lot!

Original comment by jplpinka...@gmail.com on 22 May 2012 at 10:44

@GoogleCodeExporter
Copy link
Author

Sorry, had to shelve adding this one for now. I'll put it in the next release.

Original comment by gavin.dj.pugh on 25 Jul 2012 at 1:56

  • Changed state: Started

@GoogleCodeExporter
Copy link
Author

Original comment by gavin.dj.pugh on 25 Jul 2012 at 1:56

  • Changed state: Accepted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant