New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Doc generation step in build far too slow #5145
Comments
This comment was originally written by amouravski@google.com From my brief experimentation, the actual doc extraction/generation is fairly quick, but the initial dart2js compilation is very slow. I would like it to be faster, but not sure what to do about it. Removed Area-SDK label. |
This comment was originally written by amouravski@google.com Err, issue #4527 |
I have found a quadratic solution to a linear problem in the mirror system in the number of classes documented. I'll try that out when I have figured out what is wrong on the pub builders. |
Investigating a bug in the dart2js source mirrors, I discovered that dartdoc is analyzing the same libraries twice: Parsing MDN data... That combined with a quadratic nature of docInheritance in sdk/lib/_internal/dartdoc/lib/dartdoc.dart makes me doubt that "actual doc extraction/generation is fairly quick". So I'm sending this back to dartdoc. However, it is possible that some of the problems lie in the dart2js source mirrors. I suggest that Johnni and Andrei take a look at this together. I'll be happy to provide additional details if needed, but right now I'm investigating a higher priority bug, so forgive my terseness. cc @johnniwinther. |
This comment was originally written by amouravski@google.com I have a few ideas about this and have started a few things. dgrove: Could you give me an idea of how quick you're looking to make apidoc? ~30s? ~10? ~2? Set owner to amouravski@google.com. |
My target is < 5s. |
This comment was originally written by amouravski@google.com Okay, I'll try to hook up some metrics and for the time being I'm going to call the goal 7.5x improvement (based on numbers above) so that I can measure locally/on bots. |
Just to be clear, that's 5s on a fast development box (the bots will likely take longer, but I'm not worried about them) |
Some context in case it helps: I initially added apidoc to the build as a crude smoke test. At the time, dartdoc had a few unit tests, but nothing to validate that the end-to-end generation process worked. Because of the lack of tests, I was experiencing daily breakages where some other part of the Dart platform would change and apidoc would die. Adding apidoc to the build was a simple change to try to mitigate that. If dartdoc's testing situation is in a better place, or the platform is more stable (I think at the time it was mostly frog (!) changes that were breaking it, so this is a pretty old decision), one option is to just stop building apidoc automatically. It doesn't serve much of a useful purpose: the generated docs are not part of the SDK, so it's output isn't used. Of course, making apidoc faster is always a good thing, too. But if the goal here is to speed up the build iteration loop for the Dart team, it's hard to beat 0s by removing apidoc completely. |
Added this to the M7 milestone. |
This comment was originally written by amouravski@google.com Added Triaged label. |
This comment was originally written by amouravski@google.com Removed the owner. |
Removed this from the M7 milestone. |
Added old-dartdoc label. |
I don't see us being able to do much about this. The limiting factor is that it is essentially doing a dart2js compile on the entire contents of the repository. Probably the biggest factor is that I think dart2js is doing more work than it needs to, essentially doing most of a compile in order to be able to expose a mirror system. It could, for example, skip any consideration of optimizations. Removed Priority-High, old-dartdoc labels. |
If dart2js is compiling too much, it is because you're telling it to. It supports these options: --analyze-only --analyze-signatures-only But please don't hesitate to let Johnni and I know If you think you have an issue with dart2js doing too much work. Any ideas are appreciated! If I create a source file named all.dart with this content: import "dart:async"; And I run dart2js like this: dart2js --analyze-all --analyze-only --analyze-signatures-only --categories=all all.dart The compiler completes in less than 4 seconds. So 40 seconds isn't acceptable. Removed Priority-Medium label. |
Interesting. We are already passing those options in, and it does look like I was wrong in attributing too much time to dart2js. It seems to be about 8 seconds out of 30. Trivial playing around with the profiler suggests that the actual culprit is markdown parsing of comments. Removed Priority-High label. |
I took the measurement on my 2 GHz Intel Core i7 MacBook Pro. I'm guessing you measured on something equivalent. So it sounds as if creating the mirror system is already twice as slow as it should be. Are you running in checked mode, or are you analyzing a lot of packages as well? |
Here's some data from running this morning on my Mac desktop (in release mode):
Thu Sep 13 08:30:23 PDT 2012 Parsing MDN data...
Thu Sep 13 08:30:24 PDT 2012 Cross-referencing dart:html...
Thu Sep 13 08:30:27 PDT 2012 Processing handwritten HTML documentation...
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/.svn/.svn.dart
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/args/args.dart
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/htmlescape/htmlescape.dart
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/logging/logging.dart
Thu Sep 13 08:30:32 PDT 2012 Generating docs...
Thu Sep 13 08:30:36 PDT 2012 Skipping MDN type UnknownElement
real 0m38.220s
user 0m37.589s
sys 0m0.571s
In the distant past, this was a 1-2 second operation.
The text was updated successfully, but these errors were encountered: