Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc generation step in build far too slow #5145

Closed
dgrove opened this issue Sep 13, 2012 · 18 comments
Closed

Doc generation step in build far too slow #5145

dgrove opened this issue Sep 13, 2012 · 18 comments
Labels
area-pkg closed-obsolete Closed as the reported issue is no longer relevant P3 A lower priority bug or feature request

Comments

@dgrove
Copy link
Contributor

dgrove commented Sep 13, 2012

Here's some data from running this morning on my Mac desktop (in release mode):

Thu Sep 13 08:30:23 PDT 2012 Parsing MDN data...
Thu Sep 13 08:30:24 PDT 2012 Cross-referencing dart:html...
Thu Sep 13 08:30:27 PDT 2012 Processing handwritten HTML documentation...
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/.svn/.svn.dart
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/args/args.dart
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/htmlescape/htmlescape.dart
Thu Sep 13 08:30:32 PDT 2012 Warning: could not find package at /Users/dgrove/repo/dart-bleeding/dart/pkg/logging/logging.dart
Thu Sep 13 08:30:32 PDT 2012 Generating docs...
Thu Sep 13 08:30:36 PDT 2012 Skipping MDN type UnknownElement

real 0m38.220s
user 0m37.589s
sys 0m0.571s

In the distant past, this was a 1-2 second operation.

@DartBot
Copy link

DartBot commented Feb 28, 2013

This comment was originally written by amouravski@google.com


From my brief experimentation, the actual doc extraction/generation is fairly quick, but the initial dart2js compilation is very slow. I would like it to be faster, but not sure what to do about it.
See issue #4257


Removed Area-SDK label.
Added Area-Dart2JS label.

@DartBot
Copy link

DartBot commented Feb 28, 2013

This comment was originally written by amouravski@google.com


Err, issue #4527

@peter-ahe-google
Copy link
Contributor

I have found a quadratic solution to a linear problem in the mirror system in the number of classes documented. I'll try that out when I have figured out what is wrong on the pub builders.

@peter-ahe-google
Copy link
Contributor

Investigating a bug in the dart2js source mirrors, I discovered that dartdoc is analyzing the same libraries twice:

Parsing MDN data...
Cross-referencing dart:html...
Running compiler null [dart:html, dart:svg, dart:web_audio]
Warning: could not find package at utils/apidoc/../../pkg/analyzer-experimental
Warning: could not find package at utils/apidoc/../../pkg/pathos
Warning: could not find package at utils/apidoc/../../pkg/browser
Generating docs...
Running compiler null [dart:async, dart:chrome, dart:collection, dart:core, dart:crypto, dart:html, dart:indexed_db, dart:io, dart:isolate, dart:json, dart:math, dart:mirrors, dart:scalarlist, dart:svg, dart:uri, dart:utf, dart:web_audio, dart:web_sql, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/meta/lib/meta.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/serialization/lib/serialization.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/yaml/lib/yaml.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/oauth2/lib/oauth2.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/intl/lib/intl.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/logging/lib/logging.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/http/lib/http.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/args/lib/args.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/scheduled_test/lib/scheduled_test.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/fixnum/lib/fixnum.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/unittest/lib/unittest.dart, file:///usr/local/google/home/ahe/Dart/all/dart/pkg/webdriver/lib/webdriver.dart]

That combined with a quadratic nature of docInheritance in sdk/lib/_internal/dartdoc/lib/dartdoc.dart makes me doubt that "actual doc extraction/generation is fairly quick".

So I'm sending this back to dartdoc. However, it is possible that some of the problems lie in the dart2js source mirrors. I suggest that Johnni and Andrei take a look at this together. I'll be happy to provide additional details if needed, but right now I'm investigating a higher priority bug, so forgive my terseness.


cc @johnniwinther.
cc @dgrove.
Removed the owner.
Removed Area-Dart2JS label.
Added Area-DartDoc label.

@DartBot
Copy link

DartBot commented Mar 8, 2013

This comment was originally written by amouravski@google.com


I have a few ideas about this and have started a few things.

dgrove: Could you give me an idea of how quick you're looking to make apidoc? ~30s? ~10? ~2?


Set owner to amouravski@google.com.
Added Started label.

@dgrove
Copy link
Contributor Author

dgrove commented Mar 8, 2013

My target is < 5s.

@DartBot
Copy link

DartBot commented Mar 8, 2013

This comment was originally written by amouravski@google.com


Okay, I'll try to hook up some metrics and for the time being I'm going to call the goal 7.5x improvement (based on numbers above) so that I can measure locally/on bots.

@dgrove
Copy link
Contributor Author

dgrove commented Mar 8, 2013

Just to be clear, that's 5s on a fast development box (the bots will likely take longer, but I'm not worried about them)

@munificent
Copy link
Member

Some context in case it helps:

I initially added apidoc to the build as a crude smoke test. At the time, dartdoc had a few unit tests, but nothing to validate that the end-to-end generation process worked. Because of the lack of tests, I was experiencing daily breakages where some other part of the Dart platform would change and apidoc would die. Adding apidoc to the build was a simple change to try to mitigate that.

If dartdoc's testing situation is in a better place, or the platform is more stable (I think at the time it was mostly frog (!) changes that were breaking it, so this is a pretty old decision), one option is to just stop building apidoc automatically. It doesn't serve much of a useful purpose: the generated docs are not part of the SDK, so it's output isn't used.

Of course, making apidoc faster is always a good thing, too. But if the goal here is to speed up the build iteration loop for the Dart team, it's hard to beat 0s by removing apidoc completely.

@larsbak
Copy link

larsbak commented Aug 27, 2013

Added this to the M7 milestone.

@DartBot
Copy link

DartBot commented Oct 8, 2013

This comment was originally written by amouravski@google.com


Added Triaged label.

@DartBot
Copy link

DartBot commented Oct 8, 2013

This comment was originally written by amouravski@google.com


Removed the owner.

@alan-knight
Copy link
Contributor

Removed this from the M7 milestone.

@anders-sandholm
Copy link
Contributor

Added old-dartdoc label.

@alan-knight
Copy link
Contributor

I don't see us being able to do much about this. The limiting factor is that it is essentially doing a dart2js compile on the entire contents of the repository. Probably the biggest factor is that I think dart2js is doing more work than it needs to, essentially doing most of a compile in order to be able to expose a mirror system. It could, for example, skip any consideration of optimizations.


Removed Priority-High, old-dartdoc labels.
Added Priority-Medium, NotPlanned labels.

@peter-ahe-google
Copy link
Contributor

If dart2js is compiling too much, it is because you're telling it to. It supports these options:

  --analyze-only
    Analyze but do not generate code.

  --analyze-signatures-only
    Skip analysis of method bodies and field initializers. This option implies
    --analyze-only.

But please don't hesitate to let Johnni and I know If you think you have an issue with dart2js doing too much work. Any ideas are appreciated!

If I create a source file named all.dart with this content:

import "dart:async";
import "dart:collection";
import "dart:convert";
import "dart:core";
import "dart:html";
import "dart:html_common";
import "dart:indexed_db";
import "dart:io";
import "dart:isolate";
import "dart:js";
import "dart:math";
import "dart:mirrors";
import "dart:typed_data";
import "dart:svg";
import "dart:web_audio";
import "dart:web_gl";
import "dart:web_sql";

And I run dart2js like this:

dart2js --analyze-all --analyze-only --analyze-signatures-only --categories=all all.dart

The compiler completes in less than 4 seconds. So 40 seconds isn't acceptable.


Removed Priority-Medium label.
Added Priority-High, Triaged labels.

@alan-knight
Copy link
Contributor

Interesting. We are already passing those options in, and it does look like I was wrong in attributing too much time to dart2js. It seems to be about 8 seconds out of 30. Trivial playing around with the profiler suggests that the actual culprit is markdown parsing of comments.


Removed Priority-High label.
Added Priority-Medium label.

@peter-ahe-google
Copy link
Contributor

I took the measurement on my 2 GHz Intel Core i7 MacBook Pro. I'm guessing you measured on something equivalent. So it sounds as if creating the mirror system is already twice as slow as it should be. Are you running in checked mode, or are you analyzing a lot of packages as well?

@dgrove dgrove added Type-Defect P3 A lower priority bug or feature request area-pkg closed-obsolete Closed as the reported issue is no longer relevant labels Feb 13, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-pkg closed-obsolete Closed as the reported issue is no longer relevant P3 A lower priority bug or feature request
Projects
None yet
Development

No branches or pull requests

7 participants