Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Analyzer using excessive memory/out of memory crash #20399

Closed
leafpetersen opened this issue Aug 6, 2014 · 4 comments
Closed

Analyzer using excessive memory/out of memory crash #20399

leafpetersen opened this issue Aug 6, 2014 · 4 comments
Assignees
Labels
area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. closed-obsolete Closed as the reported issue is no longer relevant

Comments

@leafpetersen
Copy link
Member

With a clean bleeding_edge checkout of dart, running the analyzer tests as

./tools/test.py --time --report --mode=release -p verbose --compiler=dartanalyzer --runtime=none

brings my Z620 to its knees. Adding -j 1 and watching top, I see the resident memory size grow steadily to about 6G.

On my mac laptop, running as above (with -j 1) I get an out of memory crash as shown below.

FAILED: dartanalyzer-none release_ia32 analyze_library/lib/_internal/pub/lib/src/barback/asset_environment
Expected: Pass
Actual: Crash
CommandOutput[dartanalyzer]:

stderr:
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:64)
    at java.lang.StringBuffer.<init>(StringBuffer.java:108)
    at java.net.URI.resolvePath(URI.java:1999)
    at java.net.URI.resolve(URI.java:2066)
    at java.net.URI.resolve(URI.java:1022)
    at com.google.dart.engine.source.FileBasedSource.resolveRelative(FileBasedSource.java:192)
    at com.google.dart.engine.source.SourceFactory.internalResolveUri(SourceFactory.java:231)
    at com.google.dart.engine.source.SourceFactory.resolveUri(SourceFactory.java:164)
    at com.google.dart.engine.internal.task.ParseDartTask.resolveDirective(ParseDartTask.java:69)
    at com.google.dart.engine.internal.task.ParseDartTask.internalPerform(ParseDartTask.java:303)
    at com.google.dart.engine.internal.task.AnalysisTask.safelyPerform(AnalysisTask.java:119)
    at com.google.dart.engine.internal.task.AnalysisTask.perform(AnalysisTask.java:73)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.cacheDartParseData(AnalysisContextImpl.java:2655)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.computeResolvableCompilationUnit(AnalysisContextImpl.java:1269)
    at com.google.dart.engine.internal.resolver.Library.getAST(Library.java:139)
    at com.google.dart.engine.internal.resolver.Library.getDefiningCompilationUnit(Library.java:181)
    at com.google.dart.engine.internal.resolver.LibraryElementBuilder.buildLibrary(LibraryElementBuilder.java:87)
    at com.google.dart.engine.internal.resolver.LibraryResolver.buildElementModels(LibraryResolver.java:559)
    at com.google.dart.engine.internal.resolver.LibraryResolver.resolveLibrary(LibraryResolver.java:291)
    at com.google.dart.engine.internal.task.ResolveDartLibraryTask.internalPerform(ResolveDartLibraryTask.java:97)
    at com.google.dart.engine.internal.task.AnalysisTask.safelyPerform(AnalysisTask.java:119)
    at com.google.dart.engine.internal.task.AnalysisTask.perform(AnalysisTask.java:73)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.cacheDartResolutionData(AnalysisContextImpl.java:2695)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.getDartResolutionData(AnalysisContextImpl.java:3434)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.getDartResolutionData(AnalysisContextImpl.java:3466)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.resolveCompilationUnit(AnalysisContextImpl.java:1993)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.resolveCompilationUnit(AnalysisContextImpl.java:1987)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.cacheDartVerificationData(AnalysisContextImpl.java:2773)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.getDartVerificationData(AnalysisContextImpl.java:3541)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.computeErrors(AnalysisContextImpl.java:1171)
    at com.google.dart.command.analyze.AnalyzerImpl.getAllErrors(AnalyzerImpl.java:257)
    at com.google.dart.command.analyze.AnalyzerImpl.performAnalysis(AnalyzerImpl.java:141)

Command[dartanalyzer]: sdk/bin/dartanalyzer_developer --ignore-unrecognized-flags --show-package-warnings --package-root=xcodebuild/ReleaseIA32/packages/ --fatal-warnings --fatal-type-errors --show-sdk-warnings --machine --no-hints /Users/leafp/src/dart-repo/dart/sdk/lib/_internal/pub/lib/src/barback/asset_environment.dart
Took 0:00:53.740000

@bwilkerson
Copy link
Member

I wasn't able to reproduce this behavior on my Mac laptop. When I run the analyzer tests I use the following command:

./tools/test.py --mode release --compiler dartanalyzer --runtime none --progress color --report --tasks=2

This starts two JVM's, both of which reach a maximum memory usage of around 625MB (as reported by Activity Monitor).

So then I tried the command you are using:

./tools/test.py --time --report --mode=release -p verbose --compiler=dartanalyzer --runtime=none -j 1

And it started a single JVM that never went above 615MB.

Rico: any insight you can offer would be appreciated.


Set owner to @ricowind.
Removed Priority-Unassigned label.
Added Priority-Medium label.

@leafpetersen
Copy link
Member Author

Couple of clarifications.

  1. On my linux box it uses 6G, on my mac laptop it peaks out at 615M or so, as you observe.
  2. Nonetheless, the out of memory error is pretty reproducible on my laptop, just ran it again with the crash log below (again, with the peak memory as reported by the activity monitor at 615M or so). This was using the command line you suggested above.
  3. This isn't really blocking me - I just have to remember to run analyzer tests on my linux box and use -j n with small n. Just reported it since I thought it might be of interest. It looks a lot like something is leaking memory over time, since the memory usage grows over time, and is much larger than the memory needed for any single file (and since apparently much of the heap is live, per the out of memory error below).

Here's the java version from my machine:

leafp-macbookpro:dart leafp$ java -version
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Expected: Pass
Actual: Crash
CommandOutput[dartanalyzer]:

stderr:
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.nio.HeapCharBuffer.subSequence(HeapCharBuffer.java:576)
    at java.nio.HeapCharBuffer.subSequence(HeapCharBuffer.java:42)
    at com.google.dart.engine.scanner.CharSequenceReader.getString(CharSequenceReader.java:62)
    at com.google.dart.engine.scanner.Scanner.tokenizeIdentifier(Scanner.java:773)
    at com.google.dart.engine.scanner.Scanner.tokenizeKeywordOrIdentifier(Scanner.java:826)
    at com.google.dart.engine.scanner.Scanner.bigSwitch(Scanner.java:231)
    at com.google.dart.engine.scanner.Scanner.tokenize(Scanner.java:183)
    at com.google.dart.engine.internal.task.ScanDartTask.internalPerform(ScanDartTask.java:147)
    at com.google.dart.engine.internal.task.AnalysisTask.safelyPerform(AnalysisTask.java:119)
    at com.google.dart.engine.internal.task.AnalysisTask.perform(AnalysisTask.java:73)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.cacheDartScanData(AnalysisContextImpl.java:2731)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.cacheDartParseData(AnalysisContextImpl.java:2654)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.computeResolvableCompilationUnit(AnalysisContextImpl.java:1269)
    at com.google.dart.engine.internal.resolver.Library.getAST(Library.java:139)
    at com.google.dart.engine.internal.resolver.Library.getDefiningCompilationUnit(Library.java:181)
    at com.google.dart.engine.internal.resolver.LibraryElementBuilder.buildLibrary(LibraryElementBuilder.java:87)
    at com.google.dart.engine.internal.resolver.LibraryResolver.buildElementModels(LibraryResolver.java:559)
    at com.google.dart.engine.internal.resolver.LibraryResolver.resolveLibrary(LibraryResolver.java:291)
    at com.google.dart.engine.internal.task.ResolveDartLibraryTask.internalPerform(ResolveDartLibraryTask.java:97)
    at com.google.dart.engine.internal.task.AnalysisTask.safelyPerform(AnalysisTask.java:119)
    at com.google.dart.engine.internal.task.AnalysisTask.perform(AnalysisTask.java:73)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.cacheDartResolutionData(AnalysisContextImpl.java:2695)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.getDartResolutionData(AnalysisContextImpl.java:3434)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.getDartResolutionData(AnalysisContextImpl.java:3466)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.resolveCompilationUnit(AnalysisContextImpl.java:1993)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.resolveCompilationUnit(AnalysisContextImpl.java:1987)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.cacheDartVerificationData(AnalysisContextImpl.java:2773)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.getDartVerificationData(AnalysisContextImpl.java:3541)
    at com.google.dart.engine.internal.context.AnalysisContextImpl.computeErrors(AnalysisContextImpl.java:1171)
    at com.google.dart.command.analyze.AnalyzerImpl.getAllErrors(AnalyzerImpl.java:257)
    at com.google.dart.command.analyze.AnalyzerImpl.performAnalysis(AnalyzerImpl.java:141)
    at com.google.dart.command.analyze.AnalyzerImpl.analyze(AnalyzerImpl.java:110)

Command[dartanalyzer]: sdk/bin/dartanalyzer_developer --ignore-unrecognized-flags --show-package-warnings --package-root=xcodebuild/ReleaseIA32/packages/ --fatal-warnings --fatal-type-errors --show-sdk-warnings --machine --no-hints /Users/leafp/src/dart-repo/dart/sdk/lib/_internal/pub/bin/pub.dart

@bwilkerson
Copy link
Member

@rico: Changed area in case you filter by area when looking at issues.


Removed Area-Analyzer label.
Added Area-Infrastructure label.

@ricowind
Copy link
Contributor

ricowind commented Jun 2, 2015

Added AssumedStale label.

@leafpetersen leafpetersen added Type-Defect area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. closed-obsolete Closed as the reported issue is no longer relevant labels Jun 2, 2015
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. closed-obsolete Closed as the reported issue is no longer relevant
Projects
None yet
Development

No branches or pull requests

3 participants