runtime: expose total stack usage in MemStats #7468
Labels
Milestone
Comments
> Right now they are treated as regular malloc'd memory. They are not profiled by mprof. > Non-standard-sized stack allocations should be accounted under StackSys. If we just account it in StackSys, then part of stacks (std size) will be accounted only in StackSys, and some part will be accounted both in StackSys and HeapSys. Since you don't know what is the proportion between these parts, you can't make sense out of HeapSys and heap profile -- heap profile shows less than HeapSys, but you don't know what part of that difference is stacks. mprof profile and memstats must be consistent and allow to verify sanity (tell whether mprof lies or not). Labels changed: added release-go1.3, repo-main. |
Dmitry, did your change https://golang.org/cl/72440043 fix this? |
StackInuse is also broken (StackInuse decreases during function execution):
go version devel +3cd546ac3a99 Tue Mar 11 16:36:14 2014 +1100 linux/amd64
$ go test -v -cpu 1,1,1,1 runtime -test.run=TestStack
=== RUN TestStackOverflow
--- PASS: TestStackOverflow (0.68 seconds)
=== RUN TestStackSplit
--- PASS: TestStackSplit (0.01 seconds)
=== RUN TestStackMem
--- PASS: TestStackMem (3.09 seconds)
stack_test.go:1576: Consumed 2MB for stack mem
stack_test.go:1582: Inuse 0MB for stack mem
=== RUN TestStackOverflow
--- PASS: TestStackOverflow (0.67 seconds)
=== RUN TestStackSplit
--- PASS: TestStackSplit (0.01 seconds)
=== RUN TestStackMem
--- FAIL: TestStackMem (3.10 seconds)
stack_test.go:1576: Consumed 0MB for stack mem
stack_test.go:1582: Inuse 17592186044415MB for stack mem
stack_test.go:1584: Stack inuse: want 4194304, got 18446744073709547520
|
StackSys is listed as one of a collection under "Low-level fixed-size structure allocator statistics." More generally the Sys numbers are supposed to correspond 1:1 with mmap requests made to the system. I don't think we should redefine what StackSys/StackInuse mean to be different than the other Sys/Inuse pairs. In particular the sum of all the XxxSys fields needs to match the Sys field. Maybe instead we can add a new field that counts total stack usage? Maybe just StackAlloc. It's fine for Go 1.3 if this is a noninvasive change. Labels changed: added release-go1.3maybe, removed release-go1.3. Status changed to Accepted. |
> Maybe instead we can add a new field that counts total stack usage? Maybe just StackAlloc. We also must be able to understand what is heap size w/o stacks. There is already StackInuse which currently counts only fixed-size cache allocations. Maybe we can make StackInuse represent total stack memory (cache + mallocgc), and then subtract malloced stacks from HeapAlloc? But this introduces another problem: HeapAlloc and HeapSys won't match... |
|
This issue was updated by revision 00e6fc1. LGTM=rsc R=golang-codereviews, bradfitz CC=golang-codereviews, khr, rsc https://golang.org/cl/74010043 |
CL https://golang.org/cl/104200047 mentions this issue. |
|
This issue was closed by revision 7c13860. Status changed to Fixed. |
|
This issue was closed by revision 3cf83c1. |
CL https://golang.org/cl/112240044 mentions this issue. |
|
This issue was closed by revision f378f30. |
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 25, 2018
In order to move malloc to Go, we need to have a separate stack allocator. If we run out of stack during malloc, malloc will not be available to allocate a new stack. Stacks are the last remaining FlagNoGC objects in the GC heap. Once they are out, we can get rid of the distinction between the allocated/blockboundary bits. (This will be in a separate change.) Fixes golang#7468 Fixes golang#7424 LGTM=rsc, dvyukov R=golang-codereviews, dvyukov, khr, dave, rsc CC=golang-codereviews https://golang.org/cl/104200047
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 25, 2018
Breaks windows and race detector. TBR=rsc ««« original CL description runtime: stack allocator, separate from mallocgc In order to move malloc to Go, we need to have a separate stack allocator. If we run out of stack during malloc, malloc will not be available to allocate a new stack. Stacks are the last remaining FlagNoGC objects in the GC heap. Once they are out, we can get rid of the distinction between the allocated/blockboundary bits. (This will be in a separate change.) Fixes golang#7468 Fixes golang#7424 LGTM=rsc, dvyukov R=golang-codereviews, dvyukov, khr, dave, rsc CC=golang-codereviews https://golang.org/cl/104200047 »»» TBR=rsc CC=golang-codereviews https://golang.org/cl/101570044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jun 25, 2018
redo stack allocation. This is mostly the same as the original CL with a few bug fixes. 1. add racemalloc() for stack allocations 2. fix poolalloc/poolfree to terminate free lists correctly. 3. adjust span ref count correctly. 4. don't use cache for sizes >= StackCacheSize. Should fix bugs and memory leaks in original changelist. ««« original CL description undo CL 104200047 / 318b04f28372 Breaks windows and race detector. TBR=rsc ««« original CL description runtime: stack allocator, separate from mallocgc In order to move malloc to Go, we need to have a separate stack allocator. If we run out of stack during malloc, malloc will not be available to allocate a new stack. Stacks are the last remaining FlagNoGC objects in the GC heap. Once they are out, we can get rid of the distinction between the allocated/blockboundary bits. (This will be in a separate change.) Fixes golang#7468 Fixes golang#7424 LGTM=rsc, dvyukov R=golang-codereviews, dvyukov, khr, dave, rsc CC=golang-codereviews https://golang.org/cl/104200047 »»» TBR=rsc CC=golang-codereviews https://golang.org/cl/101570044 »»» LGTM=dvyukov R=dvyukov, dave, khr, alex.brainman CC=golang-codereviews https://golang.org/cl/112240044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 9, 2018
In order to move malloc to Go, we need to have a separate stack allocator. If we run out of stack during malloc, malloc will not be available to allocate a new stack. Stacks are the last remaining FlagNoGC objects in the GC heap. Once they are out, we can get rid of the distinction between the allocated/blockboundary bits. (This will be in a separate change.) Fixes golang#7468 Fixes golang#7424 LGTM=rsc, dvyukov R=golang-codereviews, dvyukov, khr, dave, rsc CC=golang-codereviews https://golang.org/cl/104200047
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 9, 2018
Breaks windows and race detector. TBR=rsc ««« original CL description runtime: stack allocator, separate from mallocgc In order to move malloc to Go, we need to have a separate stack allocator. If we run out of stack during malloc, malloc will not be available to allocate a new stack. Stacks are the last remaining FlagNoGC objects in the GC heap. Once they are out, we can get rid of the distinction between the allocated/blockboundary bits. (This will be in a separate change.) Fixes golang#7468 Fixes golang#7424 LGTM=rsc, dvyukov R=golang-codereviews, dvyukov, khr, dave, rsc CC=golang-codereviews https://golang.org/cl/104200047 »»» TBR=rsc CC=golang-codereviews https://golang.org/cl/101570044
wheatman
pushed a commit
to wheatman/go-akaros
that referenced
this issue
Jul 9, 2018
redo stack allocation. This is mostly the same as the original CL with a few bug fixes. 1. add racemalloc() for stack allocations 2. fix poolalloc/poolfree to terminate free lists correctly. 3. adjust span ref count correctly. 4. don't use cache for sizes >= StackCacheSize. Should fix bugs and memory leaks in original changelist. ««« original CL description undo CL 104200047 / 318b04f28372 Breaks windows and race detector. TBR=rsc ««« original CL description runtime: stack allocator, separate from mallocgc In order to move malloc to Go, we need to have a separate stack allocator. If we run out of stack during malloc, malloc will not be available to allocate a new stack. Stacks are the last remaining FlagNoGC objects in the GC heap. Once they are out, we can get rid of the distinction between the allocated/blockboundary bits. (This will be in a separate change.) Fixes golang#7468 Fixes golang#7424 LGTM=rsc, dvyukov R=golang-codereviews, dvyukov, khr, dave, rsc CC=golang-codereviews https://golang.org/cl/104200047 »»» TBR=rsc CC=golang-codereviews https://golang.org/cl/101570044 »»» LGTM=dvyukov R=dvyukov, dave, khr, alex.brainman CC=golang-codereviews https://golang.org/cl/112240044
This issue was closed.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The text was updated successfully, but these errors were encountered: