Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: garbage collection confused by static tables on 386 #1925

Closed
gopherbot opened this issue Jun 7, 2011 · 10 comments
Closed

runtime: garbage collection confused by static tables on 386 #1925

gopherbot opened this issue Jun 7, 2011 · 10 comments
Milestone

Comments

@gopherbot
Copy link

by peter.polakovic:

When working with large images I found, that windows port doesn't free large chunks of
memory.

The following code

package main

import "fmt"
import "runtime"

func main() {
  for i:=0; i<10; i++ {
    a := make([]byte, 5000000)
    if a == nil { }
    a = nil    
    runtime.GC()
    fmt.Println(i, runtime.MemStats.Alloc, runtime.MemStats.Sys, runtime.MemStats.HeapObjects)
  }
  fmt.Println()
  for i:=0; i<10; i++ {
    a := make([]byte, 20000000)
    if a == nil { }
    a = nil    
    runtime.GC()
    fmt.Println(i, runtime.MemStats.Alloc, runtime.MemStats.Sys, runtime.MemStats.HeapObjects)
  }  
}

produces the following output

0 5176576 8887420 422
1 5213848 14564476 433
2 5213848 14564476 433
3 5213848 14564476 433
4 5213848 14564476 433
5 5213848 14564476 433
6 5213848 14564476 433
7 5213848 14564476 433
8 5213848 14564476 433
9 5213848 14564476 433

0 20250336 37125244 436
1 40288024 59686012 442
2 60325656 82246780 444
3 80363288 104807548 446
4 100400920 127368316 448
5 100437784 149929084 449
6 120438552 149929084 450
7 120475416 172489852 451
8 140476184 172489852 452
9 140513048 195050620 453

On OS X port first part of output seems similar to second one (allocated memory and
number of heap objects is the same in each iteration).

The same result when using runtime.Malloc() and runtime.Free().

I'm using binary distribution of release.r57.1 on Windows7 (32 bit).
@peterGo
Copy link
Contributor

peterGo commented Jun 7, 2011

Comment 1:

This is not a Windows port issue; it's a GOARCH=386 memory management issue. This issue
is likely a duplicate of issue #909.

@rsc
Copy link
Contributor

rsc commented Jun 7, 2011

Comment 2:

If it's a dup of 909 I am not sure where it is coming from.
Does the problem go away if you remove the import of "fmt"
and just use println?

Status changed to WaitingForReply.

@peterGo
Copy link
Contributor

peterGo commented Jun 7, 2011

Comment 3:

Yes.
fmt.Println -> println
windows/386
0 5070800 8887420 110
1 5107664 14564476 111
2 5107664 14564476 111
3 5107664 14564476 111
4 5107664 14564476 111
5 5107664 14564476 111
6 5107664 14564476 111
7 5107664 14564476 111
8 5107664 14564476 111
9 5107664 14564476 111
0 20144144 37125244 113
1 20181008 59686012 114
2 20181008 59686012 114
3 20181008 59686012 114
4 20181008 59686012 114
5 20181008 59686012 114
6 20181008 59686012 114
7 20181008 59686012 114
8 20181008 59686012 114
9 20181008 59686012 114
linux/386
0 5095152 9018492 52
1 5132016 14695548 53
2 5132016 14695548 53
3 5132016 14695548 53
4 5132016 14695548 53
5 5132016 14695548 53
6 5132016 14695548 53
7 5132016 14695548 53
8 5132016 14695548 53
9 5132016 14695548 53
0 20168496 37256316 55
1 20205360 59817084 56
2 20205360 59817084 56
3 20205360 59817084 56
4 20205360 59817084 56
5 20205360 59817084 56
6 20205360 59817084 56
7 20205360 59817084 56
8 20205360 59817084 56
9 20205360 59817084 56

@rsc
Copy link
Contributor

rsc commented Jun 7, 2011

Comment 4:

fmt imports unicode, which has some tables.
They should all be in static memory and thus
invisible to the garbage collector (and so not
compounding issue #909), but apparently they are not.
I have been meaning to fix this for months.

Owner changed to @rsc.

Status changed to 386GCconfusedbystatictables.

@rsc
Copy link
Contributor

rsc commented Jul 25, 2011

Comment 5:

Status changed to Accepted.

@gopherbot
Copy link
Author

Comment 6 by mt4swm:

Seeing it also on Linux/386 (golang tip revision), when
using index/suffixarray on quite large data (e.g. 4MB
text). Qsufsort internally creates a temporarily
inverted array `inv' of the same size as the suffix array
(http://golang.org/src/pkg/index/suffixarray/qsufsort.go?h=inv#L37).
It appears not to be freed after runtime.GC(), which --
in case of multiple search indices -- can sum up to a quite
large amount of data.
I created a minimal version of the program, which allocates 
temporary slices in a loop. The effect only shows up if "fmt" is
imported, and slices >= 4MB are allocated. It vanishes if "println" is
used instead (as in #3), or smaller slices are allocated.
There is also an effect of "saturation" visible:
    Nothing is freed up to an amount of 165MB, then the
    sum of allocated memory remains stable, i.e. on each
    GC run 4MB are freed:
  gc1: 0+0+0 ms 0 -> 0 MB 104 -> 102 (104-2) objects 40 pointer lookups (29 size, 11 addr)
  gc2: 0+0+0 ms 4 -> 4 MB 273 -> 233 (275-42) objects 242 pointer lookups (159 size, 83 addr)
  gc3: 0+0+0 ms 4 -> 4 MB 233 -> 233 (275-42) objects 243 pointer lookups (159 size, 84 addr)
  gc4: 0+0+0 ms 8 -> 8 MB 235 -> 235 (277-42) objects 247 pointer lookups (160 size, 87 addr)
  gc5: 0+0+0 ms 12 -> 12 MB 237 -> 237 (279-42) objects 260 pointer lookups (161 size, 99 addr)
  ...
  gc41: 8+1+0 ms 157 -> 157 MB 309 -> 309 (351-42) objects 1130 pointer lookups (197 size, 933 addr)
  gc42: 13+0+0 ms 161 -> 161 MB 311 -> 311 (353-42) objects 1132 pointer lookups (198 size, 934 addr)
  gc43: 5+0+0 ms 165 -> 165 MB 313 -> 313 (355-42) objects 1133 pointer lookups (199 size, 934 addr)
  gc44: 18+0+0 ms 169 -> 165 MB 315 -> 314 (357-43) objects 1134 pointer lookups (200 size, 934 addr)
  gc45: 11+0+0 ms 169 -> 165 MB 315 -> 314 (358-44) objects 1134 pointer lookups (200 size, 934 addr)
  ...
  gc81: 20+0+0 ms 169 -> 165 MB 315 -> 314 (394-80) objects 1134 pointer lookups (200 size, 934 addr)
  gc82: 16+0+0 ms 169 -> 165 MB 315 -> 314 (395-81) objects 1134 pointer lookups (200 size, 934 addr)
  In use: 173596448 337231104
By applying some debug output to runtime/mgc0.c I noticed that
it detected 41 pointers (41*4MB = 164MB) to the beginning of
the slice buffers while "consulting span table".

Attachments:

  1. test.go (420 bytes)

@rsc
Copy link
Contributor

rsc commented Dec 9, 2011

Comment 7:

Labels changed: added priority-later, removed priority-medium.

@rsc
Copy link
Contributor

rsc commented Dec 12, 2011

Comment 8:

Labels changed: added priority-go1.

@robpike
Copy link
Contributor

robpike commented Jan 13, 2012

Comment 9:

Owner changed to builder@golang.org.

@rsc
Copy link
Contributor

rsc commented Feb 19, 2012

Comment 11:

Status changed to Duplicate.

Merged into issue #909.

@rsc rsc added this to the Go1 milestone Apr 10, 2015
@rsc rsc removed the priority-go1 label Apr 10, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants