Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak related to string <-> []byte conversion #5109

Closed
gopherbot opened this issue Mar 22, 2013 · 5 comments
Closed

Memory leak related to string <-> []byte conversion #5109

gopherbot opened this issue Mar 22, 2013 · 5 comments

Comments

@gopherbot
Copy link

by mux2005:

What steps will reproduce the problem?

Run the attached program.


What is the expected output?

Something like this

40836728
53903472
79921352
105976016
105976016
<last number repeated>

What do you see instead?

The memory usage keeps increasing till the program dies with out of memory.


Which compiler are you using (5g, 6g, 8g, gccgo)?

8g

Which operating system are you using?

Ubuntu 10.04 32bit


Which version are you using?  (run 'go version')

go version go1.0.2 +2d8bc3c94ecb
(from http://ppa.launchpad.net/gophers/go/ubuntu/)
Judged by the hg revision this seems to be go 1.0.3 even though the version string still
says 1.0.2.


Please provide any additional information below.

1. The problem depends on the size of fragment_size in the example program. Up to a
certain size everything is fine, the memory usage stabilizes. But once it exceeds the
safe number there is unlimited growth.

2. I came upon this problem while examining why our server application keeps crashing
with out of memory. In that application the problem occurs when the String() function on
large (about 20MB) bytes.Buffer objects is called repeatedly. Either the strings or the
buffers or both never get freed even after references to both have been dropped.

3. The problem does not occur with golang-tip from the gophers PPA, but I don't know if
this is because of an actual fix in the GC or because the optimizer optimizes away some
of the useless code. I cannot test golang-tip with our actual server application because
golang-tip is still too buggy (The application compiles but doesn't pass the regression
test suite).

Attachments:

  1. testgarbage.go (731 bytes)
@DanielMorsing
Copy link
Contributor

Comment 1:

What you're seeing is most likely a result of the work on the GC since 1.0.3
If possible, use a 64-bit system. The issues with the conservative garbage collector is
not as pronounced on the increased address space.

Status changed to Duplicate.

Merged into issue #909.

@gopherbot
Copy link
Author

Comment 2 by mux2005:

But from reading 909 that seems to be related to the GC seeing data as pointers that
isn't. I don't see how that could happen in my example program, since all the data is 0s.

@remyoudompheng
Copy link
Contributor

Comment 3:

It's not about your data containing pointers, it's about your data being the target of
pointers.
A 13MB chunk of memory has a 1/315 probability of being the target of a random number
interpreted as a pointer.
The packages you import ("fmt" is extremely complex) may contain global variables, where
extra data is interpreted as pointers, and the runtime contains global variables, for
example heap profiling data, which can quickly pin the various chunks of memory you are
using.
With 13MB sized chunks you only need 300 misinterpreted numbers to fill out 4GB. It
happens very quickyl considering the various sources of global variables that exist in a
typical program.

@remyoudompheng
Copy link
Contributor

Comment 4:

For example, if you set runtime.Memprofilerate to zero, and don't import the "fmt"
package (use println instead), does the memory increase as fast?

@gopherbot
Copy link
Author

Comment 5:

"... I don't see how that could happen in my example program, since all the data is 0s."
It can happen in the example program because there is exists other (internal) data in
Go's memory that points to memory allocated by the example program.
The leak is fixed in Go tip:
$ ./issue5109
40365608
40416560
66397496
66397496
66397496
66397496
66397496
66397496
66397496
...

@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants