My favorites | Sign in
go
Project Home Downloads Wiki Issues Source Code Search
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 2323: runtime: windows DLL mappings interfere with 32-bit memory allocation
21 people starred this issue and may be notified of changes. Back to list
Status:  HelpWanted
Owner:  buil...@golang.org
Cc:  i...@golang.org


Sign in to add a comment
 
Reported by zippo...@gmail.com, Sep 30, 2011
Before filing a bug, please check whether it has been fixed since
the latest release: run "hg pull", "hg update default", rebuild, and retry
what you did to
reproduce the problem.  Thanks.

What steps will reproduce the problem?
1. Installing gomingw installer for build of release r60.1
2. Executing make, then make install in the (attached to this issue) package's directory.
3. Testing the installed package by calling Now().

What is the expected output?
The x position of the cursor.

What do you see instead?
An error:
>> throw: runtime: cannot reserve arena virtual address space

Which compiler are you using (5g, 6g, 8g, gccgo)?
8g

Which operating system are you using?
Windows 7

Which revision are you using?  (hg identify)
I ran "Installer for Win32 (experimental) build of release.r60.1"
(hd identify doesn't cooperate, sorry)

Please provide any additional information below.
somecgopackage.go
188 bytes   View   Download
Makefile
122 bytes   View   Download
Oct 3, 2011
Project Member #1 alex.bra...@gmail.com
I can't reproduce your problem on Windows 7:

$ cat Makefile
include $(GOROOT)/src/Make.inc

TARG=a
CGOFILES=\
        a.go\

include $(GOROOT)/src/Make.pkg
$ cat a.go
package a
/*
#include <windows.h>
int xPlease() {
        POINT p;
        GetCursorPos(&p);
        return p.x;
}
*/
import "C"

func Now() {
        println("I got that. It's: ", int(C.xPlease()))
}
$ cat a_test.go
package a_test

import (
        "a"
        "testing"
)

func TestA(t *testing.T) {
        a.Now()
}
$ make test
gotest
rm -f _test/a.a
CGOPKGPATH= cgo --  a.go
touch _obj/_cgo_run
8g  -o _gotest_.8  _obj/a.cgo1.go _obj/_cgo_gotypes.go
8c -FVw -Ic:/MinGW/go/pkg/windows_386 -I . -o "_cgo_defun.8" _obj/_cgo_defun.c
gcc -m32 -I . -g -fPIC -O2 -o _cgo_main.o -c   _obj/_cgo_main.c
_obj/_cgo_main.c:1:0: warning: -fPIC ignored for target (all code is position independent)
gcc -m32 -I . -g -fPIC -O2 -o a.cgo2.o -c   _obj/a.cgo2.c
_obj/a.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position independent)
gcc -m32 -I . -g -fPIC -O2 -o _cgo_export.o -c   _obj/_cgo_export.c
_obj/_cgo_export.c:1:0: warning: -fPIC ignored for target (all code is position independent)
gcc -m32 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o a.cgo2.o _cgo_export.o
cgo -dynimport _cgo1_.o >_obj/_cgo_import.c_ && mv -f _obj/_cgo_import.c_ _obj/_cgo_import.c
8c -FVw -I . -o "_cgo_import.8" _obj/_cgo_import.c
rm -f _test/a.a
gopack grc _test/a.a _gotest_.8  _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export.o
I got that. It's:  1115
PASS
$ 
Status: WaitingForReply
Owner: alex.bra...@gmail.com
Oct 4, 2011
#2 zippo...@gmail.com
I have same files and I typed the same commands shown in your comment, but I get:
$ make test
gotest
rm -f _test/a.a
CGOPKGPATH= cgo --  a.go
touch _obj/_cgo_run
8g  -o _gotest_.8  _obj/a.cgo1.go _obj/_cgo_gotypes.go
8c -FVw -IC:/Go/pkg/windows_386 -I . -o "_cgo_defun.8" _obj/_cgo_defun.c
gcc -m32 -I . -g -fPIC -O2 -o _cgo_main.o -c   _obj/_cgo_main.c
_obj/_cgo_main.c:1:0: warning: -fPIC ignored for target (all code is position in
dependent)
gcc -m32 -I . -g -fPIC -O2 -o a.cgo2.o -c   _obj/a.cgo2.c
_obj/a.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position indep
endent)
gcc -m32 -I . -g -fPIC -O2 -o _cgo_export.o -c   _obj/_cgo_export.c
_obj/_cgo_export.c:1:0: warning: -fPIC ignored for target (all code is position
independent)
gcc -m32 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o a.cgo2.o _cgo_export.o
cgo -dynimport _cgo1_.o >_obj/_cgo_import.c_ && mv -f _obj/_cgo_import.c_ _obj/_
cgo_import.c
8c -FVw -I . -o "_cgo_import.8" _obj/_cgo_import.c
rm -f _test/a.a
gopack grc _test/a.a _gotest_.8  _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export
.o
throw: runtime: cannot reserve arena virtual address space

gotest: "./8.out.exe" failed: exit status 2
make: *** [test] Error 2

===================
Maybe cgo doesn't use the right C++ compiler? (I don't really understand these things, I just installed mingw with it's compiler).
My UAC (security settings) are turned off, btw.
Oct 4, 2011
Project Member #3 alex.bra...@gmail.com
I do not what your problem is. But before we do anything else, maybe you can tell us why this error is happening. If you could change your $GOROOT/src/pkg/runtime/windows/mem.c file like this:

diff -r 6aa111bd2d11 src/pkg/runtime/windows/mem.c
--- a/src/pkg/runtime/windows/mem.c	Tue Oct 04 15:07:28 2011 +1100
+++ b/src/pkg/runtime/windows/mem.c	Wed Oct 05 14:21:57 2011 +1100
@@ -55,7 +55,12 @@
 		return v;
 	
 	// Next let the kernel choose the address.
-	return runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE, (uintptr)PAGE_EXECUTE_READWRITE);
+	v = runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE, (uintptr)PAGE_EXECUTE_READWRITE);
+	if(v == nil) {
+		runtime·printf("VirtualAlloc failed with errno=%d\n", runtime·getlasterror());
+		runtime·throw("VirtualAlloc");
+	}
+	return v;
 }
 
 void

Then you would need to rebuild "runtime" package, like

cd $GOROOT/src/pkg/runtime
make clean install

Then re-run your program and tell us what errno VirtualAlloc function returns. Thank you.

Alex
Oct 4, 2011
#4 zippo...@gmail.com
Compiling with your patch added these lines:
VirtualAlloc failed with errno=8
throw: VirtualAlloc

Hope this helps.
Oct 5, 2011
Project Member #5 alex.bra...@gmail.com
errno=8 is ERROR_NOT_ENOUGH_MEMORY, so it sounds like your program fails to allocate required block of address space, perhaps, because there is not enough room to do it.

I take it, some simple program like:

package main

func main() {
        println("Hello")
}

works. Right?

Could you, please revert change to your $GOROOT/src/pkg/runtime/windows/mem.c file, and apply this one, instead:

diff -r 6aa111bd2d11 src/pkg/runtime/windows/mem.c
--- a/src/pkg/runtime/windows/mem.c	Tue Oct 04 15:07:28 2011 +1100
+++ b/src/pkg/runtime/windows/mem.c	Thu Oct 06 12:21:13 2011 +1100
@@ -48,14 +48,25 @@
 void*
 runtime·SysReserve(void *v, uintptr n)
 {
+	runtime·printf("SysReserve: v=%p n=%d\n", v, n);
 	// v is just a hint.
 	// First try at v.
 	v = runtime·stdcall(runtime·VirtualAlloc, 4, v, n, (uintptr)MEM_RESERVE, (uintptr)PAGE_EXECUTE_READWRITE);
-	if(v != nil)
+	if(v != nil) {
+		runtime·printf("SysReserve: hint worked v=%p\n", v);
 		return v;
+	} else {
+		runtime·printf("SysReserve: hint failed with errno=%d\n", runtime·getlasterror());
+	}
 	
 	// Next let the kernel choose the address.
-	return runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE, (uintptr)PAGE_EXECUTE_READWRITE);
+	v = runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE, (uintptr)PAGE_EXECUTE_READWRITE);
+	if(v == nil) {
+		runtime·printf("SysReserve: accepting any address failed with errno=%d\n", runtime·getlasterror());
+		runtime·throw("VirtualAlloc");
+	}
+	runtime·printf("SysReserve: accepting any address worked v=%p\n", v);
+	return v;
 }
 
 void

Then re-run you test again and show us the output. Please, also run little program above and send its output too. I would like to compare the two.

I still do not know what is going on, perhaps, when you link extra mingw windows functions, they use some extra address space, maybe they load some extra dlls, that we have not enough room to move afterwards.

Thank you.

Alex
Oct 5, 2011
#6 zippo...@gmail.com
Printing "Hello" works of course, or I wasn't moving on to Windows API :P
I have 3.49GB usable memory and I successfully called Windows API with C++ (using mingw gcc).

Okay I applied your patch and here is the output for the test:
$ make test
gotest
SysReserve: v=0x920118 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address worked v=0x38020000
rm -f _test/a.a
rm -f _test/a.a
gopack grc _test/a.a _gotest_.8  _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export
.o
SysReserve: v=0x8fa1e0 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address failed with errno=8
throw: VirtualAlloc

gotest: "./8.out.exe" failed: exit status 2
make: *** [test] Error 2

And for the little program:
$ 8.out
SysReserve: v=0x832e58 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address worked v=0x38020000
Hello
Oct 6, 2011
Project Member #7 alex.bra...@gmail.com
I am running out of ideas. :-)

The problem here is not the amount of memory you have in your computer, but free block of address space available in your process. Go memory manager is designed to use 1 big block of address space (see that request for 805306368 bytes). It asks OS to reserve single contiguous block of addresses for future use. No real memory is associated with these addresses at the start, but will get assigned as addresses are used by the program. The problem here is that OS refuses to allocate this 805306368 bytes block. I don't know why. Perhaps, there is no contiguous block of such size at the time, because memory is used by other parts of your process (dlls and such).

What I would do next is put a bug in mem.c: replace line with "runtime·throw("VirtualAlloc");" with "*(int32*)0 = 0;". This will raise an exception at the right moment in your program. If you run it under debugger (I use http://www.ollydbg.de/), you should be able to see memory map of your process, this will give you a clue if there is free block of addresses or not and why. Here is my run:

SysReserve: v=0xa00000 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address worked v=0xd30000

here is my map:

Memory map
Address   Size      Owner            Section    Contains                 Type  Access   Initial >Mapped as
00010000  00001000 >                >           Environment              Priv >RW       RW
00020000  00001000 >                >           Process Parameters       Priv >RW       RW
0005F000  00001000 >                >                                    Priv >RW  Guar>RW  Guar>
00060000  00010000 >                >           Stack of main thread     Priv >RW       RW
00070000  00006000 >                >                                    Priv >RW       RW
00170000  00003000 >                >                                    Map  >RW       RW
00180000  00016000 >                >                                    Map  >R        R        C:\WINNT\system32\unicode.nls
001A0000  0002F000 >                >                                    Map  >R        R        C:\WINNT\system32\locale.nls
001D0000  00041000 >                >                                    Map  >R        R        C:\WINNT\system32\sortkey.nls
00220000  00004000 >                >                                    Map  >R        R        C:\WINNT\system32\sorttbls.nls
00230000  00008000 >                >                                    Map  >R E      R E
002F0000  00002000 >                >                                    Map  >R E      R E
00300000  00043000 >                >           GDI handles              Map  >R        R
00400000  00001000 >8_out           >           PE header                Img  >R        RWE Copy>
00401000  000E9000 >8_out           >.text      Code                     Img  >R E      RWE Copy>
004EA000  00421000 >8_out           >.data      Data                     Img  >RW  Copy>RWE Copy>
0090B000  00001000 >8_out           >/4                                  Img  >R        RWE Copy>
0090C000  0000E000 >8_out           >/18                                 Img  >R        RWE Copy>
0091A000  0000A000 >8_out           >/30                                 Img  >R        RWE Copy>
00924000  0002B000 >8_out           >/43                                 Img  >R        RWE Copy>
0094F000  00010000 >8_out           >/55                                 Img  >R        RWE Copy>
0095F000  00003000 >8_out           >/71                                 Img  >R        RWE Copy>
00962000  00001000 >8_out           >/87                                 Img  >R        RWE Copy>
00963000  00001000 >8_out           >/102                                Img  >R        RWE Copy>
00964000  00001000 >8_out           >.idata     Imports                  Img  >RW  Copy>RWE Copy>
00965000  00001000 >8_out           >.edata     Exports                  Img  >R        RWE Copy>
00966000  00001000 >8_out           >.symtab                             Img  >R        RWE Copy>
00970000  000B5000 >                >                                    Map  >R E      R E
00C70000  00001000 >                >                                    Priv >RW       RW
00C80000  00001000 >                >                                    Priv >RW       RW
00C90000  00001000 >                >                                    Priv >RW       RW
00D10000  00005000 >                >                                    Priv >RW       RW
00D20000  00002000 >                >                                    Map  >R        R        C:\WINNT\system32\ctype.nls
77570000  00001000 >winmm           >           PE header                Img  >R        RWE Copy>
77571000  0001F000 >winmm           >.text      Code,imports,exports     Img  >R E      RWE Copy>
77590000  00005000 >winmm           >.data      Data                     Img  >RW  Copy>RWE Copy>
77595000  00009000 >winmm           >.rsrc      Resources                Img  >R        RWE Copy>
7759E000  00002000 >winmm           >.reloc     Relocations              Img  >R        RWE Copy>
77D30000  00001000 >RPCRT4          >           PE header                Img  >R        RWE Copy>
77D31000  00067000 >RPCRT4          >.text,.orp>Code,imports,exports     Img  >R E      RWE Copy>
77D98000  00002000 >RPCRT4          >.data      Data                     Img  >RW       RWE Copy>
77D9A000  00001000 >RPCRT4          >.rsrc      Resources                Img  >R        RWE Copy>
77D9B000  00004000 >RPCRT4          >.reloc     Relocations              Img  >R        RWE Copy>
77E10000  00001000 >USER32          >           PE header                Img  >R        RWE Copy>
77E11000  00052000 >USER32          >.text      Code,imports,exports     Img  >R E      RWE Copy>
77E63000  00001000 >USER32          >.data      Data                     Img  >RW       RWE Copy>
77E64000  00008000 >USER32          >.rsrc      Resources                Img  >R        RWE Copy>
77E6C000  00003000 >USER32          >.reloc     Relocations              Img  >R        RWE Copy>
77F40000  00001000 >GDI32           >           PE header                Img  >R        RWE Copy>
77F41000  00038000 >GDI32           >.text      Code,imports,exports     Img  >R E      RWE Copy>
77F79000  00001000 >GDI32           >.data      Data                     Img  >RW       RWE Copy>
77F7A000  00001000 >GDI32           >.rsrc      Resources                Img  >R        RWE Copy>
77F7B000  00002000 >GDI32           >.reloc     Relocations              Img  >R        RWE Copy>
77F80000  00001000 >ntdll           >           PE header                Img  >R        RWE Copy>
77F81000  0004D000 >ntdll           >.text,ECOD>Code,exports             Img  >R E      RWE Copy>
77FCE000  00003000 >ntdll           >.data      Data                     Img  >RW  Copy>RWE Copy>
77FD1000  00001000 >ntdll           >EDATA                               Img  >RW  Copy>RWE Copy>
77FD2000  00027000 >ntdll           >.rsrc      Resources                Img  >R        RWE Copy>
77FF9000  00003000 >ntdll           >.reloc     Relocations              Img  >R        RWE Copy>
78000000  00001000 >msvcrt          >           PE header                Img  >R        RWE Copy>
78001000  00031000 >msvcrt          >.text      Code                     Img  >R E      RWE Copy>
78032000  00008000 >msvcrt          >.rdata     Imports,exports          Img  >R        RWE Copy>
7803A000  00007000 >msvcrt          >.data      Data                     Img  >RW  Copy>RWE Copy>
78041000  00001000 >msvcrt          >.rsrc      Resources                Img  >R        RWE Copy>
78042000  00003000 >msvcrt          >.reloc     Relocations              Img  >R        RWE Copy>
7C2D0000  00001000 >ADVAPI32        >           PE header                Img  >R        RWE Copy>
7C2D1000  0005A000 >ADVAPI32        >.text      Code,imports,exports     Img  >R E      RWE Copy>
7C32B000  00004000 >ADVAPI32        >.data      Data                     Img  >RW  Copy>RWE Copy>
7C32F000  00002000 >ADVAPI32        >.rsrc      Resources                Img  >R        RWE Copy>
7C331000  00004000 >ADVAPI32        >.reloc     Relocations              Img  >R        RWE Copy>
7C340000  00001000 >Secur32         >           PE header                Img  >R        RWE Copy>
7C341000  0000B000 >Secur32         >.text      Code,imports,exports     Img  >R E      RWE Copy>
7C34C000  00001000 >Secur32         >.data      Data                     Img  >RW       RWE Copy>
7C34D000  00001000 >Secur32         >.rsrc      Resources                Img  >R        RWE Copy>
7C34E000  00001000 >Secur32         >.reloc     Relocations              Img  >R        RWE Copy>
7C570000  00001000 >KERNEL32        >           PE header                Img  >R        RWE Copy>
7C571000  0005A000 >KERNEL32        >.text      Code,imports,exports     Img  >R E      RWE Copy>
7C5CB000  00004000 >KERNEL32        >.data      Data                     Img  >RW       RWE Copy>
7C5CF000  00051000 >KERNEL32        >.rsrc      Resources                Img  >R        RWE Copy>
7C620000  00004000 >KERNEL32        >.reloc     Relocations              Img  >R        RWE Copy>
7F6F0000  00007000 >                >                                    Map  >R E      R E
7FFB0000  00024000 >                >           Code pages               Map  >R        R
7FFDE000  00001000 >                >           Data block of main threa>Priv >RWE      RWE
7FFDF000  00001000 >                >           Process Environment Bloc>Priv >RWE      RWE
7FFE0000  00001000 >                >           User Shared Data         Priv >R        R

As you can see, OS gave me address 0xd30000 which is right after end of program

00D20000  00002000 >                >                                    Map  >R        R        C:\WINNT\system32\ctype.nls

and before start of first dll

77570000  00001000 >winmm           >           PE header                Img  >R        RWE Copy>

Perhaps, you will see that there is no hole big enough to fit 805306368 bytes. Otherwise, OS is lying when returns ERROR_NOT_ENOUGH_MEMORY and there is a different reason for failure. Perhaps, it is security related.

Alex
Oct 7, 2011
#8 zippo...@gmail.com
Here is my output:
SysReserve: v=0x8fa1d0 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address failed with errno=8

And my memory map:

Memory map
Address   Size      Owner            Section    Contains                 Type  Access   Initial >Mapped as
00010000  00010000 >                >                                    Map  >RW       RW
00020000  00010000 >                >                                    Map  >RW       RW
0006D000  00001000 >                >                                    Priv >RW  Guar>RW  Guar>
0006E000  00002000 >                >           Stack of main thread     Priv >RW       RW
00070000  00004000 >                >                                    Map  >R        R
00080000  00001000 >                >                                    Priv >RW       RW
00090000  00067000 >                >                                    Map  >R        R        C:\Windows\System32\locale.nls
00100000  0000C000 >                >                                    Map  >R        R
001C0000  00003000 >                >                                    Map  >R        R
001D0000  00001000 >                >                                    Priv >RW       RW
001E0000  00001000 >                >                                    Priv >RW       RW
00200000  00003000 >                >                                    Priv >RW       RW
00230000  00006000 >                >                                    Priv >RW       RW
00400000  00001000 >8_out           >           PE header                Img  >R        RWE Copy>
00401000  000E2000 >8_out           >.text      Code                     Img  >R E      RWE Copy>
004E3000  00418000 >8_out           >.data      Data                     Img  >RW  Copy>RWE Copy>
008FB000  00001000 >8_out           >/4                                  Img  >R        RWE Copy>
008FC000  0000D000 >8_out           >/18                                 Img  >R        RWE Copy>
00909000  00009000 >8_out           >/30                                 Img  >R        RWE Copy>
00912000  00027000 >8_out           >/43                                 Img  >R        RWE Copy>
00939000  0000F000 >8_out           >/55                                 Img  >R        RWE Copy>
00948000  00003000 >8_out           >/71                                 Img  >R        RWE Copy>
0094B000  00001000 >8_out           >/87                                 Img  >R        RWE Copy>
0094C000  00001000 >8_out           >/102                                Img  >R        RWE Copy>
0094D000  00001000 >8_out           >.idata     Imports                  Img  >RW  Copy>RWE Copy>
0094E000  00001000 >8_out           >.edata     Exports                  Img  >R        RWE Copy>
0094F000  00001000 >8_out           >.symtab                             Img  >R        RWE Copy>
00950000  00101000 >                >           GDI handles              Map  >R        R
00A60000  00108000 >                >                                    Map  >R        R
0DCE0000  00001000 >KERNELBASE      >           PE header                Img  >R        RWE Copy>
0DCE1000  00043000 >KERNELBASE      >.text      Code,imports,exports     Img  >R E      RWE Copy>
0DD24000  00002000 >KERNELBASE      >.data      Data                     Img  >RW       RWE Copy>
0DD26000  00001000 >KERNELBASE      >.rsrc      Resources                Img  >R        RWE Copy>
0DD27000  00003000 >KERNELBASE      >.reloc     Relocations              Img  >R        RWE Copy>
38010000  00001000 >                >                                    Img  >R        RWE Copy>
402C0000  00001000 >LPK             >           PE header                Img  >R        RWE Copy>
402C1000  00006000 >LPK             >.text      Code,imports,exports     Img  >R E      RWE Copy>
402C7000  00001000 >LPK             >.data      Data                     Img  >RW       RWE Copy>
402C8000  00001000 >LPK             >.rsrc      Resources                Img  >R        RWE Copy>
402C9000  00001000 >LPK             >.reloc     Relocations              Img  >R        RWE Copy>
41840000  00001000 >IMM32           >           PE header                Img  >R        RWE Copy>
41841000  00017000 >IMM32           >.text      Code,imports,exports     Img  >R E      RWE Copy>
41858000  00001000 >IMM32           >.data      Data                     Img  >RW       RWE Copy>
41859000  00005000 >IMM32           >.rsrc      Resources                Img  >R        RWE Copy>
4185E000  00001000 >IMM32           >.reloc     Relocations              Img  >R        RWE Copy>
6F8E0000  00001000 >USP10           >           PE header                Img  >R        RWE Copy>
6F8E1000  0005B000 >USP10           >.text      Code,imports,exports     Img  >R E      RWE Copy>
6F93C000  00002000 >USP10           >.data      Data                     Img  >RW       RWE Copy>
6F93E000  0002A000 >USP10           >Shared                              Img  >R        RWE Copy>
6F968000  00012000 >USP10           >.rsrc      Resources                Img  >R        RWE Copy>
6F97A000  00003000 >USP10           >.reloc     Relocations              Img  >R        RWE Copy>
6FF50000  00001000 >msvcrt          >           PE header                Img  >R        RWE Copy>
6FF51000  0009F000 >msvcrt          >.text      Code,imports,exports     Img  >R E      RWE Copy>
6FFF0000  00007000 >msvcrt          >.data      Data                     Img  >RW  Copy>RWE Copy>
6FFF7000  00001000 >msvcrt          >.rsrc      Resources                Img  >R        RWE Copy>
6FFF8000  00004000 >msvcrt          >.reloc     Relocations              Img  >R        RWE Copy>
70990000  00001000 >MSCTF           >           PE header                Img  >R        RWE Copy>
70991000  00083000 >MSCTF           >.text      Code,imports,exports     Img  >R E      RWE Copy>
70A14000  00002000 >MSCTF           >.data      Data                     Img  >RW  Copy>RWE Copy>
70A16000  00041000 >MSCTF           >.rsrc      Resources                Img  >R        RWE Copy>
70A57000  00005000 >MSCTF           >.reloc     Relocations              Img  >R        RWE Copy>
77B60000  00001000 >GDI32           >           PE header                Img  >R        RWE Copy>
77B61000  00048000 >GDI32           >.text      Code,imports,exports     Img  >R E      RWE Copy>
77BA9000  00002000 >GDI32           >.data      Data                     Img  >RW       RWE Copy>
77BAB000  00001000 >GDI32           >.rsrc      Resources                Img  >R        RWE Copy>
77BAC000  00002000 >GDI32           >.reloc     Relocations              Img  >R        RWE Copy>
77D10000  00001000 >user32          >           PE header                Img  >R        RWE Copy>
77D11000  00068000 >user32          >.text      Code,imports,exports     Img  >R E      RWE Copy>
77D79000  00001000 >user32          >.data      Data                     Img  >RW       RWE Copy>
77D7A000  0005B000 >user32          >.rsrc      Resources                Img  >R        RWE Copy>
77DD5000  00004000 >user32          >.reloc     Relocations              Img  >R        RWE Copy>
77DE0000  00001000 >kernel32        >           PE header                Img  >R        RWE Copy>
77DE1000  000C5000 >kernel32        >.text      Code,imports,exports     Img  >R E      RWE Copy>
77EA6000  00001000 >kernel32        >.data      Data                     Img  >RW       RWE Copy>
77EA7000  00001000 >kernel32        >.rsrc      Resources                Img  >R        RWE Copy>
77EA8000  0000C000 >kernel32        >.reloc     Relocations              Img  >R        RWE Copy>
77EC0000  00001000 >ntdll           >           PE header                Img  >R        RWE Copy>
77EC1000  000D5000 >ntdll           >.text      Code,exports             Img  >R E      RWE Copy>
77F96000  00001000 >ntdll           >RT                                  Img  >R E      RWE Copy>
77F97000  00009000 >ntdll           >.data      Data                     Img  >RW  Copy>RWE Copy>
77FA0000  00057000 >ntdll           >.rsrc      Resources                Img  >R        RWE Copy>
77FF7000  00005000 >ntdll           >.reloc     Relocations              Img  >R        RWE Copy>
7F6F0000  00005000 >                >                                    Map  >R        R
7FFB0000  00023000 >                >           Code pages               Map  >R        R
7FFD6000  00001000 >                >           Process Environment Bloc>Priv >RW       RW
7FFDF000  00001000 >                >           Data block of main threa>Priv >RW       RW
7FFE0000  00001000 >                >           User Shared Data         Priv >R        R
80000000  7FFF0000 >                >           Kernel memory            Kern >

So you are right :P
The program ends at 0xB68000 (0xA60000 + 0x108000) and the first dll starts at 0xDCE0000. The size of the hole is 219643904 (smaller than 805306368) bytes.
Why? :/
Oct 9, 2011
Project Member #9 alex.bra...@gmail.com
I do not know how KERNELBASE.DLL happened to be in the middle of your address space. And I don't know how to control it. When I run your process on all computers I have: XP, W2K, Windows 7; none of them have KERNELBASE.DLL loaded when I stop process in runtime·SysReserve as you did. Perhaps, someone else will suggest something new.
Status: Accepted
Labels: OS-Windows
Oct 10, 2011
#10 zippo...@gmail.com
I just tried running a go program called Pokemon Universe (http://pokemon-universe.com/) and got the same error (throw: runtime: cannot reserve arena virtual address space). This game worked not long time ago, on my old computer.
So this might be a problem with my current windows installation - it can't run go programs, or more exactly, go programs using cgo.
I'm only worried because so many other applications are working perfectly. So part of the problem is with go and the other is with my windows installation.
Oct 10, 2011
Project Member #11 alex.bra...@gmail.com
I think the problem you are seeing is because KERNELBASE.DLL happend to be loaded in the middle of your address space. Your address space got fragmented, so go runtime can't allocate this big chunk of memory it uses for garbage collector.

Non-cgo programs shouldn't be affected, because go runtime loads most of its dlls after this big block of memory is allocated. So Windows just find another place for all these dlls.

Cgo programs, on the other hand, have dlls loaded during process load time, even before process started running. Dlls are listed in the pe executable file header. And Windows process loader just loads them all alongside with the executable. Loader picks whatever addresses it likes. This makes your address fragmented even before your process starts.

I am surprised, no-one reported anything like that before.

If nothing else, we would have to "reserve" this big contiguous block of addresses somehow in our pe file description. Maybe create a special pe section for it, or include it as part of existing BSS section.

Alex
Oct 11, 2011
#12 zippo...@gmail.com
Or to reinstall my windows. I tried adding a section to the PE file with the size of 805306368, but windows complained about validity. I'm (almost) sure I will be able to execute cgo programs with the right person editing the PE file, but after every compilation?

So I will reinstall windows soon, unless someone resists.
And if the problem ever appears again, I will know if it's a software that I installed or a windows setting that I changed.
Proper cgo programs ran successfully in so many windows installations (I take Pokemon Universe as an example) and nobody complained about "throw: runtime: cannot reserve arena virtual address space", so what's the chance that this problem will appear again?

Thank you Alex for exploring the problem (until actually discovering it). Seriously, without your help I was quitting go (until the world moves unix or until I format my windows and accidentally trying a cgo program).
Oct 11, 2011
Project Member #13 alex.bra...@gmail.com
Adding big pe section is not enough (even if you do it properly). You also need to make go use it during runtime.

I have a feeling we will hear more about your problem. Lets keep this issue opened.
Dec 9, 2011
Project Member #14 rsc@golang.org
(No comment was entered for this change.)
Labels: -Priority-Medium Priority-Later
Dec 12, 2011
Project Member #15 rsc@golang.org
(No comment was entered for this change.)
Labels: Priority-Go1
Jan 13, 2012
Project Member #16 r@golang.org
(No comment was entered for this change.)
Owner: buil...@golang.org
Jan 23, 2012
Project Member #17 rsc@golang.org
(No comment was entered for this change.)
Summary: runtime: out of memory on Windows
Jan 30, 2012
#18 waqas20
Some quick googling hints at kernelbase.dll being loaded in 32-bit applications running on 64-bit Windows. I don't have a 64-bit Windows install available at the moment to test.
Feb 18, 2012
Project Member #19 rsc@golang.org
This won't be fixed for Go 1.

Summary: runtime: windows DLL mappings interfere with 32-bit memory allocation
Status: HelpWanted
Labels: -Priority-Go1 Priority-Later
Apr 9, 2012
Project Member #21 a...@mgk.ro
I did some more research on this issue, particularly the issue reported in comment #20. I've talked to the author of that blog post, and cgo is NOT used.

Windows shared libraries are not relocatable, if the dynamic linker can't load it at the preferred base address specified in the PE header, it has to rebase it. Apart from slowing loading of binaries, rebasing has the side effect of not allowing that dubious benefit dynamic linking fans brag about -- it's not possible to have a single copy of a shared object mapped in every process.

Windows tries really really hard to avoid rebasing, for obvious reasons. All the system DLLs are crafted with a base address close to the 2GB mark in a way that rebasing will never be necessary. Also, by default, the compiler will generate shared objects with a base address far from the space system DLLs are using, so 3rd party DLLs will not ask for the same part of the address space as the system DLLs.

  >dumpbin /headers KernelBase.dll | findstr /C:"image base"
          7D850000 image base (7D850000 to 7D895FFF)

We can see that KernelBase.dll prefers to be loaded at 0x7D850000, but it rebased at 0x0DCE0000. This is rather unusual in the Windows world. It's very common for 3rd party DLLs to be rebased, but it's unusual for system DLLs to be rebased. Even if a 3rd party DLL asks for the same address space reservation as a system DLL, the dynamic loader will rather rebase the 3rd party DLL, and map in the process the same system DLL used by other processes.

If a system DLL was rebased, it means it was rebased in the first process of that session (a session is a Windows thing irrelevant to the discussion), and the rebased copy is shared between all processes in that session. It has to be the first process, because it's the only process the system DLL has not been loaded yet.

If someone, accidentally or not, forced a system DLL to be rebased in the first process, that someone injected a DLL in the first process where the system one wants to be. This is a way of doing user mode hooking, generally by malware, but it's also done by purposely legitimate "security" software. In fact, before Windows Vista, the only sanctioned way of doing registry monitoring, was through a user mode hook!

I don't care about "fixing" Go to run with malware, but I do care about cgo, which has the same effect (and is the purpose of this issue report anyway). We could make the address space reservation in the PE header, but it pains me that it would require a different mechanism for Windows and non-Windows. It also wouldn't be such a great idea if we ever get shared libraries, but I don't care about that crap the slightest.


Apr 10, 2012
Project Member #22 alex.bra...@gmail.com
Creating special pe section (with IMAGE_SCN_CNT_UNINITIALIZED_DATA set so it does not take up any disk space) will not work. This memory gets committed at program load as far as I can see. This is unacceptable for our purpose.

Alex
Apr 11, 2012
Project Member #23 a...@mgk.ro
If that's the case, we are out of luck.

Jun 8, 2012
#24 erics...@gmail.com
I also see this problem while running my application (it uses cgo) on a VMWare virtual machine running a fresh install of Windows XP Pro.  The virtual machine and os are 32 bit, but the host system is 64 bit running Windows 7.

I've read the discussion above, but are we moving towards a solution to this issue?  This is a pretty serious problem, one that pretty much eliminates Go as a viable option for Windows software development.
Jun 8, 2012
Project Member #25 0xE2.0x9A.0x9B@gmail.com
Go implementation contains code like this one:

   // Words outside the arena cannot be pointers.
   if((byte*)obj < arena_start || (byte*)obj >= arena_used)
       continue;

In order to fix this Windows issue, Go would need to abandon the single arena approach and use a different way of detecting whether an arbitrary pointer points to memory allocated by the Go runtime.

One possible approach would be to put the allocated regions in a tree-like structure (like paging on CPUs).
Jun 9, 2012
Project Member #26 alex.bra...@gmail.com
@ericsroy,

> ... are we moving towards a solution to this issue? ...

As far as I know, there are no proposals.

Alex
Jun 12, 2012
#27 erics...@gmail.com
Quick question -

I understand why the memory arena is a contiguous block of virtual memory (it would be significantly more complex and less efficient to spread it over several memory ranges), but why must the memory information bitmap be contiguous with the arena memory?

Rather than trying to make one huge 768mb allocation (bitmap + arena), could the memory info bitmap not be allocated elsewhere?  So that we make a 256mb allocation and then another 512mb allocation?

On the system that I have which is exhibiting this problem, the two allocations would almost certainly succeed, while the monolithic one does not.

(This question is based on what I see in the code in malloc.goc, correct me if i've misunderstood anything)

Thanks,

-Eric
Jun 13, 2012
#28 erics...@gmail.com
Here's a workaround for anyone experiencing this problem with cgo (rather than with strangely rebased system dlls):

Custom and 3rd party dlls seem to be based at 0x10000000 by default (at least if you create them with Visual Studio they are).  If you rebase your dlls to somewhere closer to the executable image, you can clear a hole in the address space that (in my case) is large enough to allow the memory allocation to succeed.

You can use Visual Studio's editbin.exe tool to rebase your dlls.  However, I can't seem to rebase dlls that have been digitally signed, d3dx9_43.dll is one example.  But if you're only using unsigned dlls then this might be an acceptable workaround.

-Eric
Jun 13, 2012
Project Member #29 a...@mgk.ro
Please not that ASLR causes DLLs to be rebased anyway.
Jun 13, 2012
#30 erics...@gmail.com
Good point, I don't think my dlls had ASLR enabled, but I also explicitly disabled it with editbin's /DYNAMICBASE option.
Jun 13, 2012
Project Member #31 alex.bra...@gmail.com
@ericsroy,

What you have to be aware of, that any non-cgo Go executable specifies to load only 2 dlls: kernel32.dll and winmm.dll. These are loaded by Windows program loader before Go executable even runs, and we have no control over that. We could, probably, get rid of winmm.dll if that is an issue, so that would leave only kernel32.dll to fight with. Any other dlls that Go program might use are loaded during run time well after gc memory is reserved, so they wouldn't affect the allocation.

On the other hand, if you use cgo - then you include code that is build with gcc and uses whatever gcc runtime provides. So, whatever other dlls are included by cgo program are out of Go control.

Also, sometimes Windows will load some dlls into your process without you asking for it. Some antivirus, firewalls, "drivers" just load some dlls into every process to do what they need to do. In that instance you are at the mercy of whoever written that software. And sometimes that software is buggy and poorly designed.

Alex
Jun 13, 2012
#32 erics...@gmail.com
@Alex,

Yes, I understand.  I've only been seeing this problem with dlls that I've introduced into my program by using cgo.  In my particular case, winmm and kernel32 are in high memory where we'd like them to be.  I haven't as of yet seen the scenario where unwanted dlls are loaded, or where system dlls are rebased to significantly lower memory locations.

For the sake of others, I just thought it was worth mentioning that naive use of even a single dll through cgo could be enough to cause the issue.

-Eric
Jun 13, 2012
Project Member #33 alex.bra...@gmail.com
@ericsroy,

> Rather than trying to make one huge 768mb allocation (bitmap + arena),
> could the memory info bitmap not be allocated elsewhere?  So that we
> make a 256mb allocation and then another 512mb allocation?

I do not know anything about how memory manager works, but I tried allocating bitmap and arena in separate / disjoint areas of memory and that fails, because some code, for example pkg/runtime/mgc0.c:1256, assumes that bitmap and arena are together in one chunk in that order. Perhaps it is easy to change all that code, but I do not know.

I could be wrong about that, so leaving for others to comment.

Alex
Sep 12, 2012
Project Member #34 rsc@golang.org
(No comment was entered for this change.)
Labels: Go1.1Maybe
Mar 7, 2013
Project Member #35 r@golang.org
(No comment was entered for this change.)
Labels: -Go1.1Maybe
May 13, 2013
#36 jack.rad...@gmail.com
How about SizeOfHeapReserve in IMAGE_OPTIONAL_HEADER ?
    The number of bytes to reserve for the local heap. Only the memory specified by the SizeOfHeapCommit
member is committed at load time; the rest is made available one page at a time until this reserve size is reached.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms680339%28v=vs.85%29.aspx
May 14, 2013
Project Member #37 alex.bra...@gmail.com
@jack.radiance, how do you propose to use that memory?

Alex
May 14, 2013
#38 jack.rad...@gmail.com
Well, you need a solid memory block to fit 0x30000000 bytes? I thought this will do.
May 14, 2013
#39 jack.rad...@gmail.com
Here http://i.imgur.com/h2UEHmy.png
I thought that if this memory range is reserved, then it will not be occupied with kernelbase
May 14, 2013
Project Member #40 alex.bra...@gmail.com
> Well, you need a solid memory block to fit 0x30000000 bytes? ...

That is too simple of a description. What we really need is to re-implement runtime·SysReserve (https://code.google.com/p/go/source/browse/src/pkg/runtime/mem_windows.c#50) and runtime·SysMap (https://code.google.com/p/go/source/browse/src/pkg/runtime/mem_windows.c#63) functions. How do you propose we do that?

Alex
Nov 27, 2013
Project Member #41 rsc@golang.org
(No comment was entered for this change.)
Labels: Go1.3Maybe
Dec 3, 2013
Project Member #42 rsc@golang.org
(No comment was entered for this change.)
Labels: -Go1.3Maybe Release-None
Dec 3, 2013
Project Member #43 rsc@golang.org
(No comment was entered for this change.)
Labels: Repo-Main
Aug 19, 2014
#44 ollie.wa...@gmail.com
This used to be a common problem for HPC applications on 32bit windows.

Adding -largeaddessaware to LDFLAGS should address this (or editbin /LARGEADDRESSAWARE).

On 32bit windows this gives an additional 1GB block of virtual address space for 32bit processes. On x64 this gives an additional 2GB address space. The library loader doesn't appear to touch this address space - keeping all dlls below the 2GB boundary - so it's a contiguous block.




Aug 19, 2014
Project Member #45 d...@cheney.net
@ollie. I'm not sure how much that would help. The problem is not that there is not enough virtual address space available to the process, but that it has been fragmented by the system (and other dll's which like to insert themselves into processes) dlls. The Go garbage collector currently requires a contiguous heap, and it is these dlls which limit the maximum size of the heap by fragmenting it.

passing /LARGEADDRESSAWARE may help, but it feels like a loosing battle.
Aug 19, 2014
Project Member #46 alex.bra...@gmail.com
@ollie.walsh, re "... Adding -largeaddessaware to LDFLAGS should address this ..." - how exactly does it work?

Alex
Sign in to add a comment

Powered by Google Project Hosting