runtime: windows DLL mappings interfere with 32-bit memory allocation #2323
Labels
Milestone
Comments
I can't reproduce your problem on Windows 7:
$ cat Makefile
include $(GOROOT)/src/Make.inc
TARG=a
CGOFILES=\
a.go\
include $(GOROOT)/src/Make.pkg
$ cat a.go
package a
/*
#include <windows.h>
int xPlease() {
POINT p;
GetCursorPos(&p);
return p.x;
}
*/
import "C"
func Now() {
println("I got that. It's: ", int(C.xPlease()))
}
$ cat a_test.go
package a_test
import (
"a"
"testing"
)
func TestA(t *testing.T) {
a.Now()
}
$ make test
gotest
rm -f _test/a.a
CGOPKGPATH= cgo -- a.go
touch _obj/_cgo_run
8g -o _gotest_.8 _obj/a.cgo1.go _obj/_cgo_gotypes.go
8c -FVw -Ic:/MinGW/go/pkg/windows_386 -I . -o "_cgo_defun.8" _obj/_cgo_defun.c
gcc -m32 -I . -g -fPIC -O2 -o _cgo_main.o -c _obj/_cgo_main.c
_obj/_cgo_main.c:1:0: warning: -fPIC ignored for target (all code is position
independent)
gcc -m32 -I . -g -fPIC -O2 -o a.cgo2.o -c _obj/a.cgo2.c
_obj/a.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position independent)
gcc -m32 -I . -g -fPIC -O2 -o _cgo_export.o -c _obj/_cgo_export.c
_obj/_cgo_export.c:1:0: warning: -fPIC ignored for target (all code is position
independent)
gcc -m32 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o a.cgo2.o _cgo_export.o
cgo -dynimport _cgo1_.o >_obj/_cgo_import.c_ && mv -f _obj/_cgo_import.c_
_obj/_cgo_import.c
8c -FVw -I . -o "_cgo_import.8" _obj/_cgo_import.c
rm -f _test/a.a
gopack grc _test/a.a _gotest_.8 _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export.o
I got that. It's: 1115
PASS
$
Owner changed to @alexbrainman. Status changed to WaitingForReply. |
I have same files and I typed the same commands shown in your comment, but I get: $ make test gotest rm -f _test/a.a CGOPKGPATH= cgo -- a.go touch _obj/_cgo_run 8g -o _gotest_.8 _obj/a.cgo1.go _obj/_cgo_gotypes.go 8c -FVw -IC:/Go/pkg/windows_386 -I . -o "_cgo_defun.8" _obj/_cgo_defun.c gcc -m32 -I . -g -fPIC -O2 -o _cgo_main.o -c _obj/_cgo_main.c _obj/_cgo_main.c:1:0: warning: -fPIC ignored for target (all code is position in dependent) gcc -m32 -I . -g -fPIC -O2 -o a.cgo2.o -c _obj/a.cgo2.c _obj/a.cgo2.c:1:0: warning: -fPIC ignored for target (all code is position indep endent) gcc -m32 -I . -g -fPIC -O2 -o _cgo_export.o -c _obj/_cgo_export.c _obj/_cgo_export.c:1:0: warning: -fPIC ignored for target (all code is position independent) gcc -m32 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o a.cgo2.o _cgo_export.o cgo -dynimport _cgo1_.o >_obj/_cgo_import.c_ && mv -f _obj/_cgo_import.c_ _obj/_ cgo_import.c 8c -FVw -I . -o "_cgo_import.8" _obj/_cgo_import.c rm -f _test/a.a gopack grc _test/a.a _gotest_.8 _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export .o throw: runtime: cannot reserve arena virtual address space gotest: "./8.out.exe" failed: exit status 2 make: *** [test] Error 2 =================== Maybe cgo doesn't use the right C++ compiler? (I don't really understand these things, I just installed mingw with it's compiler). My UAC (security settings) are turned off, btw. |
I do not what your problem is. But before we do anything else, maybe you can tell us why
this error is happening. If you could change your $GOROOT/src/pkg/runtime/windows/mem.c
file like this:
diff -r 6aa111bd2d11 src/pkg/runtime/windows/mem.c
--- a/src/pkg/runtime/windows/mem.c Tue Oct 04 15:07:28 2011 +1100
+++ b/src/pkg/runtime/windows/mem.c Wed Oct 05 14:21:57 2011 +1100
@@ -55,7 +55,12 @@
return v;
// Next let the kernel choose the address.
- return runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+ v = runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+ if(v == nil) {
+ runtime·printf("VirtualAlloc failed with errno=%d\n", runtime·getlasterror());
+ runtime·throw("VirtualAlloc");
+ }
+ return v;
}
void
Then you would need to rebuild "runtime" package, like
cd $GOROOT/src/pkg/runtime
make clean install
Then re-run your program and tell us what errno VirtualAlloc function returns. Thank you.
Alex
|
errno=8 is ERROR_NOT_ENOUGH_MEMORY, so it sounds like your program fails to allocate
required block of address space, perhaps, because there is not enough room to do it.
I take it, some simple program like:
package main
func main() {
println("Hello")
}
works. Right?
Could you, please revert change to your $GOROOT/src/pkg/runtime/windows/mem.c file, and
apply this one, instead:
diff -r 6aa111bd2d11 src/pkg/runtime/windows/mem.c
--- a/src/pkg/runtime/windows/mem.c Tue Oct 04 15:07:28 2011 +1100
+++ b/src/pkg/runtime/windows/mem.c Thu Oct 06 12:21:13 2011 +1100
@@ -48,14 +48,25 @@
void*
runtime·SysReserve(void *v, uintptr n)
{
+ runtime·printf("SysReserve: v=%p n=%d\n", v, n);
// v is just a hint.
// First try at v.
v = runtime·stdcall(runtime·VirtualAlloc, 4, v, n, (uintptr)MEM_RESERVE, (uintptr)PAGE_EXECUTE_READWRITE);
- if(v != nil)
+ if(v != nil) {
+ runtime·printf("SysReserve: hint worked v=%p\n", v);
return v;
+ } else {
+ runtime·printf("SysReserve: hint failed with errno=%d\n", runtime·getlasterror());
+ }
// Next let the kernel choose the address.
- return runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+ v = runtime·stdcall(runtime·VirtualAlloc, 4, nil, n, (uintptr)MEM_RESERVE,
(uintptr)PAGE_EXECUTE_READWRITE);
+ if(v == nil) {
+ runtime·printf("SysReserve: accepting any address failed with errno=%d\n",
runtime·getlasterror());
+ runtime·throw("VirtualAlloc");
+ }
+ runtime·printf("SysReserve: accepting any address worked v=%p\n", v);
+ return v;
}
void
Then re-run you test again and show us the output. Please, also run little program above
and send its output too. I would like to compare the two.
I still do not know what is going on, perhaps, when you link extra mingw windows
functions, they use some extra address space, maybe they load some extra dlls, that we
have not enough room to move afterwards.
Thank you.
Alex
|
Printing "Hello" works of course, or I wasn't moving on to Windows API :P I have 3.49GB usable memory and I successfully called Windows API with C++ (using mingw gcc). Okay I applied your patch and here is the output for the test: $ make test gotest SysReserve: v=0x920118 n=805306368 SysReserve: hint failed with errno=487 SysReserve: accepting any address worked v=0x38020000 rm -f _test/a.a rm -f _test/a.a gopack grc _test/a.a _gotest_.8 _cgo_defun.8 _cgo_import.8 a.cgo2.o _cgo_export .o SysReserve: v=0x8fa1e0 n=805306368 SysReserve: hint failed with errno=487 SysReserve: accepting any address failed with errno=8 throw: VirtualAlloc gotest: "./8.out.exe" failed: exit status 2 make: *** [test] Error 2 And for the little program: $ 8.out SysReserve: v=0x832e58 n=805306368 SysReserve: hint failed with errno=487 SysReserve: accepting any address worked v=0x38020000 Hello |
I am running out of ideas. :-)
The problem here is not the amount of memory you have in your computer, but free block
of address space available in your process. Go memory manager is designed to use 1 big
block of address space (see that request for 805306368 bytes). It asks OS to reserve
single contiguous block of addresses for future use. No real memory is associated with
these addresses at the start, but will get assigned as addresses are used by the
program. The problem here is that OS refuses to allocate this 805306368 bytes block. I
don't know why. Perhaps, there is no contiguous block of such size at the time, because
memory is used by other parts of your process (dlls and such).
What I would do next is put a bug in mem.c: replace line with
"runtime·throw("VirtualAlloc");" with "*(int32*)0 = 0;". This will raise an exception
at the right moment in your program. If you run it under debugger (I use
http://www.ollydbg.de/), you should be able to see memory map of your process, this will
give you a clue if there is free block of addresses or not and why. Here is my run:
SysReserve: v=0xa00000 n=805306368
SysReserve: hint failed with errno=487
SysReserve: accepting any address worked v=0xd30000
here is my map:
Memory map
Address Size Owner Section Contains Type Access
Initial >Mapped as
00010000 00001000 > > Environment Priv >RW
RW
00020000 00001000 > > Process Parameters Priv >RW
RW
0005F000 00001000 > > Priv >RW
Guar>RW Guar>
00060000 00010000 > > Stack of main thread Priv >RW
RW
00070000 00006000 > > Priv >RW
RW
00170000 00003000 > > Map >RW
RW
00180000 00016000 > > Map >R
R C:\WINNT\system32\unicode.nls
001A0000 0002F000 > > Map >R
R C:\WINNT\system32\locale.nls
001D0000 00041000 > > Map >R
R C:\WINNT\system32\sortkey.nls
00220000 00004000 > > Map >R
R C:\WINNT\system32\sorttbls.nls
00230000 00008000 > > Map >R E
R E
002F0000 00002000 > > Map >R E
R E
00300000 00043000 > > GDI handles Map >R R
00400000 00001000 >8_out > PE header Img >R
RWE Copy>
00401000 000E9000 >8_out >.text Code Img >R E
RWE Copy>
004EA000 00421000 >8_out >.data Data Img >RW
Copy>RWE Copy>
0090B000 00001000 >8_out >/4 Img >R
RWE Copy>
0090C000 0000E000 >8_out >/18 Img >R
RWE Copy>
0091A000 0000A000 >8_out >/30 Img >R
RWE Copy>
00924000 0002B000 >8_out >/43 Img >R
RWE Copy>
0094F000 00010000 >8_out >/55 Img >R
RWE Copy>
0095F000 00003000 >8_out >/71 Img >R
RWE Copy>
00962000 00001000 >8_out >/87 Img >R
RWE Copy>
00963000 00001000 >8_out >/102 Img >R
RWE Copy>
00964000 00001000 >8_out >.idata Imports Img >RW
Copy>RWE Copy>
00965000 00001000 >8_out >.edata Exports Img >R
RWE Copy>
00966000 00001000 >8_out >.symtab Img >R
RWE Copy>
00970000 000B5000 > > Map >R E
R E
00C70000 00001000 > > Priv >RW
RW
00C80000 00001000 > > Priv >RW
RW
00C90000 00001000 > > Priv >RW
RW
00D10000 00005000 > > Priv >RW
RW
00D20000 00002000 > > Map >R
R C:\WINNT\system32\ctype.nls
77570000 00001000 >winmm > PE header Img >R
RWE Copy>
77571000 0001F000 >winmm >.text Code,imports,exports Img >R E
RWE Copy>
77590000 00005000 >winmm >.data Data Img >RW
Copy>RWE Copy>
77595000 00009000 >winmm >.rsrc Resources Img >R
RWE Copy>
7759E000 00002000 >winmm >.reloc Relocations Img >R
RWE Copy>
77D30000 00001000 >RPCRT4 > PE header Img >R
RWE Copy>
77D31000 00067000 >RPCRT4 >.text,.orp>Code,imports,exports Img >R E
RWE Copy>
77D98000 00002000 >RPCRT4 >.data Data Img >RW
RWE Copy>
77D9A000 00001000 >RPCRT4 >.rsrc Resources Img >R
RWE Copy>
77D9B000 00004000 >RPCRT4 >.reloc Relocations Img >R
RWE Copy>
77E10000 00001000 >USER32 > PE header Img >R
RWE Copy>
77E11000 00052000 >USER32 >.text Code,imports,exports Img >R E
RWE Copy>
77E63000 00001000 >USER32 >.data Data Img >RW
RWE Copy>
77E64000 00008000 >USER32 >.rsrc Resources Img >R
RWE Copy>
77E6C000 00003000 >USER32 >.reloc Relocations Img >R
RWE Copy>
77F40000 00001000 >GDI32 > PE header Img >R
RWE Copy>
77F41000 00038000 >GDI32 >.text Code,imports,exports Img >R E
RWE Copy>
77F79000 00001000 >GDI32 >.data Data Img >RW
RWE Copy>
77F7A000 00001000 >GDI32 >.rsrc Resources Img >R
RWE Copy>
77F7B000 00002000 >GDI32 >.reloc Relocations Img >R
RWE Copy>
77F80000 00001000 >ntdll > PE header Img >R
RWE Copy>
77F81000 0004D000 >ntdll >.text,ECOD>Code,exports Img >R E
RWE Copy>
77FCE000 00003000 >ntdll >.data Data Img >RW
Copy>RWE Copy>
77FD1000 00001000 >ntdll >EDATA Img >RW
Copy>RWE Copy>
77FD2000 00027000 >ntdll >.rsrc Resources Img >R
RWE Copy>
77FF9000 00003000 >ntdll >.reloc Relocations Img >R
RWE Copy>
78000000 00001000 >msvcrt > PE header Img >R
RWE Copy>
78001000 00031000 >msvcrt >.text Code Img >R E
RWE Copy>
78032000 00008000 >msvcrt >.rdata Imports,exports Img >R
RWE Copy>
7803A000 00007000 >msvcrt >.data Data Img >RW
Copy>RWE Copy>
78041000 00001000 >msvcrt >.rsrc Resources Img >R
RWE Copy>
78042000 00003000 >msvcrt >.reloc Relocations Img >R
RWE Copy>
7C2D0000 00001000 >ADVAPI32 > PE header Img >R
RWE Copy>
7C2D1000 0005A000 >ADVAPI32 >.text Code,imports,exports Img >R E
RWE Copy>
7C32B000 00004000 >ADVAPI32 >.data Data Img >RW
Copy>RWE Copy>
7C32F000 00002000 >ADVAPI32 >.rsrc Resources Img >R
RWE Copy>
7C331000 00004000 >ADVAPI32 >.reloc Relocations Img >R
RWE Copy>
7C340000 00001000 >Secur32 > PE header Img >R
RWE Copy>
7C341000 0000B000 >Secur32 >.text Code,imports,exports Img >R E
RWE Copy>
7C34C000 00001000 >Secur32 >.data Data Img >RW
RWE Copy>
7C34D000 00001000 >Secur32 >.rsrc Resources Img >R
RWE Copy>
7C34E000 00001000 >Secur32 >.reloc Relocations Img >R
RWE Copy>
7C570000 00001000 >KERNEL32 > PE header Img >R
RWE Copy>
7C571000 0005A000 >KERNEL32 >.text Code,imports,exports Img >R E
RWE Copy>
7C5CB000 00004000 >KERNEL32 >.data Data Img >RW
RWE Copy>
7C5CF000 00051000 >KERNEL32 >.rsrc Resources Img >R
RWE Copy>
7C620000 00004000 >KERNEL32 >.reloc Relocations Img >R
RWE Copy>
7F6F0000 00007000 > > Map >R E
R E
7FFB0000 00024000 > > Code pages Map >R R
7FFDE000 00001000 > > Data block of main threa>Priv >RWE
RWE
7FFDF000 00001000 > > Process Environment Bloc>Priv >RWE
RWE
7FFE0000 00001000 > > User Shared Data Priv >R R
As you can see, OS gave me address 0xd30000 which is right after end of program
00D20000 00002000 > > Map >R
R C:\WINNT\system32\ctype.nls
and before start of first dll
77570000 00001000 >winmm > PE header Img >R
RWE Copy>
Perhaps, you will see that there is no hole big enough to fit 805306368 bytes.
Otherwise, OS is lying when returns ERROR_NOT_ENOUGH_MEMORY and there is a different
reason for failure. Perhaps, it is security related.
Alex
|
Here is my output: SysReserve: v=0x8fa1d0 n=805306368 SysReserve: hint failed with errno=487 SysReserve: accepting any address failed with errno=8 And my memory map: Memory map Address Size Owner Section Contains Type Access Initial >Mapped as 00010000 00010000 > > Map >RW RW 00020000 00010000 > > Map >RW RW 0006D000 00001000 > > Priv >RW Guar>RW Guar> 0006E000 00002000 > > Stack of main thread Priv >RW RW 00070000 00004000 > > Map >R R 00080000 00001000 > > Priv >RW RW 00090000 00067000 > > Map >R R C:\Windows\System32\locale.nls 00100000 0000C000 > > Map >R R 001C0000 00003000 > > Map >R R 001D0000 00001000 > > Priv >RW RW 001E0000 00001000 > > Priv >RW RW 00200000 00003000 > > Priv >RW RW 00230000 00006000 > > Priv >RW RW 00400000 00001000 >8_out > PE header Img >R RWE Copy> 00401000 000E2000 >8_out >.text Code Img >R E RWE Copy> 004E3000 00418000 >8_out >.data Data Img >RW Copy>RWE Copy> 008FB000 00001000 >8_out >/4 Img >R RWE Copy> 008FC000 0000D000 >8_out >/18 Img >R RWE Copy> 00909000 00009000 >8_out >/30 Img >R RWE Copy> 00912000 00027000 >8_out >/43 Img >R RWE Copy> 00939000 0000F000 >8_out >/55 Img >R RWE Copy> 00948000 00003000 >8_out >/71 Img >R RWE Copy> 0094B000 00001000 >8_out >/87 Img >R RWE Copy> 0094C000 00001000 >8_out >/102 Img >R RWE Copy> 0094D000 00001000 >8_out >.idata Imports Img >RW Copy>RWE Copy> 0094E000 00001000 >8_out >.edata Exports Img >R RWE Copy> 0094F000 00001000 >8_out >.symtab Img >R RWE Copy> 00950000 00101000 > > GDI handles Map >R R 00A60000 00108000 > > Map >R R 0DCE0000 00001000 >KERNELBASE > PE header Img >R RWE Copy> 0DCE1000 00043000 >KERNELBASE >.text Code,imports,exports Img >R E RWE Copy> 0DD24000 00002000 >KERNELBASE >.data Data Img >RW RWE Copy> 0DD26000 00001000 >KERNELBASE >.rsrc Resources Img >R RWE Copy> 0DD27000 00003000 >KERNELBASE >.reloc Relocations Img >R RWE Copy> 38010000 00001000 > > Img >R RWE Copy> 402C0000 00001000 >LPK > PE header Img >R RWE Copy> 402C1000 00006000 >LPK >.text Code,imports,exports Img >R E RWE Copy> 402C7000 00001000 >LPK >.data Data Img >RW RWE Copy> 402C8000 00001000 >LPK >.rsrc Resources Img >R RWE Copy> 402C9000 00001000 >LPK >.reloc Relocations Img >R RWE Copy> 41840000 00001000 >IMM32 > PE header Img >R RWE Copy> 41841000 00017000 >IMM32 >.text Code,imports,exports Img >R E RWE Copy> 41858000 00001000 >IMM32 >.data Data Img >RW RWE Copy> 41859000 00005000 >IMM32 >.rsrc Resources Img >R RWE Copy> 4185E000 00001000 >IMM32 >.reloc Relocations Img >R RWE Copy> 6F8E0000 00001000 >USP10 > PE header Img >R RWE Copy> 6F8E1000 0005B000 >USP10 >.text Code,imports,exports Img >R E RWE Copy> 6F93C000 00002000 >USP10 >.data Data Img >RW RWE Copy> 6F93E000 0002A000 >USP10 >Shared Img >R RWE Copy> 6F968000 00012000 >USP10 >.rsrc Resources Img >R RWE Copy> 6F97A000 00003000 >USP10 >.reloc Relocations Img >R RWE Copy> 6FF50000 00001000 >msvcrt > PE header Img >R RWE Copy> 6FF51000 0009F000 >msvcrt >.text Code,imports,exports Img >R E RWE Copy> 6FFF0000 00007000 >msvcrt >.data Data Img >RW Copy>RWE Copy> 6FFF7000 00001000 >msvcrt >.rsrc Resources Img >R RWE Copy> 6FFF8000 00004000 >msvcrt >.reloc Relocations Img >R RWE Copy> 70990000 00001000 >MSCTF > PE header Img >R RWE Copy> 70991000 00083000 >MSCTF >.text Code,imports,exports Img >R E RWE Copy> 70A14000 00002000 >MSCTF >.data Data Img >RW Copy>RWE Copy> 70A16000 00041000 >MSCTF >.rsrc Resources Img >R RWE Copy> 70A57000 00005000 >MSCTF >.reloc Relocations Img >R RWE Copy> 77B60000 00001000 >GDI32 > PE header Img >R RWE Copy> 77B61000 00048000 >GDI32 >.text Code,imports,exports Img >R E RWE Copy> 77BA9000 00002000 >GDI32 >.data Data Img >RW RWE Copy> 77BAB000 00001000 >GDI32 >.rsrc Resources Img >R RWE Copy> 77BAC000 00002000 >GDI32 >.reloc Relocations Img >R RWE Copy> 77D10000 00001000 >user32 > PE header Img >R RWE Copy> 77D11000 00068000 >user32 >.text Code,imports,exports Img >R E RWE Copy> 77D79000 00001000 >user32 >.data Data Img >RW RWE Copy> 77D7A000 0005B000 >user32 >.rsrc Resources Img >R RWE Copy> 77DD5000 00004000 >user32 >.reloc Relocations Img >R RWE Copy> 77DE0000 00001000 >kernel32 > PE header Img >R RWE Copy> 77DE1000 000C5000 >kernel32 >.text Code,imports,exports Img >R E RWE Copy> 77EA6000 00001000 >kernel32 >.data Data Img >RW RWE Copy> 77EA7000 00001000 >kernel32 >.rsrc Resources Img >R RWE Copy> 77EA8000 0000C000 >kernel32 >.reloc Relocations Img >R RWE Copy> 77EC0000 00001000 >ntdll > PE header Img >R RWE Copy> 77EC1000 000D5000 >ntdll >.text Code,exports Img >R E RWE Copy> 77F96000 00001000 >ntdll >RT Img >R E RWE Copy> 77F97000 00009000 >ntdll >.data Data Img >RW Copy>RWE Copy> 77FA0000 00057000 >ntdll >.rsrc Resources Img >R RWE Copy> 77FF7000 00005000 >ntdll >.reloc Relocations Img >R RWE Copy> 7F6F0000 00005000 > > Map >R R 7FFB0000 00023000 > > Code pages Map >R R 7FFD6000 00001000 > > Process Environment Bloc>Priv >RW RW 7FFDF000 00001000 > > Data block of main threa>Priv >RW RW 7FFE0000 00001000 > > User Shared Data Priv >R R 80000000 7FFF0000 > > Kernel memory Kern > So you are right :P The program ends at 0xB68000 (0xA60000 + 0x108000) and the first dll starts at 0xDCE0000. The size of the hole is 219643904 (smaller than 805306368) bytes. Why? :/ |
I do not know how KERNELBASE.DLL happened to be in the middle of your address space. And I don't know how to control it. When I run your process on all computers I have: XP, W2K, Windows 7; none of them have KERNELBASE.DLL loaded when I stop process in runtime·SysReserve as you did. Perhaps, someone else will suggest something new. Labels changed: added os-windows. Status changed to Accepted. |
I just tried running a go program called Pokemon Universe (http://pokemon-universe.com/) and got the same error (throw: runtime: cannot reserve arena virtual address space). This game worked not long time ago, on my old computer. So this might be a problem with my current windows installation - it can't run go programs, or more exactly, go programs using cgo. I'm only worried because so many other applications are working perfectly. So part of the problem is with go and the other is with my windows installation. |
I think the problem you are seeing is because KERNELBASE.DLL happend to be loaded in the middle of your address space. Your address space got fragmented, so go runtime can't allocate this big chunk of memory it uses for garbage collector. Non-cgo programs shouldn't be affected, because go runtime loads most of its dlls after this big block of memory is allocated. So Windows just find another place for all these dlls. Cgo programs, on the other hand, have dlls loaded during process load time, even before process started running. Dlls are listed in the pe executable file header. And Windows process loader just loads them all alongside with the executable. Loader picks whatever addresses it likes. This makes your address fragmented even before your process starts. I am surprised, no-one reported anything like that before. If nothing else, we would have to "reserve" this big contiguous block of addresses somehow in our pe file description. Maybe create a special pe section for it, or include it as part of existing BSS section. Alex |
Or to reinstall my windows. I tried adding a section to the PE file with the size of 805306368, but windows complained about validity. I'm (almost) sure I will be able to execute cgo programs with the right person editing the PE file, but after every compilation? So I will reinstall windows soon, unless someone resists. And if the problem ever appears again, I will know if it's a software that I installed or a windows setting that I changed. Proper cgo programs ran successfully in so many windows installations (I take Pokemon Universe as an example) and nobody complained about "throw: runtime: cannot reserve arena virtual address space", so what's the chance that this problem will appear again? Thank you Alex for exploring the problem (until actually discovering it). Seriously, without your help I was quitting go (until the world moves unix or until I format my windows and accidentally trying a cgo program). |
|
Owner changed to builder@golang.org. |
I did some more research on this issue, particularly the issue reported in comment #20.
I've talked to the author of that blog post, and cgo is NOT used.
Windows shared libraries are not relocatable, if the dynamic linker can't load it at the
preferred base address specified in the PE header, it has to rebase it. Apart from
slowing loading of binaries, rebasing has the side effect of not allowing that dubious
benefit dynamic linking fans brag about -- it's not possible to have a single copy of a
shared object mapped in every process.
Windows tries really really hard to avoid rebasing, for obvious reasons. All the system
DLLs are crafted with a base address close to the 2GB mark in a way that rebasing will
never be necessary. Also, by default, the compiler will generate shared objects with a
base address far from the space system DLLs are using, so 3rd party DLLs will not ask
for the same part of the address space as the system DLLs.
>dumpbin /headers KernelBase.dll | findstr /C:"image base"
7D850000 image base (7D850000 to 7D895FFF)
We can see that KernelBase.dll prefers to be loaded at 0x7D850000, but it rebased at
0x0DCE0000. This is rather unusual in the Windows world. It's very common for 3rd party
DLLs to be rebased, but it's unusual for system DLLs to be rebased. Even if a 3rd party
DLL asks for the same address space reservation as a system DLL, the dynamic loader will
rather rebase the 3rd party DLL, and map in the process the same system DLL used by
other processes.
If a system DLL was rebased, it means it was rebased in the first process of that
session (a session is a Windows thing irrelevant to the discussion), and the rebased
copy is shared between all processes in that session. It has to be the first process,
because it's the only process the system DLL has not been loaded yet.
If someone, accidentally or not, forced a system DLL to be rebased in the first process,
that someone injected a DLL in the first process where the system one wants to be. This
is a way of doing user mode hooking, generally by malware, but it's also done by
purposely legitimate "security" software. In fact, before Windows Vista, the only
sanctioned way of doing registry monitoring, was through a user mode hook!
I don't care about "fixing" Go to run with malware, but I do care about cgo, which has
the same effect (and is the purpose of this issue report anyway). We could make the
address space reservation in the PE header, but it pains me that it would require a
different mechanism for Windows and non-Windows. It also wouldn't be such a great idea
if we ever get shared libraries, but I don't care about that crap the slightest.
|
I also see this problem while running my application (it uses cgo) on a VMWare virtual machine running a fresh install of Windows XP Pro. The virtual machine and os are 32 bit, but the host system is 64 bit running Windows 7. I've read the discussion above, but are we moving towards a solution to this issue? This is a pretty serious problem, one that pretty much eliminates Go as a viable option for Windows software development. |
Go implementation contains code like this one:
// Words outside the arena cannot be pointers.
if((byte*)obj < arena_start || (byte*)obj >= arena_used)
continue;
In order to fix this Windows issue, Go would need to abandon the single arena approach
and use a different way of detecting whether an arbitrary pointer points to memory
allocated by the Go runtime.
One possible approach would be to put the allocated regions in a tree-like structure
(like paging on CPUs).
|
Quick question - I understand why the memory arena is a contiguous block of virtual memory (it would be significantly more complex and less efficient to spread it over several memory ranges), but why must the memory information bitmap be contiguous with the arena memory? Rather than trying to make one huge 768mb allocation (bitmap + arena), could the memory info bitmap not be allocated elsewhere? So that we make a 256mb allocation and then another 512mb allocation? On the system that I have which is exhibiting this problem, the two allocations would almost certainly succeed, while the monolithic one does not. (This question is based on what I see in the code in malloc.goc, correct me if i've misunderstood anything) Thanks, -Eric |
Here's a workaround for anyone experiencing this problem with cgo (rather than with strangely rebased system dlls): Custom and 3rd party dlls seem to be based at 0x10000000 by default (at least if you create them with Visual Studio they are). If you rebase your dlls to somewhere closer to the executable image, you can clear a hole in the address space that (in my case) is large enough to allow the memory allocation to succeed. You can use Visual Studio's editbin.exe tool to rebase your dlls. However, I can't seem to rebase dlls that have been digitally signed, d3dx9_43.dll is one example. But if you're only using unsigned dlls then this might be an acceptable workaround. -Eric |
@ericsroy, What you have to be aware of, that any non-cgo Go executable specifies to load only 2 dlls: kernel32.dll and winmm.dll. These are loaded by Windows program loader before Go executable even runs, and we have no control over that. We could, probably, get rid of winmm.dll if that is an issue, so that would leave only kernel32.dll to fight with. Any other dlls that Go program might use are loaded during run time well after gc memory is reserved, so they wouldn't affect the allocation. On the other hand, if you use cgo - then you include code that is build with gcc and uses whatever gcc runtime provides. So, whatever other dlls are included by cgo program are out of Go control. Also, sometimes Windows will load some dlls into your process without you asking for it. Some antivirus, firewalls, "drivers" just load some dlls into every process to do what they need to do. In that instance you are at the mercy of whoever written that software. And sometimes that software is buggy and poorly designed. Alex |
@Alex, Yes, I understand. I've only been seeing this problem with dlls that I've introduced into my program by using cgo. In my particular case, winmm and kernel32 are in high memory where we'd like them to be. I haven't as of yet seen the scenario where unwanted dlls are loaded, or where system dlls are rebased to significantly lower memory locations. For the sake of others, I just thought it was worth mentioning that naive use of even a single dll through cgo could be enough to cause the issue. -Eric |
@ericsroy, > Rather than trying to make one huge 768mb allocation (bitmap + arena), > could the memory info bitmap not be allocated elsewhere? So that we > make a 256mb allocation and then another 512mb allocation? I do not know anything about how memory manager works, but I tried allocating bitmap and arena in separate / disjoint areas of memory and that fails, because some code, for example pkg/runtime/mgc0.c:1256, assumes that bitmap and arena are together in one chunk in that order. Perhaps it is easy to change all that code, but I do not know. I could be wrong about that, so leaving for others to comment. Alex |
How about SizeOfHeapReserve in IMAGE_OPTIONAL_HEADER ?
The number of bytes to reserve for the local heap. Only the memory specified by the SizeOfHeapCommit
member is committed at load time; the rest is made available one page at a time until
this reserve size is reached.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms680339%28v=vs.85%29.aspx
|
Here http://i.imgur.com/h2UEHmy.png I thought that if this memory range is reserved, then it will not be occupied with kernelbase |
> Well, you need a solid memory block to fit 0x30000000 bytes? ... That is too simple of a description. What we really need is to re-implement runtime·SysReserve (http://code.google.com/p/go/source/browse/src/pkg/runtime/mem_windows.c#50) and runtime·SysMap (http://code.google.com/p/go/source/browse/src/pkg/runtime/mem_windows.c#63) functions. How do you propose we do that? Alex |
This used to be a common problem for HPC applications on 32bit windows. Adding -largeaddessaware to LDFLAGS should address this (or editbin /LARGEADDRESSAWARE). On 32bit windows this gives an additional 1GB block of virtual address space for 32bit processes. On x64 this gives an additional 2GB address space. The library loader doesn't appear to touch this address space - keeping all dlls below the 2GB boundary - so it's a contiguous block. |
@ollie. I'm not sure how much that would help. The problem is not that there is not enough virtual address space available to the process, but that it has been fragmented by the system (and other dll's which like to insert themselves into processes) dlls. The Go garbage collector currently requires a contiguous heap, and it is these dlls which limit the maximum size of the heap by fragmenting it. passing /LARGEADDRESSAWARE may help, but it feels like a loosing battle. |
|
I tried this. not reproduced. prefer closing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
by zippoxer:
Attachments:
The text was updated successfully, but these errors were encountered: