Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FileDescriptorTables *** glibc detected *** double free or corruption (out) #194

Closed
cloudkey opened this issue Feb 4, 2015 · 5 comments
Closed

FileDescriptorTables *** glibc detected *** double free or corruption (out) #194

cloudkey opened this issue Feb 4, 2015 · 5 comments

Comments

@cloudkey
Copy link

cloudkey commented Feb 4, 2015

I use Lua to write a C++.So shared library, which contains protobuf.
My WebService is nginx, the request is dealt with the Lua, and at the end of request,write a Pb log. But when nginx worker exit, it will be core:
#0 0x000000302af2e2ed in raise () from /lib64/tls/libc.so.6
#1 0x000000302af2fa3e in abort () from /lib64/tls/libc.so.6
#2 0x000000302af62db1 in __libc_message () from /lib64/tls/libc.so.6
#3 0x000000302af6888e in int_free () from /lib64/tls/libc.so.6
#4 0x000000302af68bd6 in free () from /lib64/tls/libc.so.6
#5 0x000000302d3ae19e in operator delete(void*) () from /usr/lib64/libstdc++.so.6
#6 0x0000000000620298 in gnu_cxx::hashtable<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const>, std::pair<google::protobuf::EnumDescriptor const, int>, google::protobuf::(anonymous namespace)::PointerIntegerPairHash<std::pair<google::protobuf::EnumDescriptor const*, int> >, std::Select1st<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const> >, std::equal_to<std::pair<google::protobuf::EnumDescriptor const, int> >, std::allocator<google::protobuf::EnumValueDescriptor const*> >::clear (

this=0xa25760 <google::protobuf::FileDescriptorTables::kEmpty+160>)
at /usr/lib/gcc/x86_64-redhat-linux/3.4.5/../../../../include/c++/3.4.5/ext/new_allocator.h:86

#7 0x00000000006094e0 in ~hashtable (this=, __in_chrg=)

at /usr/lib/gcc/x86_64-redhat-linux/3.4.5/../../../../include/c++/3.4.5/ext/hashtable.h:327

#8 0x00000000006094e0 in ~hash_map (this=, __in_chrg=)
#9 0x00000000006094e0 in ~hash_map (this=0x242e, __in_chrg=)
#10 0x00000000006094e0 in google::protobuf::FileDescriptorTables::~FileDescriptorTables (this=0xa256c0 google::protobuf::FileDescriptorTables::kEmpty,

__in_chrg=<optimized out>)

#11 0x000000302af30c65 in exit () from /lib64/tls/libc.so.6
#12 0x00000000004e4420 in ngx_worker_process_exit (cycle=0xdcb1a0) at src/os/unix/ngx_process_cycle.c:1103
#13 0x00000000004e3cc6 in ngx_worker_process_cycle (cycle=0xdcb1a0, data=) at src/os/unix/ngx_process_cycle.c:844
#14 0x00000000004e1bab in ngx_spawn_process (cycle=0xdcb1a0, proc=0x4e3bcd <ngx_worker_process_cycle>, data=0x0, name=0x7da525 "worker process", respawn=-3)

at src/os/unix/ngx_process.c:198

#15 0x00000000004e33ce in ngx_start_worker_processes (cycle=0xdcb1a0, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:394
#16 0x00000000004e2dd8 in ngx_master_process_cycle (cycle=0xdcb1a0) at src/os/unix/ngx_process_cycle.c:166
#17 0x00000000004c70fe in main (argc=5, argv=) at src/core/nginx.c:412

I guess this is related to protobuf FileDescriptorTables?
@cfallin
Copy link
Contributor

cfallin commented Feb 5, 2015

It's impossible to say more without actually looking at your source. A double free is often tricky to debug because you can do something incorrectly (free an object too early) and get a stack trace only much later, on the second free.

FileDescriptorTables is supposed to free the descriptors that it holds. That is correct behavior.

Likely what is happening is that you have some code that is calling delete on a descriptor, which is incorrect. This would then cause the double-free error with the program shuts down.

Maybe try running your program under Valgrind?

@cloudkey
Copy link
Author

cloudkey commented Feb 6, 2015

thanks first!
Yesterday, I finally found out the reason, a module which embedded in our nginx contains protobuf, compiled into nginx bin. But, my Lua code calls the C++.So also uses protobuf. I found, when I put the nginx to remove the protobuf, not core.
Also, I find some global static variable in descriptor.cc~ Whether it is for this reason ?

@cloudkey
Copy link
Author

cloudkey commented Feb 6, 2015

Valgrind information:
==22518== Invalid read of size 8
==22518== at 0x620287: gnu_cxx::hashtable<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const>, std::pair<google::prot
obuf::EnumDescriptor const, int>, google::protobuf::(anonymous namespace)::PointerIntegerPairHash<std::pair<google::protobuf::EnumDescriptor const*, int> >, std::Select1st<std::pair<std::
pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const> >, std::equal_to<std::pair<google::protobuf::EnumDescriptor const_, int> >, std::allo
cator<google::protobuf::EnumValueDescriptor const*> >::clear() (stl_vector.h:462)
==22518== by 0x6094DF: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (hashtable.h:327)
==22518== by 0x302AF30C64: exit (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x4E441F: ngx_worker_process_exit (ngx_process_cycle.c:1103)
==22518== by 0x4E3CC5: ngx_worker_process_cycle (ngx_process_cycle.c:844)
==22518== by 0x4E1BAA: ngx_spawn_process (ngx_process.c:198)
==22518== by 0x4E33CD: ngx_start_worker_processes (ngx_process_cycle.c:394)
==22518== by 0x4E2DD7: ngx_master_process_cycle (ngx_process_cycle.c:166)
==22518== Address 0x4d62570 is 0 bytes inside a block of size 1,544 free'd
==22518== at 0x490796C: operator delete(void*) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30F4A: __cxa_finalize (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x678F1C2: ???
==22518== by 0x694CBAC: ???
==22518== by 0x302AD0AA5F: _dl_catch_error (in /lib64/ld-2.3.4.so)
==22518== by 0x302B201551: _dlerror_run (in /lib64/libdl-2.3.4.so)
==22518== by 0x302B2010DE: dlclose (in /lib64/libdl-2.3.4.so)
==22518==

==22518== Invalid write of size 8
==22518== at 0x6202A9: gnu_cxx::hashtable<std::pair<std::pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const>, std::pair<google::prot
obuf::EnumDescriptor const, int>, google::protobuf::(anonymous namespace)::PointerIntegerPairHash<std::pair<google::protobuf::EnumDescriptor const*, int> >, std::Select1st<std::pair<std::
pair<google::protobuf::EnumDescriptor const*, int> const, google::protobuf::EnumValueDescriptor const> >, std::equal_to<std::pair<google::protobuf::EnumDescriptor const_, int> >, std::allo
cator<google::protobuf::EnumValueDescriptor const*> >::clear() (stl_vector.h:462)
==22518== by 0x6094DF: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (hashtable.h:327)
==22518== by 0x302AF30C64: exit (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x4E441F: ngx_worker_process_exit (ngx_process_cycle.c:1103)
==22518== by 0x4E3CC5: ngx_worker_process_cycle (ngx_process_cycle.c:844)
==22518== by 0x4E1BAA: ngx_spawn_process (ngx_process.c:198)
==22518== by 0x4E33CD: ngx_start_worker_processes (ngx_process_cycle.c:394)
==22518== by 0x4E2DD7: ngx_master_process_cycle (ngx_process_cycle.c:166)
==22518== Address 0x4d62570 is 0 bytes inside a block of size 1,544 free'd
==22518== at 0x490796C: operator delete(void*) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30F4A: __cxa_finalize (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x678F1C2: ???
==22518== by 0x694CBAC: ???
==22518== by 0x302AD0AA5F: _dl_catch_error (in /lib64/ld-2.3.4.so)
==22518== by 0x302B201551: _dlerror_run (in /lib64/libdl-2.3.4.so)
==22518== by 0x302B2010DE: dlclose (in /lib64/libdl-2.3.4.so)
==22518==

==22518== Invalid free() / delete / delete[] / realloc()
==22518== at 0x490796C: operator delete(void_) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30C64: exit (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x4E441F: ngx_worker_process_exit (ngx_process_cycle.c:1103)
==22518== by 0x4E3CC5: ngx_worker_process_cycle (ngx_process_cycle.c:844)
==22518== by 0x4E1BAA: ngx_spawn_process (ngx_process.c:198)
==22518== by 0x4E33CD: ngx_start_worker_processes (ngx_process_cycle.c:394)
==22518== by 0x4E2DD7: ngx_master_process_cycle (ngx_process_cycle.c:166)
==22518== Address 0x4d62570 is 0 bytes inside a block of size 1,544 free'd
==22518== at 0x490796C: operator delete(void_) (vg_replace_malloc.c:480)
==22518== by 0x6094F0: google::protobuf::FileDescriptorTables::~FileDescriptorTables() (new_allocator.h:86)
==22518== by 0x302AF30F4A: __cxa_finalize (in /lib64/tls/libc-2.3.4.so)
==22518== by 0x678F1C2: ???
==22518== by 0x694CBAC: ???
==22518== by 0x302AD0AA5F: _dl_catch_error (in /lib64/ld-2.3.4.so)
==22518== by 0x302B201551: _dlerror_run (in /lib64/libdl-2.3.4.so)
==22518== by 0x302B2010DE: dlclose (in /lib64/libdl-2.3.4.so)
==22518==

@cfallin
Copy link
Contributor

cfallin commented Feb 6, 2015

Yes, there are some global variables, for example the "generated descriptor pool". But just using global variables doesn't make a program double-free. Do you have more specific theories or evidence?

You haven't given us any information on what your program is doing. There are many ways for a C++ program to corrupt memory, usually due to some API misuse, stray pointer, or lifetime issue. If you can give us a program that will reproduce this error, we will try to look into it. But right now there is not enough information to diagnose further. The fact that the final crash happens during FileDescriptorTables cleanup does not mean that FileDescriptorTables is doing something wrong (again, as I said before: it is correctly delete-ing its memory), only that something else in your program has corrupted memory.

@drubinstein drubinstein mentioned this issue Nov 8, 2015
Closed
@rinatz rinatz mentioned this issue Dec 18, 2015
Merged
@rinatz
Copy link
Contributor

rinatz commented Dec 19, 2015

@cloudkey I sent the pull request to fix this problem and it was merged.

@xfxyjwf xfxyjwf closed this as completed Jan 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cfallin @cloudkey @xfxyjwf @rinatz