My favorites | Sign in
Project Home Issues
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 9713: ndk-gdb crash trying to debug NDK sample "san-angeles"
59 people starred this issue and may be notified of changes. Back to list
Status:  Released
Owner:  digit%an...@gtempaccount.com
Closed:  Feb 2011


Sign in to add a comment
 
Reported by asad...@gmail.com, Jul 12, 2010
Host OS: Windows XP + Cygwin 1.7.4(0.225/5/3)
NDK Version: android-ndk-r4

STEPS TO REPRODUCE:
1. Locate $NDK/samples/san-angeles/AndroidManifest.xml and set android:debuggable="true"
2. Open cygwin shell and navigate to $NDK/samples/san-angeles
3. Execute command  ../../ndk-build
4. Build .apk using Android Eclipse SDK, by using “Create Project
from existing source”
5. Install and run .apk to a Android 2.2 emulator.
6. Execute command  ../../ndk-build -B , way-around to   issue 8893  .
7. Execute command ../../ndk-gdb 
8. Execute gdb command 'list' to verify that debugger is connected and symbols are loaded.
9. Execute gdb command 'info threads' you will get something like:

(gdb) info threads
Cannot access memory at address 0x0
  1 Thread 1010  0xafd0eb08 in ?? ()
warning: Couldn't restore frame in current thread, at frame 0
0xafd0eb08 in ?? ()

It fails to recognize GLThread, all native code runs in this thread's context.

10. Execute gdb command 'break Java_com_example_SanAngeles_DemoRenderer_nativeRender' and press 'c' to continue the application. GDB will crash with following error.

(gdb) break Java_com_example_SanAngeles_DemoRenderer_nativeRender
Breakpoint 1 at 0x80902f8a: file /cygdrive/e/Android/working_folder/ndk/samples/san-angeles/jni/app-android.c, line 92.
(gdb) c
Continuing.

Child terminated with signal = 5

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
The program no longer exists.
(gdb)
Child terminated with signal = 0x5 (SIGTRAP)
GDBserver exiting


ADDITIONAL INFORMATION:
OpenGL applications run the rendering code in a separate thread. Typically the rendering is implemented on native side for performance reasons. Lack of thread support in current GDB limits it usability for most serious applications, where debugger is needed the most.

Jul 12, 2010
#1 xav%andr...@gtempaccount.com
(No comment was entered for this change.)
Status: Assigned
Owner: digit.android
Jul 30, 2010
#2 xav%andr...@gtempaccount.com
(No comment was entered for this change.)
Owner: di...@android.com
Aug 6, 2010
#3 tim.men...@gmail.com
I've been looking into this issue myself. It appears that gdbserver is built without thread support.

When I tried to build gdbserver using the scripts, the "configure" script failed to find libthread_db, and disabled threading support. I was able to tweak the configure script to properly detect libthread_db by changing line 3660 to read:

      LIBS="$LIBS -lc -lthread_db -lc"

However, the resulting gdbserver I was able to build failed to work properly, so there are possibly deeper issues in getting gdbserver to work with the Bionic pthreads/thread_db implementation.

I also verified that the Google gdbserver appears to be built without thread support by running arm-eabi-strings on the original and my new gdbserver build:

tim@Panther /f/Devel/android-ndk-r4-crystax/build/prebuilt/windows/arm-eabi-4.4.0/bin
$ ./arm-eabi-strings.exe gdbserver.tim|grep thread
tried to change thread ID after multiple threads are created
thread:%x;
Previously current thread died.
no thread to satisfy query
invalid thread handle
invalid thread agent
application not linked with libthread
no thread-specific data for this thread
X register set not available for this thread
version mismatch between libthread_db and libpthread
unknown thread_db error '%d'
Cannot get thread info: %s
Attaching to thread %ld (LWP %d)
Could not attach to thread %ld (LWP %d)
Cannot enable thread event reporting for %d: %s
thread getmsg err: %s
Unable to set global thread event mask: %s
Unable to get location for thread creation breakpoint: %s
Cannot find new threads: %s
error initializing thread_db library: %s
_thread_created_hook

tim@Panther /f/Devel/android-ndk-r4-crystax/build/prebuilt/windows/arm-eabi-4.4.0/bin
$ ./arm-eabi-strings.exe gdbserver|grep thread
tried to change thread ID after multiple threads are created
thread:%x;
Previously current thread died.

This is a critical bug for me: gdb is basically worthless if you can't set a breakpoint in a non-main thread.
Aug 9, 2010
#4 tim.men...@gmail.com
I've tried several other builds of gdbserver, both 6.6 and 7.1.x, this time with threads enabled, and I still get no thread support, so this must be a deeper issue.

Any updates as to the status of this bug would be very much appreciated. As I said, this is a really, really critical bug--"Medium" priority doesn't BEGIN to address it! 
GDB is completely worthless in debugging for me at present because of this bug. 
Aug 9, 2010
#5 joe.lund...@gmail.com
This bug makes it essentially impossible to debug native GL code. I'm trying to port a game using the GSoC SDL port, and I'm currently working around this by blocking the UI thread completely just to capture the startup crashes, but this is obviously not going to work long term.
Sep 1, 2010
#6 nuskooler
This is making it impossible to complete our development. Have tried standard NDK/gdbserver as well as build with threads enabled with the same results. GDB simply will not hit breakpoints if not in the main thread (and half the time not in the main thread either).
Sep 3, 2010
#7 Rutto...@googlemail.com
I am also not lucky with this bug as I am also stuck with my project without GL debugging.
The normal ndk-built-process does not enable thread-capabilities for gdbserver, a small change in the configuration process for gdb-server enables it.
The results are similar to the stock-ndk-gdbserver:
See here:
(gdb) info threads
Cannot access memory at address 0x0
  1 Thread 381  0xafd0eb08 in ?? ()
warning: Couldn't restore frame in current thread, at frame 0
0xafd0eb08 in ?? ()
(gdb) thread 1
[Switching to thread 1 (Thread 381)]#0  0xafd0eb08 in ?? ()
(gdb) c
Continuing.

Program terminated with signal SIGTRAP, Trace/breakpoint trap.
The program no longer exists.
(gdb) 

Thread info gives some results, but I cannot say if its accurate.
I have set some breakpoints before this snipplet, that should have been reached after c(continuing). But I am stuck with the usual SIGTRAP.

Sep 22, 2010
#8 michael....@gmail.com
I'd been interested in a more or less official statement if this is just an error some people are fighting with due to user error or it's really a problem with gdbserver, bionic or something else? 

Thanks!
Sep 22, 2010
#9 tim.men...@gmail.com
The bug has been accepted and assigned; I think that's an official statement right there. 

There has also been a lot of discussion of this bug on the NDK list, which Googlers (including the one this bug is assigned to, if I'm guessing correctly) monitor, and none of them have said this isn't a real bug, and none have mentioned a workaround.

That said, a note that someone had reproduced the problem at Google would be appreciated. As to WHY the bug occurs -- I think by the time they know that, it will have been fixed, or nearly so. 

I'm hoping that fixing the build problem I mentioned above will actually solve the problem, and that something was just broken about my build environment that made it fail differently. 
Oct 6, 2010
#10 MattFisc...@gmail.com
A coworker and I managed to get this working.  He had already built a separate gdbserver like tim.mensch did, so I'm not sure if these instructions will work with the normal gdbserver or not.  The evidence in tim's post makes it look like the default gdbserver does not, in fact, have thread support, so that may still be necessary in addition to this.

The problem is that gdbserver cannot access the thread creation breakpoint.  This is a function whose sole purpose is to be called whenever a new thread is created.  That way, gdb can break on that function, and use it to be notified when new threads are created.  In Bionic, this function is called _thread_created_hook(), located in bionic/ptrace.c.

GDB needs to know the address of this symbol, so that it can set a breakpoint on it.  The symbol lives in libc.so, so the client-side gdb program needs to have access to that file in order for this to work.  Right now, though, the only file that the ndk-gdb script pulls across is the main program binary, app_process, which it puts in obj/local/armeabi.  If a copy of libc.so is also put into this directory, then gdb can gain access to its symbols, allowing it to properly locate the thread creation hook.  You can do this by typing 'adb pull /system/lib/libc.so obj/local/armeabi' from your app directory.  Long-term, I think Google just needs to update the ndk-gdb script to do this automatically.

For us, doing this along with using a custom build of gdbserver (which we pulled from a full build of Android) was enough to get access to all of the threads.  I haven't tested it with the stock build of gdbserver, so I don't know if that will work or not.
Oct 12, 2010
#11 olaf...@gmail.com
MattFischer84: could you be so kind to paste here any patches or instructions on how you enabled threads before you made the full build of android ?

I can confirm that pulling libc.so from the emulator (2.2) alone won't work.
Oct 12, 2010
#12 tim.men...@gmail.com
I said it above, but if you look at the configure script for gdbserver, line 3660 should have a LIBS= statement; change it to read:

      LIBS="$LIBS -lc -lthread_db -lc"

You have to be set up to build gdbserver, of course, which is its own adventure, and one that I apparently didn't succeed at, since mine didn't work correctly. :(
Oct 12, 2010
#13 olaf...@gmail.com
That's why I asked Matt, since his worked and yours didn't :) 
Oct 12, 2010
#14 tim.men...@gmail.com
My change makes it detect threading support correctly; I suspect it's the build environment or something else that needs to be improved to make it build correctly. I just didn't have any more time to play with it; dig around and look for instructions on how to build the toolchain, and follow those instructions better than I did. ;)
Oct 12, 2010
#15 MattFisc...@gmail.com
The gdbserver program is prebuilt, so it isn't actually built by the Android build process.  You should just be able to yank it out of prebuilt/android-arm/gdbserver from an Android source checkout.
Oct 12, 2010
#16 tim.men...@gmail.com
Umm, Matt -- it's apparently built without thread support. That's part of the problem with this bug. If it were built correctly, then we would already have a workaround. But we don't.
Oct 13, 2010
#17 olaf...@gmail.com
I got thread debugging to work! With emulator - does it require root on the phone ?

Right.  
Apparently MattFischer was right.  The prebuilt version in the Android Git repository can handle threads.  It's not the same as comes with NDK (version 4b).

Now, here's what I did.

Downloaded the gdbserver prebuilt from git repository.  Already had the entire platform checked out so I set to tag "repo init -b android-2.2_r1.3".  The md5 sum and location of the file is:
555a6bf047747b411fd13a6901e918cf prebuilt/android-arm/gdbserver

In my NDK4b is replaced /build/prebuilt/windows/arm-eabi-4.4.0/bin/gdbserver with the new one, so it would end up in my .apk files automatically.  (Rename the old one is case things go haywire).

Now, I couldn't start the gdbserver via ndk-gdb, since it couldn't create the socket in the emulator.

Goin in via "adb shell":
su # not sure this is necessary
cd /data/data/com.your.package/lib
ps # to find the PID of your application
gdbserver :5039 --attach YOURPID

Now you have the debugger running ... forward the port to it from your computer:
adb forward tcp:5039 tcp:5039

Now connect to it from your already existing Eclipse setup or 
/build/prebuilt/windows/arm-eabi-4.4.0/bin/arm-eabi-gdb.exe \
   -x /pato/to/obj/local/armeabi/gdb.setup \
   -e /path/to/obj/local/armeabi/app_process


If it doesn't work:
Make sure you have "set solib-search-path /path/to/obj/local/armeabi" in your gdb.setup

I had .so files (including libc.so) from emulator, from when i was trying to get the Tegra stuff to work ( http://tegradeveloper.nvidia.com/tegra/forum/android-debugging-eclipsegdb )
sh
for file in $(adb shell ls /system/lib | tr "\n" " " | tr "\r" " "); do
    adb pull /system/lib/$file lib
done

Now, if someone get this working by changing the ndk-gdb script, please post it :)
And, there might be an issue running this on the phone if it requires root ?
Not keen on voiding my warranty by rooting by N1.

Hope this will come as a relieve to someone!
Oct 13, 2010
#18 tim.men...@gmail.com
Ahh, well not creating the socket in the emulator is exactly how MY build failed. So I guess my build worked after all. 

Matt -- I was rebuilding the "prebuilt" tools. 

Thanks for figuring out how to make this work!
Oct 13, 2010
#19 MattFisc...@gmail.com
Cool, glad it's working for others too!

So, concrete issues for Google to fix:
  1.  Rebuild the NDK's gdbserver to include thread support (as per Tim's instructions.)  Using the full Android build's gdbserver will work, but since it doesn't have the magic socket stuff, the ndk-gdb script as it stands won't work.  Ideally the NDK needs a gdbserver with both thread support and the extra socket support.
  2.  Modify ndk-gdb to pull over libc.so, similar to how it currently pulls over app_process.  This is necessary so that gdb can find the thread creation breakpoint, which is necessary for any thread debugging.  It also has the happy side effect of allowing you to debug calls into the C library--it would probably be nice if that script pulled over even more system libraries just for that reason.
Oct 13, 2010
#20 djmuhles...@gmail.com
I think you've summed it up OK, Matt.  As is, the prebuilt gdbserver from android sources without the socket mods isn't very useful unless you have a rooted device.  I haven't found a way around the permission denied error for opening the socket.  

As far as pulling libc.so over, that is a quick change anyone can make to their ndk-gdb script if they choose in the mean time.

Does anyone have a link to a good reference for rebuilding gdbserver yourself? I tried checking out the toolchain/gdb sources and then using the build-gdbserver.sh script in the ndk/build/tools/ directory but had compile errors with missing libraries.  Haven't given much time yet.  I figured I was probably missing something in my setup.
Oct 13, 2010
#21 michael....@gmail.com
Thanks for the hard work pals! Could someone post a working gdbserver arm-binary somewhere. That would be much appreciated!
Nov 3, 2010
#22 mitchell...@gmail.com
I was able to get debugging w/ threads working using gdbserver located at: http://android.git.kernel.org/?p=platform/prebuilt.git;a=blob_plain;f=android-arm/gdbserver/gdbserver;hb=d4f4703a1b23e6a73becec455c3d561ae3829a65

This matches the md5sum provided by MattFischer84 in comment #17. 
Nov 9, 2010
#23 DrT...@gmail.com
--Works but debugging super slow in Eclipse...?
I'm finding that although the modified gdbserver + hacks listed above *do* work for multithreaded apps (fantastic, phew), and result in a functional multithreaded debugging setup, debugger operations via Eclipse (windows, using emulator) are incredibly slow; e.g. a single step of one instruction takes 10-20 seconds.

Turning on "verbose" output in eclipse debug config tab reveals that two gdb commands are super slow;
"info sharedlibrary"
and 
"info threads"
when run manually in gdb (not using eclipse) these commands take an age (e.g. several seconds) to run. Don't know why, don't care. 

--Quick horrible hack fix
Not confirmed whether this is on the modified-gdbserver end (I assume so), but I have a horrible quick'n'dirty hack for y'all. It's easier than rebuilding eclipse CDT or gdb or gdbserver; 
simply hex edit your local "arm-eabi-gdb.exe" executable (the one that eclipse invokes) and hack the "sharedlibrary" string (e.g. to 'Xharedlibrary') - this is at offset 0x11eae in the 4.4.0 windows binary I have. This means eclipse fails to invoke that command (doesn't matter) and debugging is much quicker. Could do the same for "info threads" but you might miss having that. Ugly but instant fix.

Thanks for info above everyone - lifesaver!  

--- Iterate faster when recompiling native code
Save a lot of time when iterating;
1) don't rebuild your whole APK to test changes; just recompile your library .so
2) kill any existing yourclass process (adb shell kill [pid])
3) use "adb push" to load the recompiled .so directly into the /data/data/yourclass/lib/
4) use "adb am -n yourclass" to start an instance of your app again
5) (optionally restart gdbserver, open tcp port, etc)

Cheers,
DrTune
Nov 16, 2010
#24 sich...@mail.ru
Not sure if it is allowed here to post links to external sites - but I have put up a small set of instructions for all those who do not like hacks and gdbserver rebuilding stuff and just need the multithreaded debugging works - sichent.wordpress.com (see the last post from November 16, 2010).

Regards all 
sich
Feb 21, 2011
#25 digit%an...@gtempaccount.com
Released in NDK r5, unfortunately debugging threaded-code only works on Android 2.3 due to a platform bug that couldn't be fixed before.

Status: Released
Feb 28, 2011
#26 revert...@gmail.com
Threaded code using GLSurfaceView can not be debugged at the time.
Feb 28, 2011
#27 olaf...@gmail.com
It can. Make sure you are using Android 2.3 and most recent NDK. (run on emulator if your phone doesn't have it).  At least I'm debugging my OpenGL application using GLSurfaceView and native code.
Feb 28, 2011
#28 revert...@gmail.com
I'm having this issue: http://groups.google.com/group/android-ndk/browse_thread/thread/22c6f96d4784e852

As I said there - I've tried 2.2 and 2.3.3 emulators.
Mar 2, 2011
#29 digit%an...@gtempaccount.com
Your issue is with Crystax's NDK. Again grab NDK r5b.
Apr 8, 2011
#30 oberthe...@toonboom.com
This is clearly a major pain in the butt ! 

Sign in to add a comment

Powered by Google Project Hosting