Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No filename when using llvm-symbolizer with ASan on Mac #207

Closed
ramosian-glider opened this issue Aug 31, 2015 · 25 comments
Closed

No filename when using llvm-symbolizer with ASan on Mac #207

ramosian-glider opened this issue Aug 31, 2015 · 25 comments

Comments

@ramosian-glider
Copy link
Member

Originally reported on Google Code with ID 207

What steps will reproduce the problem?
1. Build Firefox using the MOZCONFIG=~/mozconfig-asan (attached)
2. Trip https://bugzilla.mozilla.org/show_bug.cgi?id=887499

What is the expected output? What do you see instead?

With ASAN_SYMBOLIZER_PATH=~/llvm/build/Release+Asserts/bin/llvm-symbolizer, all lines
are missing source filenames and line numbers:

    #0 0x1057ac304 in nsINode::ReplaceOrInsertBefore(bool, nsINode*, nsINode*, mozilla::ErrorResult&)
(/Users/jruderman/trees/mozilla-central/obj-firefox-asan/dist/Nightly.app/Contents/MacOS/XUL+0x1577304)

With | ~/llvm/projects/compiler-rt/lib/asan/scripts/asan_symbolize.py, all lines have
source filenames and line numbers:

    #0 0x1057a7304 in nsINode::ReplaceOrInsertBefore nsINode.cpp:1616

What version of the product are you using? On what operating system?

* LLVM r185346
* Mac 10.7.5
* Firefox: https://hg.mozilla.org/mozilla-central/rev/4ffb23062b3b

Please provide any additional information below.


Reported by jruderman on 2013-07-02 03:16:31


- _Attachment: [mozconfig-asan](https://storage.googleapis.com/google-code-attachments/address-sanitizer/issue-207/comment-0/mozconfig-asan)_
@ramosian-glider
Copy link
Member Author

Have you checked whether llvm-symbolizer is actually being invoked? E.g. you can replace
llvm-symbolizer with a script and check that. I'm not sure that the path starting with
~ is correctly expanded.

Reported by ramosian.glider on 2013-07-02 08:34:14

  • Labels added: OpSys-OSX

@ramosian-glider
Copy link
Member Author

Yes, llvm-symbolizer is being invoked.  Bash expands ~ in FOO=~/bar/.

Without any symbolizer (or with an invalid symbolizer path), I don't even get a function
name:

    #0 0x1057a7304 (/Users/jruderman/trees/mozilla-central/obj-firefox-asan/dist/Nightly.app/Contents/MacOS/XUL+0x1577304)

Reported by jruderman on 2013-07-02 19:32:28

@ramosian-glider
Copy link
Member Author

*sigh*
I'll try some more questions to avoid building a Firefox myself:
1) Does standalone llvm-symbolizer works? Run "./bin/llvm-symbolizer", pass "/Users/jruderman/trees/mozilla-central/obj-firefox-asan/dist/Nightly.app/Contents/MacOS/XUL
0x1577304" to stdin and check if it prints file-line info correctly.
2) Can you check where debug info for the binary is stored? I'm interested in sections
(gobjdump -h) of /Users/jruderman/trees/mozilla-central/obj-firefox-asan/dist/Nightly.app/Contents/MacOS/XUL
and the existence and sections of dSYM-companion file (I think it's expected location
is /Users/jruderman/trees/mozilla-central/obj-firefox-asan/dist/Nightly.app/Contents/MacOS/XUL.dSYM/Contents/Resources/DWARF/XUL).

Reported by samsonov@google.com on 2013-07-03 10:28:50

@ramosian-glider
Copy link
Member Author

I'm also seeing something similar with my programs (if it's any indication it's C code).
Moreover I've got the same issue #92 concerning symbolize of lib.

Reported by mkvtoolnix.build.jonthn on 2013-07-05 14:50:40

@ramosian-glider
Copy link
Member Author

Can you provide data similar to what I asked in https://code.google.com/p/address-sanitizer/issues/detail?id=207#c3?

Reported by samsonov@google.com on 2013-07-05 15:32:36

@ramosian-glider
Copy link
Member Author

I will do it next week. Didn't had time today for it.

Reported by mkvtoolnix.build.jonthn on 2013-07-05 18:45:34

@ramosian-glider
Copy link
Member Author

1) Gives me :

functionName
??:0:0

2) The output of "gobjdump -h" below : 

program:     file format mach-o-i386

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         000a3960  000028d0  000028d0  000018d0  2**4
                  CONTENTS, ALLOC, LOAD, CODE
  1 .cstring      00005c82  000a6230  000a6230  000a5230  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .const        00006a44  000abec0  000abec0  000aaec0  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .symbol_stub  00000360  000b2904  000b2904  000b1904  2**1
                  CONTENTS, ALLOC, LOAD, CODE
  4 __TEXT.__stub_helper 00000924  000b2c64  000b2c64  000b1c64  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  5 __TEXT.__unwind_info 00000178  000b3590  000b3590  000b2590  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  6 .eh_frame     000028f0  000b3708  000b3708  000b2708  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .dyld         0000001c  000b6000  000b6000  000b5000  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  8 .mod_init_func 00000064  000b601c  000b601c  000b501c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  9 .non_lazy_symbol_pointer 000000a8  000b6080  000b6080  000b5080  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 10 .lazy_symbol_pointer 00000240  000b6128  000b6128  000b5128  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 11 .const_data   00000100  000b6380  000b6380  000b5380  2**5
                  CONTENTS, ALLOC, LOAD, DATA
 12 .data         000032a8  000b6480  000b6480  000b5480  2**5
                  CONTENTS, ALLOC, LOAD, DATA
 13 .mod_term_func 00000048  000b9728  000b9728  000b8728  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 14 .bss          00008aa0  000b9780  000b9780  00000000  2**5
                  ALLOC
 15 __DATA.__common 00000100  000c2220  000c2220  00000000  2**5
                  ALLOC
 16 LC_THREAD.x86_THREAD_STATE32.0 00000040  00000000  00000000  00000650  2**0
                  CONTENTS

In this case it's a static binary (but also got the same problem with dynamically linked
binary). But I don't know (in general) what's this dSYM-companion file.

It's on a Snow Leopard system. 

Reported by mkvtoolnix.build.jonthn on 2013-07-08 06:51:41

@ramosian-glider
Copy link
Member Author

Okay... So indeed your binary doesn't contain debug info sections, and if you've built
it with debug info, it is stored elsewhere. If the path to your binary is /path/to/foo,
do you have "/path/to/foo.dSYM" directory and, specifically, "/path/to/foo.dSYM/Contents/Resources/DWARF/foo"
file?

Which compiler did you use to build the binary - Clang or XCode?

Reported by samsonov@google.com on 2013-07-08 08:00:50

@ramosian-glider
Copy link
Member Author

I'm using clang directly (via Makefile) and I've looked for dSYM but no luck.

So I've rebuilt my binary using latest revision of Clang+Asan (185797)
And I'm still not getting any file/line in asan report using llvm-symbolizer.

I tried loading it in gdb and placing a breakpoint it works correctly and I get
line number & file, so I assume debug informations are present.

I re-ran gobjdump -h on the binary :

program:     file format mach-o-i386

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         0000ee80  00002210  00002210  00001210  2**4
                  CONTENTS, ALLOC, LOAD, CODE
  1 .cstring      00000ca0  00011090  00011090  00010090  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .const        00000008  00011d30  00011d30  00010d30  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .symbol_stub  0000033c  00011d38  00011d38  00010d38  2**1
                  CONTENTS, ALLOC, LOAD, CODE
  4 __TEXT.__stub_helper 000008c4  00012074  00012074  00011074  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  5 __TEXT.__unwind_info 000000b4  00012940  00012940  00011940  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  6 .eh_frame     00000604  000129f4  000129f4  000119f4  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .dyld         0000001c  00013000  00013000  00012000  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  8 .mod_init_func 0000000c  0001301c  0001301c  0001201c  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  9 .non_lazy_symbol_pointer 00000094  00013028  00013028  00012028  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 10 .lazy_symbol_pointer 00000228  000130bc  000130bc  000120bc  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 11 .const_data   00000008  000132e4  000132e4  000122e4  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 12 .data         00000038  000132ec  000132ec  000122ec  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 13 .bss          00000d28  00013324  00013324  00000000  2**2
                  ALLOC
 14 __DATA.__common 0000002c  0001404c  0001404c  00000000  2**2
                  ALLOC
 15 LC_THREAD.x86_THREAD_STATE32.0 00000040  00000000  00000000  0000060c  2**0
                  CONTENTS


I'm using "GNU objdump (GNU Binutils) 2.23.2" if it's any information.

Reported by mkvtoolnix.build.jonthn on 2013-07-08 10:35:25

@ramosian-glider
Copy link
Member Author

Can you recompile with -v and check that -g (or -gline-tables-only) is passed to the
compiler and linker?
When linking, Clang usually invokes dsymutil after ld - does this happen in your case?

Reported by ramosian.glider on 2013-07-08 10:43:43

@ramosian-glider
Copy link
Member Author

I also wonder if the debug info sections are present in the .o files

Reported by ramosian.glider on 2013-07-08 10:44:40

@ramosian-glider
Copy link
Member Author

The -g flag is always present in the different clang command-lines but I don't know
to what flag(s) it translates to "ld".
Also I didn't see a call to dsymutil, is it "echoed" when using -v ?

In the .o there seems to be debug informations as seen below : 

part.o:     file format mach-o-i386

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         00004006  00000000  00000000  000006d8  2**4
                  CONTENTS, ALLOC, LOAD, RELOC, CODE
  1 .debug_info   0000107d  00004006  00004006  000046de  2**0
                  CONTENTS, RELOC, DEBUGGING
  2 .debug_abbrev 000002d2  00005083  00005083  0000575b  2**0
                  CONTENTS, DEBUGGING
  3 .debug_aranges 00000000  00005355  00005355  00005a2d  2**0
                  CONTENTS, DEBUGGING
  4 .debug_macinfo 00000000  00005355  00005355  00005a2d  2**0
                  CONTENTS, DEBUGGING
  5 .debug_line   000008e3  00005355  00005355  00005a2d  2**0
                  CONTENTS, RELOC, DEBUGGING
  6 .debug_loc    000005c7  00005c38  00005c38  00006310  2**0
                  CONTENTS, RELOC, DEBUGGING
  7 .debug_pubtypes 00000155  000061ff  000061ff  000068d7  2**0
                  CONTENTS, DEBUGGING
  8 .debug_str    00000773  00006354  00006354  00006a2c  2**0
                  CONTENTS, DEBUGGING
  9 .debug_ranges 000005c0  00006ac7  00006ac7  0000719f  2**0
                  CONTENTS, RELOC, DEBUGGING
 10 .data         00000004  00007088  00007088  00007760  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 11 __DATA.__common 0000002c  000081a4  000081a4  00000000  2**2
                  ALLOC
 12 .bss          00000408  000081d0  000081d0  00000000  2**2
                  ALLOC
 13 .cstring      000003c5  0000708c  0000708c  00007764  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 14 __TEXT.__const_coal 00000008  00007454  00007454  00007b2c  2**2
                  CONTENTS, ALLOC, LOAD, CODE, DATA
 15 .mod_init_func 00000004  0000745c  0000745c  00007b34  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, DATA
 16 __DWARF.__debug_inlined 0000015d  00007460  00007460  00007b38  2**0
                  CONTENTS, RELOC, DEBUGGING
 17 __DWARF.__apple_names 0000040c  000075bd  000075bd  00007c95  2**0
                  CONTENTS, RELOC, DEBUGGING
 18 __DWARF.__apple_objc 00000024  000079c9  000079c9  000080a1  2**0
                  CONTENTS, DEBUGGING
 19 __DWARF.__apple_namespac 00000024  000079ed  000079ed  000080c5  2**0
                  CONTENTS, DEBUGGING
 20 __DWARF.__apple_types 0000057f  00007a11  00007a11  000080e9  2**0
                  CONTENTS, RELOC, DEBUGGING
 21 .non_lazy_symbol_pointer_x86 00000058  00007f90  00007f90  00008668  2**0
                  CONTENTS, ALLOC, LOAD, DATA
 22 .eh_frame     000001bc  00007fe8  00007fe8  000086c0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA

Reported by mkvtoolnix.build.jonthn on 2013-07-08 14:25:20

@ramosian-glider
Copy link
Member Author

Yes, "clang -v -g" should print a call to dsymutil. E.g. here's what I see on my machine
(see the last line):

$ ./bin/clang++ -g -v tmp/a.cc 
clang version 3.4 (185309)
Target: x86_64-apple-darwin12.4.0
Thread model: posix
<... more output here ....>
 "/usr/bin/ld" -dynamic -arch x86_64 -macosx_version_min 10.8.0 -o a.out /var/folders/00/13hl0000h01000cxqpysvccm004f2c/T/a-006133.o
-lstdc++ -lSystem
 "/usr/bin/dsymutil" -o a.out.dSYM a.out

Reported by samsonov@google.com on 2013-07-08 20:26:57

@ramosian-glider
Copy link
Member Author

When using a simple hello-world.c I can see the call to dsymutil and also the generated
dSYM directory (file...)

But in my project this behavior doesn't happen. I tried removing some flag but still
no call to dsymutil.

Reported by mkvtoolnix.build.jonthn on 2013-07-09 07:05:13

@ramosian-glider
Copy link
Member Author

Are you sure you're using Clang to link your project? Could it be that your build system
just invokes plain ld without passing -g to it?

Reported by ramosian.glider on 2013-07-09 07:09:33

@ramosian-glider
Copy link
Member Author

I'm using clang to link as I can see the clang version description as seen in your example
above. And -g is the second flag passed to it after -v.

In order to be complete about my setup, I'm using wrapper script to switch compiler
but that shouldn't be the problem.

The command more or less as I can see them(the first is the command launched
by Makefile, the second is the translation by the wrapper which 'echo' the
command that it'll launch. After that it's clang output):

MACOSX_DEPLOYMENT_TARGET=10.5 gcc -prebind -flat_namespace -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk
-o /tmp/program part1.o part2.o -L/tmp/LIB1/MACOSX -L/tmp/LIB3/MACOSX -L/tmp/LIB2/MACOSX
-Wl,-stack_size,4000000,-stack_addr,C0000000 -L/usr/X11R6/lib -dead_strip -lLIB1 -lLIB2
-lLIB3 -lpthread -lcurl -lfirewire -lXext -lXt -lXp -lX11 -lm -o /tmp/program
/tmp/clang/bin/clang -v -g -prebind -flat_namespace -arch i386 -isysroot /Developer/SDKs/MacOSX10.5.sdk
-o /tmp/program part1.o part2.o -L/tmp/LIB1/MACOSX -L/tmp/LIB3/MACOSX -L/tmp/LIB2/MACOSX
-Wl,-stack_size,4000000,-stack_addr,C0000000 -L/usr/X11R6/lib -dead_strip -lLIB1 -lLIB2
-lLIB3 -lpthread -lcurl -lfirewire -lXext -lXt -lXp -lX11 -lm -o /tmp/program
clang version 3.4 (185797)
Target: i386-apple-darwin10.8.0
Thread model: posix
 "/usr/bin/ld" -dynamic -arch i386 -dead_strip -flat_namespace -macosx_version_min
10.5.0 -prebind -syslibroot /Developer/SDKs/MacOSX10.5.sdk -o /tmp/program -lcrt1.10.5.o
-L/tmp/LIB1/MACOSX -L/tmp/LIB3/MACOSX -L/tmp/LIB2/MACOSX  -L/usr/X11R6/lib part1.o
part2.o -stack_size 4000000 -stack_addr C0000000 -lLIB1 -lLIB2 -lLIB3 -lpthread -lcurl
-lfirewire -lXext -lXt -lXp -lX11 -lm -lSystem -lgcc_s.10.5


Reported by mkvtoolnix.build.jonthn on 2013-07-09 08:37:14

@ramosian-glider
Copy link
Member Author

Your clang command line doesn't contain -fsanitize=address - am I right your final executable
is missing the debug info even when built without ASan?

Reported by ramosian.glider on 2013-07-09 08:51:21

@ramosian-glider
Copy link
Member Author

Yes I changed it for the example.
You're also right the final executable doesn't contain debug info if compiled without
ASan but as noted before gdb is still able to find debug informations (I can set a
breakpoint and it shows me the file/line).

Reported by mkvtoolnix.build.jonthn on 2013-07-09 11:00:29

@ramosian-glider
Copy link
Member Author

Hm, what if you run dsymutil on the final executable - will ASan be able to read the
debug info?
I'm still puzzled with what's going on, and I don't have enough experience with debug
info on OS X to make a good guess.

Reported by ramosian.glider on 2013-07-09 11:59:11

@ramosian-glider
Copy link
Member Author

Per your suggestion I rand dsymutil on the final bin and afterwards I've got file &
line in ASan report.

I'm really interested on what could be the source of this problem, so if I understand
correctly gdb uses information that clang/llvm-symbolizer can't; and once you've written
the DWARF information then llvm-symbolizer works.

Reported by mkvtoolnix.build.jonthn on 2013-07-09 13:25:08

@ramosian-glider
Copy link
Member Author

My understanding was that unless dsymutil is ran the debug info resided in the binary
in the .debug* sections which objdump can see. In this case llvm-symbolizer should
be able to find and use the debug info.

However it turns out that the binary doesn't contain the debug info, so I'm still curious
how gdb (and apparently atos) manages to find it.

Reported by ramosian.glider on 2013-07-09 15:01:03

@ramosian-glider
Copy link
Member Author

Can confirm that invoking dsymutil fixes the issue when using a makefile creating several
obj-files and linking them in a later step, even though every step has -g -fsanitize=address
-fno-omit-frame-pointer -O0

Tested with clang version 3.4 (tags/RELEASE_34/final) Target: x86_64-apple-darwin13.0.0
Thread model: posix under Mac OS X 10.9.1 Mavericks, clang compiled via macports with
+analyzer and +python2.7.

Just to say: I lost one whole workday for this, someone should definitively update
http://clang.llvm.org/docs/AddressSanitizer.html or AT LEAST leave a note/hint!

Reported by mathias.habluetzel on 2014-01-14 19:02:33

@ramosian-glider
Copy link
Member Author

Yeah, according to Clang source, they're only running dsymutil when compiling and linking
single source files (because the temporary object files are deleted afterwards). When
Clang is used to link several object files into a single executable, dsymutil isn't
being ran deliberately (to speed up the development cycle). In this case the DWARF
debug info resides in the object files, and the final executable contains just the
links to the object files (see http://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable
for more info; you can also look at machoread.c in the GDB source).

Right now llvm-symbolizer is unable to extract the debug info from the object files
accompanying the executable (to do so we basically need to reimplement dsymutil), so
the users must run dsymutil themselves.

I'll update the documentation. Thanks everyone for your patience.

Reported by ramosian.glider on 2014-01-16 13:06:18

  • Status changed: Accepted

@ramosian-glider
Copy link
Member Author

Updated http://clang.llvm.org/docs/AddressSanitizer.html, nothing else to do here.

Reported by ramosian.glider on 2014-01-16 14:04:08

  • Status changed: Done

@ramosian-glider
Copy link
Member Author

Adding Project:AddressSanitizer as part of GitHub migration.

Reported by ramosian.glider on 2015-07-30 09:13:41

  • Labels added: ProjectAddressSanitizer

hansungk added a commit to hansungk/ruse that referenced this issue Jun 18, 2017
Strangely enough, while a single compile-and-link command generates exe.dSYM/
dir that contains debug infos, separate compile/link invocation does not,
disabling file and line numbers report in ASan. This issue was directly
addressed in [1]. The reason for this is explained in [2].

This adds a post-build rule that runs 'dsymutil' which produces the debug
info dir from the executable.

This issue has been bugging me for a whole day now. Hopes this fixes any
debug symbol related problem once and for all.

[1] google/sanitizers#207
[2] https://stackoverflow.com/questions/32297349/why-does-a-2-stage-command-line-build-with-clang-not-generate-a-dsym-directory
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant