Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CHECK failed (assert() tls + stack) on Linux in 32bit threaded #361

Closed
ramosian-glider opened this issue Sep 1, 2015 · 12 comments
Closed

CHECK failed (assert() tls + stack) on Linux in 32bit threaded #361

ramosian-glider opened this issue Sep 1, 2015 · 12 comments
Labels
Priority-Medium ProjectAddressSanitizer Status-Accepted Type-Defect

Comments

@ramosian-glider
Copy link
Member

Originally reported on Google Code with ID 361

What steps will reproduce the problem?
1. Build clang+compiler-rt 3.5
2. Use it with attached test file to produce a 32-bit binary multi-threaded
3. AddressSanitizer CHECK failed (sanitizer_linux_libcdep.cc:433)


What is the expected output? What do you see instead?

$ clang -m32 -fsanitize=address test_thread.c -DTHREAD_USE
The program should launch and print the function name instead :
$ ./a.out

==19661==AddressSanitizer CHECK failed: /tmp/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:433
"((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0xf638f030, 0xf638f000)
    #0 0x80da2ad (/tmp/a.out+0x80da2ad)
    #1 0x80de75f (/tmp/a.out+0x80de75f)
    #2 0x80ed8f7 (/tmp/a.out+0x80ed8f7)
    #3 0x80dbe61 (/tmp/a.out+0x80dbe61)
    #4 0x80dc1f2 (/tmp/a.out+0x80dc1f2)
    #5 0x80b4afd (/tmp/a.out+0x80b4afd)
    #6 0xf76f8953 (/lib/i386-linux-gnu/libpthread.so.0+0x5953)

But in 32-bit without thread it works :

$ clang -m32 -fsanitize=address test_thread.c
$ ./a.out
threadfunc


And in 64-bit it works in every case :

clang -m64 -fsanitize=address test_thread.c -DTHREAD_USE
clang -m64 -fsanitize=address test_thread.c


What version of the product are you using? On what operating system?

Clang 3.5  (also tested on 3.4 same result)

Host for compiling the test and running it :
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.2 (wheezy)
Release:    7.2
Codename:   wheezy

Please provide any additional information below.

I used another system to build clang+asan :
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.10 (squeeze)
Release:    6.0.10
Codename:   squeeze

But even if I build the test file on this machine the resulting binary doesn't work
on 'wheezy'

I also tried changing the stack value (ulimit (-H/-S) -s) without success.

Informations from the libc (in case that would be helpful) :

/lib/i386-linux-gnu/libc-2.13.so 
GNU C Library (Debian EGLIBC 2.13-38) stable release version 2.13, by Roland McGrath
et al.
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.4.7.
Compiled on a Linux 3.2.35 system on 2012-12-30.
Available extensions:
    crypt add-on version 2.1 by Michael Glad and others
    GNU Libidn by Simon Josefsson
    Native POSIX Threads Library by Ulrich Drepper et al
    BIND-8.2.3-T5B
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.

Reported by jon3405691578 on 2014-12-04 13:57:48

- _Attachment: [test_thread.c](https://storage.googleapis.com/google-code-attachments/address-sanitizer/issue-361/comment-0/test_thread.c)_
@ramosian-glider
Copy link
Member Author 

Quick update this is also reproducible on Wheezy 7.7.

But with Jessie (Debian 8) it seems to be fixed as I'm not able to reproduce it.

Reported by jon3405691578 on 2014-12-30 09:02:17

@ramosian-glider
Copy link
Member Author 

Can you please rebuild the program with -g and:
1. Run the program with ASAN_OPTIONS=verbosity=2 and attach the output
2. Post a symbolized ASan report (http://clang.llvm.org/docs/AddressSanitizer.html)
3. Run with ASAN_OPTIONS=sleep_before_dying=100, attach gdb and post the stacktrace?

Reported by ramosian.glider on 2014-12-30 09:40:40

@ramosian-glider
Copy link
Member Author 

Thanks for your answer.

Here you go (with clang 3.5.1 225002):

1. Verbose output symbolized (so it's also 2.)

==3725==Parsed ASAN_OPTIONS: verbosity=2
==3725==AddressSanitizer: failed to intercept '__isoc99_printf'
==3725==AddressSanitizer: failed to intercept '__isoc99_sprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_snprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_fprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vsprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vsnprintf'
==3725==AddressSanitizer: failed to intercept '__isoc99_vfprintf'
==3725==AddressSanitizer: libc interceptors initialized
|| `[0x40000000, 0xffffffff]` || HighMem    ||
|| `[0x28000000, 0x3fffffff]` || HighShadow ||
|| `[0x24000000, 0x27ffffff]` || ShadowGap  ||
|| `[0x20000000, 0x23ffffff]` || LowShadow  ||
|| `[0x00000000, 0x1fffffff]` || LowMem     ||
MemToShadow(shadow): 0x24000000 0x247fffff 0x25000000 0x27ffffff
redzone=16
max_redzone=2048
quarantine_size=64M
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 20000000
==3725==Installed the sigaction for signal 11
==3725==SetCurrentThread: 0xf770e000 for thread 0xf753b6f0
==3725==T0: stack [0xff6c0000,0xffec0000) size 0x800000; local=0xffebdaec
==3725==AddressSanitizer Init done
==3725==SetCurrentThread: 0xf770e078 for thread 0xf5fffb70
==3725==AddressSanitizer CHECK failed: /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:433
"((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))" (0xf6000030, 0xf6000000)
    #0 0x80f2b86 in __asan::AsanCheckFailed(char const*, int, char const*, unsigned
long long, unsigned long long) /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:70
    #1 0x80f83bc in __sanitizer::CheckFailed(char const*, int, char const*, unsigned
long long, unsigned long long) /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:76
    #2 0x8108ef6 in __sanitizer::GetThreadStackAndTls(bool, unsigned long*, unsigned
long*, unsigned long*, unsigned long*) /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:433
    #3 0x80f57d4 in __asan::AsanThread::SetThreadStackAndTls() /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:187
    #4 0x80f57d4 in __asan::AsanThread::Init() /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:146
    #5 0x80f5ac9 in __asan::AsanThread::ThreadStart(unsigned long) /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:159
    #6 0x807a8c2 in asan_thread_start(void*) /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:170
    #7 0xf76e1953 in start_thread (/lib/i386-linux-gnu/libpthread.so.0+0x5953)

3. Stacktrace with gdb

0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
(gdb) bt
#0  0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
#1  0xf75f30c2 in sleep () from /lib/i386-linux-gnu/libc.so.6
#2  0x0810c3ff in main ()
(gdb) info threads
  Id   Target Id         Frame 
  2    Thread 0xf5fffb70 (LWP 3814) "a.out" 0xf75f329c in nanosleep ()
   from /lib/i386-linux-gnu/libc.so.6
* 1    Thread 0xf75526f0 (LWP 3813) "a.out" 0xf75f329c in nanosleep ()
   from /lib/i386-linux-gnu/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0xf5fffb70 (LWP 3814))]
#0  0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
(gdb) bt
#0  0xf75f329c in nanosleep () from /lib/i386-linux-gnu/libc.so.6
#1  0xf75f30c2 in sleep () from /lib/i386-linux-gnu/libc.so.6
#2  0x0810958b in __sanitizer::SleepForSeconds (seconds=100)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:72
#3  0x080f3450 in __asan::AsanDie ()
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:46
#4  0x080f832b in __sanitizer::Die ()
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:63
#5  0x080f2c24 in __asan::AsanCheckFailed (
    file=0x811a70c "/tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc",
line=433, 
    cond=0x811a874 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))", 
    v1=4127195184, v2=4127195136)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:71
#6  0x080f83bd in __sanitizer::CheckFailed (
    file=0x811a70c "/tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc",
line=433, 
    cond=0x811a874 "((*tls_addr + *tls_size)) <= ((*stk_addr + *stk_size))", 
    v1=4127195184, v2=4127195136)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:76
#7  0x08108ef7 in __sanitizer::GetThreadStackAndTls (main=false, 
    stk_addr=0xf7387010, stk_size=0xf7387014, tls_addr=0xf7387018, 
    tls_size=0xf5fff2e8)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:433
#8  0x080f57d5 in SetThreadStackAndTls (this=0xf7387000)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:187
#9  __asan::AsanThread::Init (this=this@entry=0xf7387000)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:146
#10 0x080f5aca in __asan::AsanThread::ThreadStart (this=0xf7387000, os_id=3814)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_thread.cc:159
#11 0x0807a8c3 in asan_thread_start (arg=0xf7387000)
    at /tmp/ABC/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:170
#12 0xf76f8954 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#13 0xf762895e in clone () from /lib/i386-linux-gnu/libc.so.6


On a side note even though it's specified to sleep for 100 seconds, I had to increase
the sleep() time in the test_thread.c code otherwise it exited after the specified
sleep time in the code (so after 1 second).
Thanks again for your help.

Reported by jon3405691578 on 2014-12-30 21:09:42

@ramosian-glider
Copy link
Member Author 

FYI this now also happens on our buildbot:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/15213/steps/run%20sanitizer%20tests%20in%20gcc%20build/logs/stdio

This one is interesting:
[ RUN      ] SanitizerLinux.ThreadDescriptorSize
/mnt/b/sanitizer-buildbot1/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/sanitizer_linux_test.cc:230:
Failure
Value of: ThreadDescriptorSize()
  Actual: 1216
Expected: (uptr)result
Which is: 1168
[  FAILED  ] SanitizerLinux.ThreadDescriptorSize (4 ms)

I've checked that versions of libc and libpthread are 2.13, but the descriptor size
we calculate seems to be coincide with size used in older versions.

Reported by samsonov@google.com on 2015-01-09 23:00:14

@ramosian-glider
Copy link
Member Author 

FYI this now also happens on our buildbot:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/15213/steps/run%20sanitizer%20tests%20in%20gcc%20build/logs/stdio

This one is interesting:
[ RUN      ] SanitizerLinux.ThreadDescriptorSize
/mnt/b/sanitizer-buildbot1/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/sanitizer_linux_test.cc:230:
Failure
Value of: ThreadDescriptorSize()
  Actual: 1216
Expected: (uptr)result
Which is: 1168
[  FAILED  ] SanitizerLinux.ThreadDescriptorSize (4 ms)

I've checked that versions of libc and libpthread are 2.13, but the descriptor size
we calculate seems to be coincide with size used in older versions.

Reported by samsonov@google.com on 2015-01-09 23:00:15

@ramosian-glider
Copy link
Member Author 

Hmm.

Here the actual size is 1168, but the one we expect is 1216. Originally, we expected
1168 on 2.13 and lower, exactly because someone had reported 1168 on 2.13 and 2.11.
This was changed after Jakub gathered descriptor sizes for different glibc versions:

https://gcc.gnu.org/ml/gcc-patches/2013-12/msg00287.html

Here we have 1216 for 2.13. I guess no one noticed at the time that Jakub's patch changed
the value for 2.13.

The question is, is it a problem with Jakub's data (unlikely because it was gathered
programmatically) or if there's another variable (besides the version number) that
can affect the descriptor size.

Reported by earthdok@google.com on 2015-01-19 15:39:07

  • Status changed: Accepted

@ramosian-glider
Copy link
Member Author 

FTR, I downloaded 2.13 from https://packages.debian.org/wheezy/libc6 and used Jakub's
method. I got FIRST_32_SECOND_64(1216, 2304) - the same values we originally used.

Jakub's binaries were from RedHat, so I wonder if it's different there?

As a temporary solution I'm going to reinstate original values for 2.13, to make our
bots green at least.

Reported by earthdok@chromium.org on 2015-01-23 21:08:23

@ramosian-glider
Copy link
Member Author 

> I got FIRST_32_SECOND_64(1216, 2304)

Typo - should be (1168, 2304)

Reported by earthdok@google.com on 2015-01-23 21:11:00

@ramosian-glider
Copy link
Member Author 

Committed r226938

Reported by earthdok@google.com on 2015-01-23 21:14:32

@ramosian-glider
Copy link
Member Author 

I just tried again with Clang 3.6.1 (clang version 3.6.1 (branches/release_36 232166)

And I can still reproduce this issue.
Do you have any tip so I could try to solve this issue ?

Reported by jon3405691578 on 2015-03-13 16:20:30

@ramosian-glider
Copy link
Member Author 

Adding Project:AddressSanitizer as part of GitHub migration.

Reported by ramosian.glider on 2015-07-30 09:06:35

  • Labels added: ProjectAddressSanitizer

@morehouse
Copy link
Contributor

No longer reproduces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Priority-Medium ProjectAddressSanitizer Status-Accepted Type-Defect
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ramosian-glider @morehouse