My favorites | Sign in
Logo
             
New issue | Search
for
| Advanced search | Search tips
Issue 18488: Possible data race on HistoryAddPageArgs
7 people starred this issue and may be notified of changes. Back to list
 
Reported by timur...@gmail.com, Aug 05, 2009
Chrome Version       : r22179
OS + version : Linux, possibly Mac as well

Hi,
I've been running ui_tests under ThreadSanitizer and got the following race
report:

==21334== INFO: T0 is program's main thread
==21334== INFO: T7 has been created by T0 at this point: {{{
==21334==     #0  clone /lib32/libc-2.7.so
==21334==     #1  pthread_create@@GLIBC_2.1 /lib32/libpthread-2.7.so
==21334==     #2  pthread_create@*
/home/timurrrr/valgrind-patches/valgrind-10695/tsan/ts_valgrind_intercepts.c:556
==21334==     #3  (anonymous namespace)::CreateThread(unsigned int, bool,
PlatformThread::Delegate*, unsigned long*) base/platform_thread_posix.cc:93
==21334==     #4  PlatformThread::Create(unsigned int,
PlatformThread::Delegate*, unsigned long*) base/platform_thread_posix.cc:105
==21334==     #5  base::Thread::StartWithOptions(base::Thread::Options
const&) base/thread.cc:84
==21334==     #6  base::Thread::Start() base/thread.cc:73
==21334==     #7  HistoryService::Init(FilePath const&, BookmarkService*)
chrome/browser/history/history.cc:128
==21334==     #8 
ProfileImpl::GetHistoryService(Profile::ServiceAccessType)
chrome/browser/profile.cc:862
==21334==     #9  VisitedLinkMaster::RebuildTableFromHistory()
chrome/browser/visitedlink_master.cc:868
==21334==     #10 VisitedLinkMaster::InitFromScratch(bool)
chrome/browser/visitedlink_master.cc:622
==21334==     #11 VisitedLinkMaster::Init()
chrome/browser/visitedlink_master.cc:265
==21334==     #12 ProfileImpl::GetVisitedLinkMaster()
chrome/browser/profile.cc:721
==21334==     #13 BrowserRenderProcessHost::InitVisitedLinks()
chrome/browser/renderer_host/browser_render_process_host.cc:576
==21334==     #14 BrowserRenderProcessHost::Init()
chrome/browser/renderer_host/browser_render_process_host.cc:492
==21334==     #15 RenderViewHost::CreateRenderView()
chrome/browser/renderer_host/render_view_host.cc:174
==21334== WARNING: Possible data race during read of size 4 at 0xB79C8F0: {{{
==21334==    T0 (locks held: {}):
==21334==     #0  std::vector<GURL, std::allocator<GURL> >::~vector()
/usr/include/c++/4.2/bits/stl_vector.h:268
==21334==     #1  history::HistoryAddPageArgs::~HistoryAddPageArgs()
chrome/browser/history/history_marshaling.h:21
==21334==     #2  base::RefCounted<history::HistoryAddPageArgs>::Release()
base/ref_counted.h:80
==21334==     #3 
scoped_refptr<history::HistoryAddPageArgs>::~scoped_refptr()
base/ref_counted.h:196
==21334==     #4  HistoryService::AddPage(GURL const&, base::Time, void
const*, int, GURL const&, unsigned int, std::vector<GURL,
std::allocator<GURL> > const&, bool) chro»
==21334==     #5  HistoryService::AddPage(GURL const&, void const*, int,
GURL const&, unsigned int, std::vector<GURL, std::allocator<GURL> > const&,
bool) chrome/browser/h»
==21334==     #6  TabContents::UpdateHistoryForNavigation(GURL const&,
NavigationController::LoadCommittedDetails const&,
ViewHostMsg_FrameNavigate_Params const&) chrome/b»
==21334==     #7  TabContents::DidNavigate(RenderViewHost*,
ViewHostMsg_FrameNavigate_Params const&)
chrome/browser/tab_contents/tab_contents.cc:1848
==21334==     #8  RenderViewHost::OnMsgNavigate(IPC::Message const&)
chrome/browser/renderer_host/render_view_host.cc:944
==21334==     #9  RenderViewHost::OnMessageReceived(IPC::Message const&)
chrome/browser/renderer_host/render_view_host.cc:723
==21334==   Concurrent write(s) happened at (OR AFTER) these points:
==21334==    T7 (locks held: {}):
==21334==     #0  std::vector<GURL, std::allocator<GURL>
>::erase(__gnu_cxx::__normal_iterator<GURL*, std::vector<GURL,
std::allocator<GURL> > >) /usr/include/c++/4.2/bits»
==21334==     #1 
history::HistoryBackend::AddPage(scoped_refptr<history::HistoryAddPageArgs>) chrome/browser/history/history_backend.cc:404
==21334==     #2  void DispatchToMethod<history::HistoryBackend, void
(history::HistoryBackend::*)(scoped_refptr<history::HistoryAddPageArgs>),
scoped_refptr<history::Hist»
==21334==     #3  RunnableMethod<history::HistoryBackend, void
(history::HistoryBackend::*)(scoped_refptr<history::HistoryAddPageArgs>),
Tuple1<scoped_refptr<history::Hist»
==21334==     #4  MessageLoop::RunTask(Task*) base/message_loop.cc:316
==21334==     #5 
MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&)
base/message_loop.cc:324
==21334==     #6  MessageLoop::DoWork() base/message_loop.cc:435
==21334==     #7 
base::MessagePumpDefault::Run(base::MessagePump::Delegate*)
base/message_pump_default.cc:23
==21334==     #8  MessageLoop::RunInternal() base/message_loop.cc:199
==21334==     #9  MessageLoop::RunHandler() base/message_loop.cc:181
==21334== }}}

Please note that HistoryAddPageArgs subclasses base::RefCounted, NOT
base::RefCountedThreadSafe
(chrome/browser/history/history_marshaling.h)
 20 // Marshalling structure for AddPage.
 21 class HistoryAddPageArgs : public base::RefCounted<HistoryAddPageArgs> {
 22  public:

and scoped_ptr<HistoryAddPageArgs> are shared between multiple threads:
(chrome/browser/history/history.cc, executed in Thread #0)
 295 
 296   scoped_refptr<history::HistoryAddPageArgs> request(
 297       new history::HistoryAddPageArgs(url, time, id_scope, page_id,
 298                                       referrer, redirects, transition,
 299                                       did_replace_entry));
 300   ScheduleAndForget(PRIORITY_NORMAL, &HistoryBackend::AddPage, request);
 301 }

(chrome/browser/history/history_backend.cc, executed in Thread #7)
 321 
 322 void HistoryBackend::AddPage(scoped_refptr<HistoryAddPageArgs> request) {
 323   DLOG(INFO) << "Adding page " << request->url.possibly_invalid_spec();
 324 

I believe the fix is simple: just replace base::RefCounted with
base::RefCountedThreadSafe

Thank you,
Timur Iskhodzhanov
Comment 1 by daniel.r.kegel, Aug 05, 2009
A cl with your suggested fix is up at http://codereview.chromium.org/160642

Do you know which ui tests in particular tickle this?
(It's nice to include a single test name in bug reports to make reproducing
easier, if possible.  That's why I use tools/valgrind/shard-all-tests.sh
or the like to run the tests one at a time when fishing for valgrind warnings.)

Comment 2 by timur...@gmail.com, Aug 06, 2009
It seems like HostResolver in net/base/host_resolver.h has a similar data race
Comment 3 by timur...@gmail.com, Aug 13, 2009
Same as UserScriptMaster in chrome/browser/extensions/user_script_master.h
Comment 4 by timur...@gmail.com, Aug 14, 2009
Same as LoadLog in net/base/load_log.h
Comment 5 by mal.chromium, Aug 21, 2009
Marking started since there's a patch out for review. 

@brettw: BTW, you're listed as the reviewer for that patch :)
Status: Started
Owner: d...@chromium.org
Cc: bre...@chromium.org
Labels: -Area-Misc Area-BrowserBackend
Comment 6 by brettw@chromium.org, Aug 21, 2009
I don't see any patches in my queue or email for this. If you sent it out, please check 
my address and re-send the email. Thanks!
Comment 7 by timur...@gmail.com, Sep 17, 2009
This may be related to http://code.google.com/p/chromium/issues/detail?id=15577
Comment 8 by brettw@chromium.org, Sep 17, 2009
What is the status of this bug? It's been open for a month and a half. Is it being 
worked on?
Comment 9 by timur...@gmail.com, Sep 17, 2009
The bugs are still opened.

I'll prepare a changelist now for all 4 races described above.
Comment 10 by timur...@gmail.com, Sep 17, 2009
http://codereview.chromium.org/215011 reviewers are welcome
Comment 11 by bugdroid1@chromium.org, Sep 17, 2009
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=26473 

------------------------------------------------------------------------
r26473 | laforge@chromium.org | 2009-09-17 13:24:45 -0700 (Thu, 17 Sep 2009) | 3 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/branches/211/src/net/base/host_resolver.h?r1=26473&r2=26472

Fixed a few data races on reference counters.
BUG=18488
http://codereview.chromium.org/215011/
------------------------------------------------------------------------

Comment 12 by bugdroid1@chromium.org, Sep 17, 2009
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=26474 

------------------------------------------------------------------------
r26474 | laforge@chromium.org | 2009-09-17 13:25:00 -0700 (Thu, 17 Sep 2009) | 3 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/branches/211/src/chrome/browser/extensions/user_script_master.h?r1=26474&r2=26473
   M http://src.chromium.org/viewvc/chrome/branches/211/src/chrome/browser/history/history_marshaling.h?r1=26474&r2=26473

Fixed a few data races on reference counters.
BUG=18488
http://codereview.chromium.org/215011/
------------------------------------------------------------------------

Comment 13 by bugdroid1@chromium.org, Sep 17, 2009
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=26476 

------------------------------------------------------------------------
r26476 | timurrrr@chromium.org | 2009-09-17 13:36:29 -0700 (Thu, 17 Sep 2009) | 3 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/extensions/user_script_master.h?r1=26476&r2=26475
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/history_marshaling.h?r1=26476&r2=26475
   M http://src.chromium.org/viewvc/chrome/trunk/src/net/base/host_resolver.h?r1=26476&r2=26475

Fixed a few data races on reference counters.
BUG=18488
Review URL: http://codereview.chromium.org/215011
------------------------------------------------------------------------

Comment 14 by shess@chromium.org, Sep 17, 2009
I find myself wondering if cases like RefCounted when you should have RefCountedThreadSafe couldn't be 
enforced using template magic.  But after experimenting, I find that my C++ Fu is not nearly up to it.  Is that 
kind of thing possible?
Comment 15 by brettw@chromium.org, Sep 17, 2009
Pawel is looking at asserting that RefCounted is being used from the same thread.
Comment 16 by scarybeasts, Sep 17, 2009
Nice detective work, Timur.
Do you have any theory on how this could cause corruption (such as 
http://code.google.com/p/chromium/issues/detail?id=15577) ?

If both threads collide in accessing the reference object, is there a case whereby a 
double-free can occur? Or a use-after-free?
My reading of the code is that Thread 0 elevates the reference count to 2 (local 
stack frame scoped_refptr and scoped_refptr copy in the new Task* object) before a 
second thread becomes involved.
Admittedly it is hard to reason about because the scoped_refptr is passed around a 
lot to various templates and functions -- sometimes by value (will increment) and 
sometimes by references (will not increment).
Comment 17 by timur...@gmail.com, Sep 18, 2009
> Pawel is looking at asserting that RefCounted is being used from the same thread.
Please, NO.
I'm sure there are dozens of classes using RefCounted in multi-threaded environment
properly in Chromium.
For example, by guarding it using a Lock.

Maybe a better idea is to add a couple of "DCHECK(refcount > 0)" and also set
"refcount = -1000" before "delete this"
Comment 18 by brettw, Sep 18, 2009
I thought about whether there would be a lot of legitimate uses, and decided it's 
unlikely. In your example, we don't use locks much. It would be interesting to know if 
we do do this a lot.
Comment 19 by timur...@gmail.com, Sep 18, 2009
Consider two threads running the same code (counter is a shared variable, initially
set to 0):

void ThreadProc() {
  for (int i = 0; i < 100000; i++)
    counter++;
}

experiments show that counter becomes 19XXXX, not 200000.

Now consider a data race on reference counter.
If two increments or decrements happen simultaneously, the reference counter may
become +1 or -1 to the correct value.
In case of "+1", we are unlikely to achieve zero refcount at all, so we are likely to
have a memory leak.
In case of "-1", reference counter may become zero more than once; and it can also
become -1.

For example, consider the following interleaving (T1, T2 = thread{1,2})
T1|T2| refcount
XXXXX|  0
++|  |  1
++|++|!!2, not 3!!
--|  |  1, not 2
  |--|  0, not 1   -> free is called
++|  |  1, not 2
--|  |  0, not 1   -> free is called the second time
--|  | -1, not 0

Please note that there is an equal number of "++" and "--" in this interleaving
Comment 20 by timur...@gmail.com, Sep 18, 2009
the last comment is for scarybeasts :-)
Comment 21 by daniel.r.kegel, Sep 18, 2009
It might still be interesting to get a list of RefCounted instances
which are used from multiple threads.  We could add an optional warning
for that and check the log from a full test suite run.  If the list
isn't horribly huge, maybe it'd be easy to eyeball the callers and
see which need locks or the like.
Comment 22 by timur...@gmail.com, Sep 18, 2009
You can get it by running ThreadSanitizer :-)
Comment 23 by bugdroid1@chromium.org, Sep 18, 2009
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=26593 

------------------------------------------------------------------------
r26593 | timurrrr@chromium.org | 2009-09-18 11:09:05 -0700 (Fri, 18 Sep 2009) | 2 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/tools/valgrind/tsan/suppressions.txt?r1=26593&r2=26592

Remove a suppression for  bug 18488  since it should be fixed already.
Review URL: http://codereview.chromium.org/215021
------------------------------------------------------------------------

Comment 24 by bugdroid1@chromium.org, Sep 21, 2009
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=26755 

------------------------------------------------------------------------
r26755 | cpu@chromium.org | 2009-09-21 15:39:09 -0700 (Mon, 21 Sep 2009) | 8 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/extensions/user_script_master.h?r1=26755&r2=26754

Making UserScriptMaster::ScriptReloader refcounted thread safe
- Being addref() in two different threads.

BUG=18488
TEST=none


Review URL: http://codereview.chromium.org/213025
------------------------------------------------------------------------

Comment 25 by bugdroid1@chromium.org, Sep 22, 2009
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=26883 

------------------------------------------------------------------------
r26883 | laforge@chromium.org | 2009-09-22 16:45:59 -0700 (Tue, 22 Sep 2009) | 6 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/branches/195/src/chrome/browser/extensions/user_script_master.h?r1=26883&r2=26882
   M http://src.chromium.org/viewvc/chrome/branches/195/src/chrome/browser/history/history_marshaling.h?r1=26883&r2=26882
   M http://src.chromium.org/viewvc/chrome/branches/195/src/net/base/host_resolver.h?r1=26883&r2=26882

Merge 26476 - Fixed a few data races on reference counters.
BUG=18488
Review URL: http://codereview.chromium.org/215011

TBR=timurrrr@chromium.org
Review URL: http://codereview.chromium.org/209073
------------------------------------------------------------------------

Comment 26 by timur...@gmail.com, Sep 23, 2009
Please note that some data races on reference counters in LoadLog are still present.
I've created a separate issue for them
http://code.google.com/p/chromium/issues/detail?id=22272
Comment 27 by huanr@chromium.org, Oct 08, 2009
 Issue 11492  has been merged into this issue.
Cc: hu...@chromium.org sh...@chromium.org c...@chromium.org
Comment 28 by huanr@chromium.org, Oct 08, 2009
(No comment was entered for this change.)
Owner: timur...@chromium.org
Cc: d...@chromium.org
Labels: -OS-Linux OS-All Valgrind Crash Pri-1
Comment 29 by huanr@chromium.org, Oct 13, 2009
(No comment was entered for this change.)
Labels: Valgrind-Tsan
Comment 30 by sky@chromium.org, Oct 13, 2009
Shouldn't this bug be marked fixed no?
Comment 31 by timur...@chromium.org, Oct 13, 2009
I think so. Thanks for mentioning!
Status: Fixed
Comment 32 by timur...@chromium.org, Oct 15, 2009
(No comment was entered for this change.)
Labels: -Valgrind-Tsan ThreadSanitizer
Comment 33 by mal.chromium, Dec 18, 2009
(No comment was entered for this change.)
Labels: -Area-BrowserBackend Area-Internals
Sign in to add a comment

Powered by Google Project Hosting