My favorites | Sign in
Project Home Downloads Wiki Issues Code Search
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 27400: Long lived connections are being dropped by some network setups
6 people starred this issue and may be notified of changes. Back to list
 
Project Member Reported by willchan@chromium.org, Nov 11, 2009
This bug has been forked from  issue 16367 .  Refer to that bug for historical 
information.

Summary:
If the server supports HTTP keep alive, then Chromium will keep around an 
idle socket for up to 10 minutes.  After the first HTTP response completes 
and the socket goes idle, the server sends TCP keep alive packets, which the 
host OS will ACK.  It appears that in certain network setups (the Google 
Reston office for example), after a certain period of time (128~ seconds at 
Reston) the network will start dropping all packets for the connection.  
Wireshark dumps indicate that the client does not receive anymore packets 
from the server for this TCP connection.  If Chromium tries to send any more 
data over this TCP connection (if the user browses to the same host again and 
requests non-cached data), the TCP packets do not seem to make it out to the 
server.  We conclude this because the server replies with neither ACKs nor 
RSTs.  The host OS tries a number of TCP retransmits before sending a TCP RST 
and closing the socket connection.  At this point, the OS sends an error to 
the application (Chromium), and the Chromium network stack will open a new 
socket connection to handle the request and that will succeed.  User visible 
symptoms are a spinning throbber for a few minutes until the OS gives up on 
the TCP connection and returns an error to Chromium, after which a new 
connection will be opened and Chromium will successfully download and display 
the page.

TODO:
Compare Chromium network behavior to Firefox & Safari using wireshark.  
Determine if there's a way we can prevent the network from killing long lived 
TCP connections (perhaps with client-side TCP keepalives?).  Otherwise, 
consider keeping idle sockets for a shorter amount of time, although we 
should run A/B experiments to make sure performance isn't adversely affected.
Dec 17, 2009
#1 or...@chromium.org
Replacing labels:
   Area-BrowserBackend by Area-Internals

Labels: -Area-BrowserBackend Area-Internals
Mar 18, 2010
#2 darin@chromium.org
(No comment was entered for this change.)
Labels: Internals-Network
Mar 24, 2010
#3 lafo...@chromium.org
Recommendations based on labels Internals-Network and os-all from activity in the last 60 days
	phajdan.jr@chromium.org
	willchan@chromium.org(Recommended)
	wtc@chromium.org
	eroman@chromium.org
	rvargas@chromium.org
	vandebo@chromium.org
Mar 24, 2010
#4 lafo...@chromium.org
This an automated note...
Recommendations based on labels Internals-Network and os-all from activity in the last 60 days
	phajdan.jr@chromium.org
	willchan@chromium.org(Recommended)
	wtc@chromium.org
	eroman@chromium.org
	rvargas@chromium.org
	vandebo@chromium.org
Labels: MrBotRecommends
Mar 29, 2010
#5 lafo...@chromium.org
(No comment was entered for this change.)
Status: Assigned
Owner: willc...@chromium.org
Labels: -Mstone-5 Mstone-6
Jun 29, 2010
#6 willchan@chromium.org
Punting to Mstone-7.  I might try to swing by the Reston office in August/Sept during my next east coast trip and will look at this then.
Labels: -Mstone-6 Mstone-7
Aug 27, 2010
#7 willchan@chromium.org
Punting again, Mstone-7 deadline was sooner than I thought.
Labels: -Mstone-7 Mstone-8
Oct 7, 2010
#8 willchan@chromium.org
Punt again :)
Labels: -Mstone-8 Mstone-X
Dec 14, 2010
#9 pacr...@gmail.com
Much as I've grown to like Chromium its nearly useless on our University network where the routers to the outside world silently kill persistent HTTP connections that are held open and inactive for more than 2 minutes.

We can fix this for Firefox by chaining network.http.keep-alive.timeout from 300 to 60 seconds.  Can we get a similar fix for Chromium?  (Yeah okay we should fix the routers but as it works for IE no-one in administration is likely to do anything for at least a decade).

Jan 7, 2011
#10 willchan@chromium.org
Oops, I seem to have missed comment #9.  I examined the code and it looks like we actually are not setting SO_KEEPALIVE.  We should probably set it and pick some "reasonable" value (30s?) for TCP keep alives.  It seems some routers are lowering their idle TCP connection timeouts to 60s or so, so I picked 30s to help mitigate this.  I don't think TCP keep alives at 30s intervals will substantially impact network traffic, but am open to discussion here.
Jan 14, 2011
#11 willchan@chromium.org
 Issue 68045  has been merged into this issue.
Jan 14, 2011
#12 willchan@chromium.org
+mbelshe who disagrees that we should add SO_KEEPALIVE.

This is the same argument as usual.  Should Chrome be more lenient to work around (arguably) buggy routers?  Or should we be stricter and act as a forcing function for "bad" routers out there to stop having such short keep alive timers?

I feel like TCP keep alives are pretty benign and don't lead to much extra traffic, so I'm going to just add them, unless people tell me otherwise.  If you disagree, this is your chance to chime in and tell me no.
Cc: mbel...@chromium.org
Labels: -Mstone-X Mstone-11
Jan 14, 2011
#13 bugdro...@chromium.org
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=71482

------------------------------------------------------------------------
r71482 | willchan@chromium.org | Fri Jan 14 12:41:27 PST 2011

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/net/socket/tcp_client_socket_libevent.cc?r1=71482&r2=71481&pathrev=71482

Set TCP keep alive on Linux and Mac.

On Linux, also set the timeouts.

BUG=27400
TEST=Start wireshark and capture packets.  Start chrome.  Open www.facebook.com.  Let it sit idle for awhile.  Wait for TCP keep alive packets to be sent out after 45 seconds on Linux.

Review URL: http://codereview.chromium.org/6162005
------------------------------------------------------------------------
Jan 18, 2011
#14 wtc@chromium.org
willchan: have you done the TODO in comment 0?

  TODO:
  Compare Chromium network behavior to Firefox & Safari using wireshark.

I searched in the Firefox 4 source tree and saw no code that sets
the SO_KEEPALIVE socket option:
http://mxr.mozilla.org/mozilla-central/search?string=SO_KEEPALIVE
http://mxr.mozilla.org/mozilla-central/ident?i=PR_SockOpt_Keepalive

Does Firefox suffer from the same problem?
Jan 19, 2011
#15 willchan@chromium.org
Yes, firefox suffers from the same problem.
Jan 24, 2011
#16 bugdro...@chromium.org
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=72421

------------------------------------------------------------------------
r72421 | willchan@chromium.org | Mon Jan 24 15:41:01 PST 2011

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/net/socket/tcp_client_socket_win.cc?r1=72421&r2=72420&pathrev=72421

Enable TCP Keep-Alive packets for Windows.

45s until first TCP Keep-Alive and 45s in between keep alives.

BUG=27400
TEST=Run Wireshark, make sure TCP Keep-Alive packets are sent.

Review URL: http://codereview.chromium.org/6300013
------------------------------------------------------------------------
Jan 24, 2011
#17 willchan@chromium.org
OK, I just landed the Windows support. So TCP keep alives are on for all platforms, at 45s intervals, except for Mac. It looks like it's possible to do something for OS X 10.6 (er, this may also exist on 10.5, I'm not sure) (http://developer.apple.com/library/mac/#DOCUMENTATION/Darwin/Reference/ManPages/man4/tcp.4.html). It only allows for setting the initial wait until TCP keep alives, but not the intervals in between keep alives.

I'm closing as fixed now, since there are solutions for all systems now, although the Mac solution may require modifying system settings: http://www.gnugk.org/keepalive.html
Status: Fixed
Jan 24, 2011
#18 wtc@chromium.org
willchan: do the TCP keep alive packets prevent a mobile device
from becoming idle and going to sleep?  Can you find out the
model number of the NAT router in question?  (I won't insist on
spending the time to report this bug to the NAT router's
manufacturer.)  A workaround for an unspecified device will be
very difficult to remove.
Jan 24, 2011
#19 willchan@chromium.org
I suspect the mobile device will just go to sleep.

From one report I've analyzed, this bug occurred when the user upgraded to the latest version of OpenWRT firmware.
Jan 24, 2011
#20 bugdro...@chromium.org
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=72426

------------------------------------------------------------------------
r72426 | willchan@chromium.org | Mon Jan 24 16:31:35 PST 2011

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/net/socket/tcp_client_socket_win.cc?r1=72426&r2=72425&pathrev=72426

Revert 72421 (broke tests) - Enable TCP Keep-Alive packets for Windows.

45s until first TCP Keep-Alive and 45s in between keep alives.

BUG=27400
TEST=Run Wireshark, make sure TCP Keep-Alive packets are sent.

Review URL: http://codereview.chromium.org/6300013

TBR=willchan@chromium.org
Review URL: http://codereview.chromium.org/6371010
------------------------------------------------------------------------
Status: Assigned
Jan 24, 2011
#21 bugdro...@chromium.org
The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=72429

------------------------------------------------------------------------
r72429 | willchan@chromium.org | Mon Jan 24 16:48:31 PST 2011

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/net/socket/tcp_client_socket_win.cc?r1=72429&r2=72428&pathrev=72429

Reland 72421 - Enable TCP Keep-Alive packets for Windows.

Dropped the DCHECk since it seems some Windows version don't initialize the unused variable.

BUG=27400
TEST=none

Review URL: http://codereview.chromium.org/6242011
------------------------------------------------------------------------
Jan 25, 2011
#22 willchan@chromium.org
(No comment was entered for this change.)
Status: Fixed
Oct 12, 2012
#23 bugdro...@chromium.org
This issue has been closed for some time. No one will pay attention to new comments.
If you are seeing this bug or have new data, please click New Issue to start a new bug.
Labels: Restrict-AddIssueComment-Commit
Mar 10, 2013
#24 bugdro...@chromium.org
(No comment was entered for this change.)
Labels: -Mstone-11 -Area-Internals -Internals-Network Cr-Internals Cr-Internals-Network M-11
Mar 13, 2013
#25 bugdro...@chromium.org
(No comment was entered for this change.)
Labels: -Restrict-AddIssueComment-Commit Restrict-AddIssueComment-EditIssue
Sign in to add a comment

Powered by Google Project Hosting