My favorites | Sign in
Logo
          
New issue | Search
for
| Advanced search | Search tips
Issue 504: 504 Gateway error
9 people starred this issue and may be notified of changes. Back to list
Status:  Started
Owner:  alobbs
Type-Defect
Priority-High
OpSys-All
Component-Logic


Sign in to add a comment
 
Reported by plundis@areaindex.com, Jun 23, 2009
What steps will reproduce the problem?
1. Run Cherokee in reverse proxy mode
2. Randomly see 504 Gateway error even though backend server seems fine and
responsive.

What is the expected output? What do you see instead?
Expect to see content from the backend server behind Cherokee.

What version of the product are you using? On what operating system?
.99-19 Ubuntu 8.04

Please provide any additional information below.
Few items to improve with the 504 Gateway error:
   1. Are we retrying the processing of the request more than once before
throwing this error?
 
   2. Can we provide a user setting to provide a custom HTML screen should
such error occur?
Comment 1 by sciyoshi, Jul 16, 2009
I can confirm this behavior with Cherokee 0.99.20-1 in reverse proxy mode for a
Django backend.
Comment 2 by alobbs, Sep 14, 2009
 Issue 503  has been merged into this issue.
Comment 3 by alobbs, Sep 14, 2009
I've been investigating this issue for a while.

So far, it's been quite tough to reproduce. Do you guys know some way to get the server to fail and return 
those 50x errors?

The change-set 3661 could help , although I couldn't confirm it quite yet:

    http://svn.cherokee-project.com/changeset/3661

Any feedback on this issue is more than welcome.
Status: Accepted
Owner: alobbs
Labels: Type-Defect Priority-High OpSys-All Component-Logic
Comment 4 by jjamor, Sep 21, 2009
I can confirm the bug with cherokee 0.99.22 without modifications, running under
Opensolaris 2009.06 and with several backends: drupal, dokuwiki, cgi and static. The
website is under test and it has no load at all.

I will test latest changes and report here.
Comment 5 by windspirit, Oct 11, 2009
Another confirmation
I have 50x errors in 

cherokee-0.99.22 + gentoo + fcgi(via 127.0.0.1:9000) + php + apc 

and I dont know how to reproduce it.
Comment 6 by ste...@konink.de, Oct 11, 2009
So why don't we all start to upgrade to at least the latest stable version and try again?
Comment 7 by windspirit, Oct 13, 2009
unfortunately, gentoo's portage was a little bit outdated and there was only 0.99.22

but after I created new ebuild and installed cherokee 0.99.24, it got much better!

I noticed only 1 gateway-timeout error during a whole day of heavy usage =)
and i belive it was not a cherokee fault that time.

Comment 8 by no.andrey, Oct 27, 2009
Guys, the 504 error really annoys me. And it's not backend problem. Debian 5, cherokee 0.99.22.
Comment 9 by ste...@konink.de, Oct 27, 2009
I guess Comment 6 still applies ;)
Comment 10 by alobbs, Oct 27, 2009
It seems there's people who have hit the issue with Cherokee 0.99.25. Looks like we haven't caught the bug yet.
Comment 11 by skarcha, Oct 28, 2009
Mmmmm... I'm getting 504 errors after 0.99.25 update... But I'm not using proxy, only
one Information Source for PHP.

I didn't remember what trunk revision I was using before... argh! :-(

Comment 12 by alobbs, Oct 28, 2009
Antonio, have you found a consistent way to reproduce the issue?
Yesterday I spent a whole lot of time trying to reproduce the issue with no luck.
Comment 13 by pubcrawler.com, Oct 28, 2009
Has anyone considered building an entirely new configuration file from scratch to
repeat your existing config?

It's worth trying. I think it might have something to do with the random errors
somehow- just unsure how.

Anyone want to test this theory on a Cherokee install that is failing?
Comment 14 by skarcha, Oct 28, 2009
Álvaro: no, I updated Cherokee on my production server from an unknown trunk revision
to HEAD, and after that I'm getting aleatory 504 errors.

Comment 15 by tahers, Oct 28, 2009
My issues with this matter are like Skarcha's. No proxy involved.

100% information source related, PHP in this case. Manually killing the idle php-cgi
instances immediately restores responsiveness.

Comment 16 by lnunez, Oct 28, 2009
php-cgi + 0.99.25-1~hardy~ppa1 
I have 1 php information source and no problems
Comment 17 by alobbs, Oct 28, 2009
Let's see whether this patch helps us to locate the issue.
I haven't managed to reproduce it more than 2 or 3 random times..
trace_504_fcgi-1.diff
1000 bytes   View   Download
Comment 18 by skarcha, Oct 29, 2009
Patch applied...
Comment 19 by skarcha, Oct 29, 2009
I just commited a patch (r3758: http://svn.cherokee-project.com/changeset/3758) to 
fix (yes, again... :( ) POST issues.

My previous patches (r3655 and r3657) were bad... sorry guys! I don't know how my 
tests works before I commited them.

Maybe it doesn't help with this bug, but some PHP software (like rgtui) works without 
504 errors now.

Anyone could give it a try?

Comment 20 by alobbs, Oct 30, 2009
Just uploaded a related QA test:

  http://svn.cherokee-project.com/browser/cherokee/trunk/qa/231-POST-4extra.py

Status: WaitingQA
Comment 21 by mictlan, Nov 18, 2009
I don't know if this is the same issue, but I have laaaaarge php scripts, I have this:

Linux arq-mpacheco 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009
x86_64 GNU/Linux

Running:

Cherokee Web Server 0.99.27

I configured the php.ini of the cgi (e.j. max_execution_time=180), enough for apache.

I try with two scripts:

<?php

        sleep(15);

seems to work

<?php

        sleep(20);

gets a 504 Gateway Timeout

what should I do?
Comment 22 by mictlan, Nov 18, 2009
Sorry, my issue actually it's 499
(http://code.google.com/p/cherokee/issues/detail?id=499&can=1&q=gateway%20error)
Comment 23 by alobbs, Nov 21, 2009
 Bug 499  is most likely tightly related to this one:

  http://code.google.com/p/cherokee/issues/detail?id=499

Could you please guys try to set a higher timeout in the FastCGI and SCGI rules? Odds
are that will solve the issue, although I'm not certain. Some testing is required..
Comment 24 by jjamor, Nov 24, 2009
I just discovered that my reverse proxy (cherokee 0.99.29) usually sends a 5xx
gateway timeout or bad gateway when trying to POST a big text (when editing a wiki,
for example).


Comment 25 by ste...@konink.de, Nov 25, 2009
Could this possibly be related to the maximum header length?
Comment 26 by alobbs, Nov 25, 2009
Something funny must be happening in the post manager. I'm checking it out right away..
Status: Started
Comment 27 by alobbs, Nov 25, 2009
@jjamor: A few days ago, I committed a few patches for adding custom timeout support
to rule entries.  That means, a specific rule can define a timeout value, as it can
define a encoding method, o a handler plug-in.

The target was to allow PHP connections to last longer and the general timeout limit,
so the server wouldn't reply a 504 error if the php-cgi was still (hanged) processing
the request when the timeout limit was reached.

I've just realized that, you might be hitting that issue in the back-end servers.
Depending on the wiki software processing speed, the back-end web server might be
returning 504 errors whenever that process takes longer than the general timeout
limit. In that case, the front-end proxy server would be just relying it to the
outside world.

I have not probed that theory yet, but it makes good sense to me.

The best way to test it would be to add a custom timeout to the Extension PHP rule of
the back-end servers. A 35 seconds value would be enough (since php.ini sets 30
seconds of processing limit by default).  Hopefully that will fix the issue.

Cherokee 0.99.30 will ship an improved version of the PHP wizard. It checks the
php.ini file in order to figure the right timeout value. In that way it can set the
right timeout value to the rule, so it does not return 504 errors when php scripts
takes a "long" time to be executed (longer than the server-wide timeout limit).

BTW, I'm talking about PHP, but the underlying idea applies to any other scripting
language as well.
Comment 29 by jjamor, Nov 25, 2009
@alobbs, does this apply to my backend, which is using Moin (python w/fcgi, not
php)? in any case, I have tested to directly submitting the big POSTs to the backend,
and the errors do not occur.

Comment 30 by skarcha, Nov 25, 2009
@jjamor, I think you must apply to your backends and to the reverse proxy rule on the 
proxy server.
Comment 31 by jjamor, Nov 25, 2009
@alobbs, @skarcha: finally the backend was moved to apache, because it was impossible
to get work it with the spawn-fcgi binary (see issue 591). However, the 5xx errors
are still here but only when using the reverse proxy (with Cherokee).

I'm going to try compilation of the SVN version and test it ...
Comment 32 by jjamor, Nov 25, 2009
Ok, compiled and setup a timeout of 60 seconds.

The problem still exists: sometimes the proxy server respond with a Bad gateway
error. The response is immediate. By analyzing the logs, the backend server always
responds correctly.

I'm going to switch the reverse proxy to apache until the problem is really fixed.

Comment 33 by skarcha, Feb 11, 2010
@jjamor: Do you still have this problem?

Comment 34 by jjamor, Feb 12, 2010
I am running  cherokee 0.99.41 since 2010-01-26 and I did not need to restart it
since then. However sometimes previous versions worked normally a month or so and
then started to serve 5xx errors until it is restarted. I'd wait some weeks more
before closing this issue.
Comment 35 by skarcha, Feb 12, 2010
Great! Thanks for the follow up ;)
Comment 36 by tahers, Jun 07, 2010
It seems this was finally fixed in February. Can anybody confirm it so we can close
the bug? Thanks guys.
Comment 37 by hendriks.luuk, Jul 12, 2010
I think I'm suffering this issue now. With a fresh Drupal install, going through the installation steps ends up in a 504 when adding the 'Administrator account' (last step). After that, the Drupal install is available though. But when I try to register a new account, I end up with a 504 after the submit. The error log doesn't show anything. I will provide a trace in a few minutes.

Software and versions:
Cherokee 1.0.5 (from source)
Drupal 6.17 (used the Cherokee-admin wizard to configure it)
Ubuntu Linux (10.04, aptitude updated just today)
php-cgi -v:
PHP 5.3.2-1ubuntu4.2 with Suhosin-Patch (cgi-fcgi)
Comment 38 by hendriks.luuk, Jul 12, 2010
Here is the trace. (used CHEROKEE_TRACE="all")
cherokee_504_trace.txt
17.7 KB   View   Download
Comment 39 by alobbs, Jul 12, 2010
@hendriks.luuk: so, it took ~30 seconds until the error message showed up, didn't it?
Comment 40 by hendriks.luuk, Jul 12, 2010
Indeed, I checked it with another user registration attempt, it took 30 seconds before the 504 showed up.
Comment 41 by hendriks.luuk, Jul 22, 2010
I tried the same thing on CentOS with Cherokee 1.0.0 (from the EPEL repo) and PHP 5.2 (from Utter-ramblings). No 504 with that configuration!
Comment 42 by henrik.eneroth, Aug 25, 2010
I'm having the same issue when performing a long operation (downloading and adding user avatars to databasse). The task itself (PHP) finishes, but the webserver gives up after almost exactly 30 seconds (with the infamous 504 Gateway timeout error)
Comment 43 by henrik.eneroth, Aug 25, 2010
Oh, using Cherokee 1.08 on OS X 10.6.
Comment 44 by ste...@konink.de, Aug 25, 2010
So what is your php execution timeout?
Sign in to add a comment

Powered by Google Project Hosting