| Issue 504: | 504 Gateway error | |
| 9 people starred this issue and may be notified of changes. | Back to list |
Sign in to add a comment
|
What steps will reproduce the problem? 1. Run Cherokee in reverse proxy mode 2. Randomly see 504 Gateway error even though backend server seems fine and responsive. What is the expected output? What do you see instead? Expect to see content from the backend server behind Cherokee. What version of the product are you using? On what operating system? .99-19 Ubuntu 8.04 Please provide any additional information below. Few items to improve with the 504 Gateway error: 1. Are we retrying the processing of the request more than once before throwing this error? 2. Can we provide a user setting to provide a custom HTML screen should such error occur? |
||||||||||||
,
Jul 16, 2009
I can confirm this behavior with Cherokee 0.99.20-1 in reverse proxy mode for a Django backend. |
|||||||||||||
,
Sep 14, 2009
Issue 503 has been merged into this issue. |
|||||||||||||
,
Sep 14, 2009
I've been investigating this issue for a while.
So far, it's been quite tough to reproduce. Do you guys know some way to get the server to fail and return
those 50x errors?
The change-set 3661 could help , although I couldn't confirm it quite yet:
http://svn.cherokee-project.com/changeset/3661
Any feedback on this issue is more than welcome.
Status: Accepted
Owner: alobbs Labels: Type-Defect Priority-High OpSys-All Component-Logic |
|||||||||||||
,
Sep 21, 2009
I can confirm the bug with cherokee 0.99.22 without modifications, running under Opensolaris 2009.06 and with several backends: drupal, dokuwiki, cgi and static. The website is under test and it has no load at all. I will test latest changes and report here. |
|||||||||||||
,
Oct 11, 2009
Another confirmation I have 50x errors in cherokee-0.99.22 + gentoo + fcgi(via 127.0.0.1:9000) + php + apc and I dont know how to reproduce it. |
|||||||||||||
,
Oct 11, 2009
So why don't we all start to upgrade to at least the latest stable version and try again? |
|||||||||||||
,
Oct 13, 2009
unfortunately, gentoo's portage was a little bit outdated and there was only 0.99.22 but after I created new ebuild and installed cherokee 0.99.24, it got much better! I noticed only 1 gateway-timeout error during a whole day of heavy usage =) and i belive it was not a cherokee fault that time. |
|||||||||||||
,
Oct 27, 2009
Guys, the 504 error really annoys me. And it's not backend problem. Debian 5, cherokee 0.99.22. |
|||||||||||||
,
Oct 27, 2009
I guess Comment 6 still applies ;) |
|||||||||||||
,
Oct 27, 2009
It seems there's people who have hit the issue with Cherokee 0.99.25. Looks like we haven't caught the bug yet. |
|||||||||||||
,
Oct 28, 2009
Mmmmm... I'm getting 504 errors after 0.99.25 update... But I'm not using proxy, only one Information Source for PHP. I didn't remember what trunk revision I was using before... argh! :-( |
|||||||||||||
,
Oct 28, 2009
Antonio, have you found a consistent way to reproduce the issue? Yesterday I spent a whole lot of time trying to reproduce the issue with no luck. |
|||||||||||||
,
Oct 28, 2009
Has anyone considered building an entirely new configuration file from scratch to repeat your existing config? It's worth trying. I think it might have something to do with the random errors somehow- just unsure how. Anyone want to test this theory on a Cherokee install that is failing? |
|||||||||||||
,
Oct 28, 2009
Álvaro: no, I updated Cherokee on my production server from an unknown trunk revision to HEAD, and after that I'm getting aleatory 504 errors. |
|||||||||||||
,
Oct 28, 2009
My issues with this matter are like Skarcha's. No proxy involved. 100% information source related, PHP in this case. Manually killing the idle php-cgi instances immediately restores responsiveness. |
|||||||||||||
,
Oct 28, 2009
php-cgi + 0.99.25-1~hardy~ppa1 I have 1 php information source and no problems |
|||||||||||||
,
Oct 28, 2009
Let's see whether this patch helps us to locate the issue. I haven't managed to reproduce it more than 2 or 3 random times.. |
|||||||||||||
,
Oct 29, 2009
Patch applied... |
|||||||||||||
,
Oct 29, 2009
I just commited a patch (r3758: http://svn.cherokee-project.com/changeset/3758) to fix (yes, again... :( ) POST issues. My previous patches (r3655 and r3657) were bad... sorry guys! I don't know how my tests works before I commited them. Maybe it doesn't help with this bug, but some PHP software (like rgtui) works without 504 errors now. Anyone could give it a try? |
|||||||||||||
,
Oct 30, 2009
Just uploaded a related QA test: http://svn.cherokee-project.com/browser/cherokee/trunk/qa/231-POST-4extra.py
Status: WaitingQA
|
|||||||||||||
,
Nov 18, 2009
I don't know if this is the same issue, but I have laaaaarge php scripts, I have this:
Linux arq-mpacheco 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009
x86_64 GNU/Linux
Running:
Cherokee Web Server 0.99.27
I configured the php.ini of the cgi (e.j. max_execution_time=180), enough for apache.
I try with two scripts:
<?php
sleep(15);
seems to work
<?php
sleep(20);
gets a 504 Gateway Timeout
what should I do?
|
|||||||||||||
,
Nov 18, 2009
Sorry, my issue actually it's 499 (http://code.google.com/p/cherokee/issues/detail?id=499&can=1&q=gateway%20error) |
|||||||||||||
,
Nov 21, 2009
Bug 499 is most likely tightly related to this one: http://code.google.com/p/cherokee/issues/detail?id=499 Could you please guys try to set a higher timeout in the FastCGI and SCGI rules? Odds are that will solve the issue, although I'm not certain. Some testing is required.. |
|||||||||||||
,
Nov 24, 2009
I just discovered that my reverse proxy (cherokee 0.99.29) usually sends a 5xx gateway timeout or bad gateway when trying to POST a big text (when editing a wiki, for example). |
|||||||||||||
,
Nov 25, 2009
Could this possibly be related to the maximum header length? |
|||||||||||||
,
Nov 25, 2009
Something funny must be happening in the post manager. I'm checking it out right away..
Status: Started
|
|||||||||||||
,
Nov 25, 2009
@jjamor: A few days ago, I committed a few patches for adding custom timeout support to rule entries. That means, a specific rule can define a timeout value, as it can define a encoding method, o a handler plug-in. The target was to allow PHP connections to last longer and the general timeout limit, so the server wouldn't reply a 504 error if the php-cgi was still (hanged) processing the request when the timeout limit was reached. I've just realized that, you might be hitting that issue in the back-end servers. Depending on the wiki software processing speed, the back-end web server might be returning 504 errors whenever that process takes longer than the general timeout limit. In that case, the front-end proxy server would be just relying it to the outside world. I have not probed that theory yet, but it makes good sense to me. The best way to test it would be to add a custom timeout to the Extension PHP rule of the back-end servers. A 35 seconds value would be enough (since php.ini sets 30 seconds of processing limit by default). Hopefully that will fix the issue. Cherokee 0.99.30 will ship an improved version of the PHP wizard. It checks the php.ini file in order to figure the right timeout value. In that way it can set the right timeout value to the rule, so it does not return 504 errors when php scripts takes a "long" time to be executed (longer than the server-wide timeout limit). BTW, I'm talking about PHP, but the underlying idea applies to any other scripting language as well. |
|||||||||||||
,
Nov 25, 2009
@alobbs, does this apply to my backend, which is using Moin (python w/fcgi, not php)? in any case, I have tested to directly submitting the big POSTs to the backend, and the errors do not occur. |
|||||||||||||
,
Nov 25, 2009
@jjamor, I think you must apply to your backends and to the reverse proxy rule on the proxy server. |
|||||||||||||
,
Nov 25, 2009
@alobbs, @skarcha: finally the backend was moved to apache, because it was impossible to get work it with the spawn-fcgi binary (see issue 591). However, the 5xx errors are still here but only when using the reverse proxy (with Cherokee). I'm going to try compilation of the SVN version and test it ... |
|||||||||||||
,
Nov 25, 2009
Ok, compiled and setup a timeout of 60 seconds. The problem still exists: sometimes the proxy server respond with a Bad gateway error. The response is immediate. By analyzing the logs, the backend server always responds correctly. I'm going to switch the reverse proxy to apache until the problem is really fixed. |
|||||||||||||
,
Feb 11, 2010
@jjamor: Do you still have this problem? |
|||||||||||||
,
Feb 12, 2010
I am running cherokee 0.99.41 since 2010-01-26 and I did not need to restart it since then. However sometimes previous versions worked normally a month or so and then started to serve 5xx errors until it is restarted. I'd wait some weeks more before closing this issue. |
|||||||||||||
,
Feb 12, 2010
Great! Thanks for the follow up ;) |
|||||||||||||
,
Jun 07, 2010
It seems this was finally fixed in February. Can anybody confirm it so we can close the bug? Thanks guys. |
|||||||||||||
,
Jul 12, 2010
I think I'm suffering this issue now. With a fresh Drupal install, going through the installation steps ends up in a 504 when adding the 'Administrator account' (last step). After that, the Drupal install is available though. But when I try to register a new account, I end up with a 504 after the submit. The error log doesn't show anything. I will provide a trace in a few minutes. Software and versions: Cherokee 1.0.5 (from source) Drupal 6.17 (used the Cherokee-admin wizard to configure it) Ubuntu Linux (10.04, aptitude updated just today) php-cgi -v: PHP 5.3.2-1ubuntu4.2 with Suhosin-Patch (cgi-fcgi) |
|||||||||||||
,
Jul 12, 2010
Here is the trace. (used CHEROKEE_TRACE="all") |
|||||||||||||
,
Jul 12, 2010
@hendriks.luuk: so, it took ~30 seconds until the error message showed up, didn't it? |
|||||||||||||
,
Jul 12, 2010
Indeed, I checked it with another user registration attempt, it took 30 seconds before the 504 showed up. |
|||||||||||||
,
Jul 22, 2010
I tried the same thing on CentOS with Cherokee 1.0.0 (from the EPEL repo) and PHP 5.2 (from Utter-ramblings). No 504 with that configuration! |
|||||||||||||
,
Aug 25, 2010
I'm having the same issue when performing a long operation (downloading and adding user avatars to databasse). The task itself (PHP) finishes, but the webserver gives up after almost exactly 30 seconds (with the infamous 504 Gateway timeout error) |
|||||||||||||
,
Aug 25, 2010
Oh, using Cherokee 1.08 on OS X 10.6. |
|||||||||||||
,
Aug 25, 2010
So what is your php execution timeout? |
|||||||||||||
| ► Sign in to add a comment | |||||||||||||