Export to GitHub

phusion-passenger - issue #654

/tmp cleaners delete /tmp/passenger-standalone.* subdirectories


Posted on Apr 4, 2011 by Happy Rhino
  • Setup * Centos 5.5 RVM 1.2.6 Apache 2.2.3 Passenger Standalone 3.0.6 (running on port 2011)
    • passenger start -p 2011 -d -e production Ruby 1.9.2 Rails 3.0.4

httpd.conf: <VirtualHost *:80> ProxyRequests Off PassengerEnabled off

ServerName www.mysite.com
ServerAlias *.mysite.com

ProxyPass / http://127.0.0.1:2011/
ProxyPassReverse / http://127.0.0.1:2011/

</VirtualHost>

  • I'm getting this error why trying to upload an image to a Rails application: * 2011/04/04 00:35:26 [crit] 8634#0: *38940 open() "/tmp/passenger-standalone.8583/client_body_temp/0000000022" failed (2: No such file or directory), client: 127.0.0.1, server: _, request NOTE: This happens randomly (or at least seemingly) after having Passenger running for a long period of time.

*Not sure if this is completely unrelated, but my Apache logs contained this shortly before things went down: * [Sun Apr 03 22:18:09 2011] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:5500 (127.0.0.1) failed [Sun Apr 03 22:18:09 2011] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1) [Sun Apr 03 22:18:10 2011] [error] proxy: HTTP: disabled connection for (127.0.0.1) [Sun Apr 03 22:29:45 2011] [error] proxy: client 68.101.121.244 given Content-Length did not match number of body bytes read [Sun Apr 03 22:29:45 2011] [error] proxy: pass request body failed to 127.0.0.1:2011 (127.0.0.1) from 68.101.121.244 () [Mon Apr 04 00:19:47 2011] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:5500 (127.0.0.1) failed [Mon Apr 04 00:19:47 2011] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1)

NOTE: You'll notice Apache trying to do a proxy pass to port 5500... that system was taken offline and we forgot to remove the vhost. Could this have caused the problem?

Comment #1

Posted on Apr 11, 2011 by Grumpy Camel

Oh no not that again..

You have a /tmp cleaner daemon on your system that deletes /tmp/passenger-standalone.* subdirectories. I'll look at implementing workaround facilities in Phusion Passenger Standalone to prevent /tmp cleaners from doing that, but in the mean time you should either disable your /tmp cleaner daemon or configure it to leave /tmp/passenger-standalone.* alone.

Comment #2

Posted on Jun 30, 2012 by Happy Bird

Ubuntu 10.04 RVM 1.13.4 Ngnix 1.2.1 Passenger Standalone 3.0.13

I'm still having these issues where the /tmp/passenger-standalone.* directory gets wiped. I think I've upped the periods where the tmp directory gets wiped but this isn't a good solution.

Comment #3

Posted on Jul 1, 2012 by Happy Bird

Symptoms: 500 errors when uploading large files after the server has been running for a while (hour to a day)

Reason: /tmp/passenger-standalone.* directory is wiped by a cronjob then passenger cannot write or read from the directory.

Solution: -For Ubuntu 10.04, unsure about other distros. 1) sudo nano /etc/init/mounted-tmp.conf

2) Find this area and add: EXCEPT='! -name . ! ( -path ./lost+found -uid 0 ) ! ( -path ./quota.user -uid 0 )
! ( -path ./aquota.user -uid 0 ) ! ( -path ./quota.group -uid 0 ) ! ( -path ./aquota.group -uid 0 ) ! ( -path ./.journal -uid 0 ) ! ( -path ./.clean -uid 0 ) ! ( -path ./passenger-standalone* -uid 0 ) # This will prevent the script from deleting the directory. ! ( -path "./...security*" -uid 0 )'

3) Restart passenger for your web app

#

I do have another question though. Does passenger clean up that tmp directory? Because our servers deal with image uploads, it can run into problems if it doesn't.

Comment #4

Posted on Jul 2, 2012 by Happy Bird

Looks like that didn't work, I just ran nginx and when in development mode reset standalone every now and then. Inconvenient but its okay.

Comment #5

Posted on Jul 10, 2012 by Grumpy Camel

Yes Phusion Passenger cleans it up automatically.

Comment #6

Posted on Mar 30, 2013 by Happy Monkey

hongli...@gmail.com,

Can you please explain your comment Apr 11th 2012 in more detail? I'm running into this issue in my Rails 3 app deployed on Amazon's Elastic Beanstalk service (64bit Amazon Linux running Ruby 1.9.3).

Where is the /tmp cleaner daemon located? How do I disable it? How can I configure it to ignore the passenger-standalone.* subdirectories?

Thanks in advance!

Comment #7

Posted on Mar 31, 2013 by Grumpy Camel

That depends on your operating system. What distro is "Amazon Linux" based on? If it's Ubuntu then your tmp cleaner is tmpreaper. See http://manpages.ubuntu.com/manpages/hardy/man8/tmpreaper.8.html. It is configured in /etc/tmpreaper.conf.

Comment #8

Posted on Apr 1, 2013 by Happy Monkey

Thanks for the reply. I believe the /tmp cleaner script was /etc/cron.daily/tmpwatch in my case. I updated that script to ignore the /tmp/passenger* pattern.

! /bin/sh

flags=-umc /usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \ -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \ -X '/tmp/hsperfdata_' -X '/tmp/passenger' 10d /tmp /usr/sbin/tmpwatch "$flags" 30d /var/tmp for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do if [ -d "$d" ]; then /usr/sbin/tmpwatch "$flags" -f 30d "$d" fi done

Comment #9

Posted on May 25, 2013 by Grumpy Camel

Fixed in commit ff2c7b4.

Comment #10

Posted on Oct 19, 2013 by Happy Giraffe

I am getting the same "/tmp/passenger-standalone.8583/client_body_temp/0000000022" failed (2: No such file or directory), client: 127.0.0.1, server: _, request error, but NOT consistently. The rails3.2.13 application has classes that own users's uploaded images or files. The first four user-owned objects are allowing users to load images TO THIS MOMENT, but newer instances are failing entirely. I have not experienced the same error with other apps that have user-uploaded images, so I doubt there is clean-up effect - the behaviour of user-owned objects is puzzling. Thus, something changed at some point while creating new records that own these images (I have a suspicion it deals with enabling the rails pipeline, but have little recollection of such) and their relation to nginx via passenger.

Comment #11

Posted on Oct 19, 2013 by Grumpy Camel

binario228: can you set the TMPDIR environment variable to some place that is not /tmp, and then start Passenger Standalone? Does that help?

Comment #12

Posted on Oct 19, 2013 by Happy Giraffe

I've never set that variable. Before I do something stupid, what is a proper way to do this?

Comment #13

Posted on Oct 21, 2013 by Grumpy Camel

I'm asking you to try setting it, and to check whether that helps. That variable will tell Passenger Standalone to use a different temp directory.

Like this:

mkdir $HOME/tmp
export TMPDIR=$HOME/tmp
cd /path-to-your-app
passenger start

Comment #14

Posted on Oct 22, 2013 by Happy Giraffe

Having a number of applications, I was cautious about changing the tmp directory for the whole lot. But your comment was useful. a simple stop and start (via cap deploy) restored all the expected functions. Now to figure out what I changed to bring this about... Thank you.

Status: Fixed

Labels:
Type-Defect Priority-Medium Milestone-4.0.4