Skip to content
This repository has been archived by the owner on Apr 16, 2021. It is now read-only.

Need a better solution for purging at 90% disk usage #143

Closed
GoogleCodeExporter opened this issue Mar 24, 2015 · 12 comments
Closed

Need a better solution for purging at 90% disk usage #143

GoogleCodeExporter opened this issue Mar 24, 2015 · 12 comments

Comments

@GoogleCodeExporter
Copy link

edit /etc/cron.d/sensor-newday and change this:
0 0     * * * root /usr/local/sbin/nsm --sensor --restart --only-daemonlogger 
>> /var/log/nsm/sensor-newday-daemonlogger.log

to this:
0 *     * * * root /usr/local/sbin/nsm --sensor --restart --only-daemonlogger 
>> /var/log/nsm/sensor-newday-daemonlogger.log

Then restart cron just to make sure it's using the new schedule:
sudo service cron restart

Original issue reported on code.google.com by doug.bu...@gmail.com on 2 Nov 2011 at 9:15

@GoogleCodeExporter
Copy link
Author

A few thoughts:

- I'd like to de-couple the daemonlogger restart from the checkdisk function.  
They don't necessarily need to be run at the same time and de-coupling them 
would give us some extra flexibility.

- The checkdisk function would be exposed like this:
/usr/local/sbin/nsm --sensor --checkdisk

- The checkdisk function would be scheduled to run hourly via 
/etc/cron.d/checkdisk:
0 *     * * * root /usr/local/sbin/nsm --sensor --checkdisk >> 
/var/log/nsm/checkdisk.log

- If you have multiple sensors, the current logic deletes all the pcaps from 
the first sensor before starting to purge the next.  It should delete from all 
sensors evenly.

-The current logic only deletes pcaps from the dailylogs directory.  It should 
also delete old argus and httpry logs.

Original comment by doug.bu...@gmail.com on 30 Nov 2011 at 9:44

  • Changed title: Need a better solution for purging at 90% disk usage
  • Changed state: Accepted
  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Original comment by doug.bu...@gmail.com on 1 Dec 2011 at 3:09

  • Added labels: Priority-High
  • Removed labels: Priority-Low

@GoogleCodeExporter
Copy link
Author

New sensor_cleandisk() function in /usr/local/lib/nsmnow/lib-nsm-sensor-utils:

sensor_cleandisk() 
{
        # grab input variables with sane defaulting
        LOG_DIR=${1:-}
        UTC=${2:N}
        WARN_DISK_USAGE=${3:-80}
        CRIT_DISK_USAGE=${4:-90}
        #echo $LOG_DIR
        cd $LOG_DIR/..
        SENSOR_DIR=`pwd`
        cd - >/dev/null
        #echo $SENSOR_DIR
        CUR_USAGE=$(df -P $SENSOR_DIR | grep -v -i filesystem | awk '{print $5}' | tr -d %)
        #echo $CUR_USAGE

        # let's change color based on severity
        if [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]; then
                USE_COL=${RED}
        elif [ "$CUR_USAGE" -gt "$WARN_DISK_USAGE" ]; then
                USE_COL=${YELLOW}
        else
                USE_COL=${GREEN}
        fi

        echo_msg 1 "disk space currently at ${USE_COL}${CUR_USAGE}%"

        if [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]
        then
                # grab time in GMT
                if [ "$UTC" == "Y" ]
                then
                        TODAY=$(date -u "+%Y-%m-%d")
                else
                        TODAY=$(date "+%Y-%m-%d")
                fi

                # Delete oldest dailylogs directory from each of the sensors in /nsm/sensor_data/
                for SENSOR in "$SENSOR_DIR"/*
                do
                        REMOVED="no"
                        # find the oldest dailylogs directory and exclude today
                        OLDEST_DIR=$(ls $SENSOR/dailylogs | sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_DIR" -o "$OLDEST_DIR" == ".." -o "$OLDEST_DIR" == "." ]
                        then
                                echo_msg 1 "${RED}no old pcaps available to clean up in $SENSOR/dailylogs/"
                        else
                                echo_msg 1 "removing directory: $SENSOR/dailylogs/$OLDEST_DIR"
                                rm -rf "$SENSOR"/dailylogs/"$OLDEST_DIR"
                                REMOVED="yes"
                        fi
                        # find the oldest argus files and exclude today
                        OLDEST_ARGUS=$(ls $SENSOR/argus | sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_ARGUS" -o "$OLDEST_ARGUS" == ".." -o "$OLDEST_ARGUS" == "." ]
                        then
                                echo_msg 1 "${RED}no old argus files available to clean up in $SENSOR/argus/"

                        else
                                echo_msg 1 "removing argus file: $SENSOR/argus/$OLDEST_ARGUS"
                                rm -f "$SENSOR"/argus/"$OLDEST_ARGUS"
                                REMOVED="yes"
                        fi
                        # find the oldest httpry files and exclude today
                        OLDEST_HTTPRY=$(ls $SENSOR/httpry | sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_HTTPRY" -o "$OLDEST_HTTPRY" == ".." -o "$OLDEST_HTTPRY" == "." ]
                        then
                                echo_msg 1 "${RED}no old httpry files available to clean up in $SENSOR/httpry/"
                        else
                                echo_msg 1 "removing httpry file: $SENSOR/httpry/$OLDEST_HTTPRY"
                                rm -f "$SENSOR"/httpry/"$OLDEST_HTTPRY"
                                REMOVED="yes"
                        fi
                        # find the oldest unified2 files and exclude today
                        OLDEST_UNIFIED2=$(ls -al $SENSOR/snort.unified2* | awk '{print $6 " " $8}' |sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_UNIFIED2" -o "$OLDEST_UNIFIED2" == ".." -o "$OLDEST_UNIFIED2" == "." ]
                        then
                                echo_msg 1 "${RED}no old unified2 files available to clean up in $SENSOR/"
                        else
                                OLDEST_UNIFIED2_FILE=`echo $OLDEST_UNIFIED2 | awk '{print $2}'`
                                echo_msg 1 "removing unified2 file: $OLDEST_UNIFIED2_FILE"
                                rm -f "$OLDEST_UNIFIED2_FILE"
                                REMOVED="yes"
                        fi
                done
                if [ "$REMOVED" == "yes" ]
                then
                        # run cleandisk again as rm'ing one file might been enough
                        # but we wait 3 secs and hope any open writes are done.
                        sync
                        sleep 3
                        sensor_cleandisk $LOG_DIR $UTC $WARN_DISK_USAGE $CRIT_DISK_USAGE
                fi
        # simple warning (no action taken) of increased disk usage
        elif [ "$CUR_USAGE" -gt "$WARN_DISK_USAGE" ]
        then
                echo_warning_msg 1 "disk space is approaching critical levels"
        fi
}

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 11:51

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

We're going to run sensor_cleandisk in an hourly cron job so it doesn't need to 
run with sensor restart.  Commented out sensor_cleandisk in 
/usr/local/sbin/nsm_sensor_ps-restart.

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 11:52

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Added --clean option to /usr/local/sbin/nsm_sensor and pointed it to a new 
executable called nsm_sensor_clean:


        --clean)
            nsm_sensor_clean "$@"
            ;;

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 11:53

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Copied nsm_sensor_clear to nsm_sensor_clean and updated as follows:


# We clean all sensors at the same time anyway, so just grab the first sensor
SENSOR_NAME=`grep -v "^#" /etc/nsm/sensortab |awk '{print $1}' |head -1`

# check that the sensor DOES exists via it's config
if [ ! -f "/etc/nsm/${SENSOR_NAME}/sensor.conf" ]
then
    echo_error_msg 0 "OOPS: The server \"${SENSOR_NAME}\" does not exist!"
    exit 1
else
    # load existing variables for the sensor
    . "/etc/nsm/${SENSOR_NAME}/sensor.conf"
fi

# check that the sensor exists
if [ ! -d "/nsm/sensor_data/${SENSOR_NAME}" ]
then
    echo_error_msg 0 "OOPS: Collected data for sensor \"${SENSOR_NAME}\" does not exist!"
    exit 1
fi

if [ "$FORCE_YES" == "" ]
then
    # prompt to clean the sensor
    prompt_user_yesno "Clean Sensors" "Old data for sensors will be cleared.\n\nDo you want to continue?" "N"
    [ "$?" -ne 0 ] && exit 1
    if [ "$PROMPT_RET" != "Y" -a "$PROMPT_RET" != "y" ]
    then
        exit 1
    fi
fi

#
# CLEAN
#

echo_msg 0 "Cleaning sensors"

# clean the files as appropriate
sensor_cleandisk $SENSOR_LOG_DIR $SENSOR_UTC

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 11:55

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

I've seen some issues with the daily restarts not setting the date correctly, 
so I changed /etc/cron.d/sensor-newday to execute the restarts at 12:01 (with 
the exception of httpry-agent which runs at 12:02):

01 0    * * * root /usr/local/sbin/nsm --sensor --restart --only-daemonlogger >> 
/var/log/nsm/sensor-newday-daemonlogger.log
01 0    * * * root /usr/local/sbin/nsm --sensor --restart --only-argus >> 
/var/log/nsm/sensor-newday-argus.log
01 0    * * * root /usr/local/sbin/nsm --sensor --restart --only-httpry >> 
/var/log/nsm/sensor-newday-httpry.log
02 0    * * * root /usr/local/sbin/nsm --sensor --restart --only-httpry-agent >> 
/var/log/nsm/sensor-newday-httpry-agent.log

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 11:57

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Created /etc/cron.d/sensor-clean as follows:

# /etc/cron.d/sensor-clean
#
# crontab entry to remove old files every hour

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 * * * * root /usr/local/sbin/nsm_sensor_clean -y >> 
/var/log/nsm/nsm_sensor_clean.log

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 11:57

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Updated purging of unified2 files to be more consistent with the other file 
types:


                        # find the oldest unified2 files and exclude today
                        OLDEST_UNIFIED2=$(ls -l $SENSOR/snort.unified2* | awk '{print $6 " " $8}' |sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_UNIFIED2" -o "$OLDEST_UNIFIED2" == ".." -o "$OLDEST_UNIFIED2" == "." ]
                        then
                                echo_msg 1 "${RED}no old unified2 files available to clean up in $SENSOR/"
                        else
                                OLDEST_UNIFIED2_DATE=`echo $OLDEST_UNIFIED2 | awk '{print $1}'`
                                OLDEST_UNIFIED2_FILE=`echo $OLDEST_UNIFIED2 | awk '{print $2}'`
                                echo_msg 1 "removing unified2 files for $OLDEST_UNIFIED2_DATE"
                                ls -l $SENSOR/snort.unified2* | grep $OLDEST_UNIFIED2_DATE | awk '{print $8}' |while read FILE
                                do
                                        echo_msg 1 "removing unified2 file: $FILE"
                                        rm -f "$FILE"
                                done
                                REMOVED="yes"
                        fi

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 4:56

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Built package as follows:

/var/lib/gems/1.8/bin/fpm -s dir -t deb -n securityonion-nsmnow-admin-scripts 
-v 20111214 /etc/init.d/nsm* /usr/share/nsmnow/ /usr/local/sbin/nsm* 
/usr/local/lib/nsmnow/ /etc/cron.d/sensor-*

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 4:57

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Found issue when running as a cronjob.  --time-style="long-iso" must be used so 
that the script works properly when the locale is not set:

                        # find the oldest unified2 files and exclude today
                        # NOTE: --time-style="long-iso" must be used so that the script works properly in a cron job where the locale is not set
                        OLDEST_UNIFIED2=$(ls -l --time-style="long-iso" $SENSOR/snort.unified2* | awk '{print $6 " " $8}' |sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_UNIFIED2" -o "$OLDEST_UNIFIED2" == ".." -o "$OLDEST_UNIFIED2" == "." ]
                        then
                                echo_msg 1 "${RED}no old unified2 files available to clean up in $SENSOR/"
                        else
                                OLDEST_UNIFIED2_DATE=`echo $OLDEST_UNIFIED2 | awk '{print $1}'`
                                OLDEST_UNIFIED2_FILE=`echo $OLDEST_UNIFIED2 | awk '{print $2}'`
                                echo_msg 1 "removing unified2 files for $OLDEST_UNIFIED2_DATE"
                                ls -l --time-style="long-iso" $SENSOR/snort.unified2* | grep $OLDEST_UNIFIED2_DATE | awk '{print $8}' |while read FILE
                                do
                                        echo_msg 1 "removing unified2 file: $FILE"
                                        rm -f "$FILE"
                                done
                                REMOVED="yes"
                        fi

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 6:36

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author

Published:
http://securityonion.blogspot.com/2011/12/security-onion-20111214-now-available.
html

Original comment by doug.bu...@gmail.com on 14 Dec 2011 at 7:46

  • Changed state: Verified
  • Added labels: ****
  • Removed labels: ****

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant