Pfsense_Lusca

Install Lusca on pfsense.

Updated Aug 28, 2011 by chudy.fernandez

Introduction

Compiled with:
'--enable-removal-policies=lru heap'
'--enable-storeio=aufs,coss,null' - null if you don't want to use your hard drive
'--enable-delay-pools'
'--enable-arp-acl' - mac address acl
'--enable-pf-transparent'
'--enable-snmp' - for MRTG

Lusca-cache patched r11. Patches applied:

ignore-must-revalidate
vary aware using storeurl
save vary object/index other than COSS (vary index will be lost on rebuild if in coss)

Instruction

Pfsense 1.2.3 or 2.0

From your "PfSense's GUI" run this at Diagnostics > Command.
^{other than lusca packages might be old. Use package.sh to use original repository.}

fetch http://pfsense-cacheboy.googlecode.com/svn/trunk/script/package.sh && chmod +x package.sh && ./package.sh

Now you may install lusca-cache at System > Packages.
To restore or use original repository.
^{you can no longer see lusca updates}

./package.sh off

Pfsense 2.0

Go to the http://pfsense/pkg_mgr_settings.php
then use chudy.0fees.net as your Package Repository

Notes
Configurations at:
/usr/local/etc/squid/dir.conf - customized cache_dir if you might need it most case if you have more than one coss, aufs
/usr/local/etc/squid/include.conf - other squid configurations
/usr/local/etc/squid/refresh.conf - refresh patterns

Troubleshoot:
chudy.0fees.net is free domain if error just retry

Pfsense GUI's doesn't save your configurations
Causes:

Previous squid packages and scripts doesn't clean up properly(pfsense limitation).
config.xml is already dirty as well as your previous installation had already been compromised.

Fixed:
Run this at Diagnostics: Execute command > PHP execute

$num_fields = array('squidcache','squid','squidauth','squidnac','squidtraffic','squidupstream','squidusers');
foreach ($num_fields as $field)
unset($config['installedpackages']["$field"]);
write_config();

for selective approach where squidcache of URL http://pfsense/pkg_edit.php?xml=squid_cache.xml&id=0 as an example that usually doesn't save.

unset($config['installedpackages'][squidcache]);
write_config();

for donation you may send it to
paypal: chudy_fernandez at yahoo dot com

Comment by cbuechler, Apr 2, 2010

WARNING: The instructions here will change your pfSense packages to no longer pull from the official package repository, which means you will no longer get any of the official packages (or get an out of date version served by this unofficial package server).

Comment by serangku, Jun 27, 2010

i always use lusca from ur source for my pfsense network. work like a solid rock. thanks for provide it for community. keep good work.

Comment by mha...@gmail.com, Jun 28, 2010

Do you know how to downgrade? r14716 dying every 15-20 min.

I wanna downgrade to r14635.

Comment by project member chudy.fernandez, Jun 29, 2010

pkg_add -rf http://pfsense-cacheboy.googlecode.com/files/lusca-cache-r14635.tbz could you show me your cache.log

Comment by ltet...@gmail.com, Jul 12, 2010

sir, I also having same problem with r14716 dying but at much longer rate (10 - 20 hours). I need more info on downgrade back to r14635. Do i need to remove the r14716 packages first. Or just put "pkg_add -rf http://pfsense-cacheboy.googlecode.com/files/lusca-cache-r14635.tbz" at shell without removing the r17416 packages. Thanks you sir.

Here's my /var/squid/log/cache.log http://pastebin.com/dKvC4ezu

Comment by fauziwrd@gmail.com, Jul 24, 2010

maaf newbie pfsense saya dah instal dan proxy dah jalan tpi untuk yang cahce lognya gak ada pergerakan ketika saya perintah tail-f /var/squid/log cahce.log disana ada tulisan store_swap_size=0k harus saya apain ini biar bisa storeurl.plnya jalan mengcache. thk..

Comment by project member chudy.fernandez, Jul 24, 2010

ltetlie: gui pkg_reinstall will do everything.
fauziwrd: I dont understand.

Comment by MR.Abduh...@gmail.com, Jul 31, 2010

sir. how to use bandwitdh limiter squid proxy?

Comment by cbexpr...@axion.ca, Aug 6, 2010

Running V r14733 on PFsense (don't see the last few updates in package manager? Also one site that is similar to craigslist will not open more than the home directory unless I bypass it's IP in the squid config. (IP 4.59.176.136) With out by passing it all sub directories bring up a blank white page.

Comment by serangku, Aug 6, 2010

r14733 stable enough than r14635 ? is it possible to implement with pf 2.0 directly ? thanks

Comment by MR.Abduh...@gmail.com, Sep 2, 2010

running on pfsense 2.0 i386/amd64 ??

Comment by mic...@samovojski.com, Sep 10, 2010

with 2.0-BETA4 (i386) built on Fri Sep 10 05:09:52 EDT 2010

/root(2): squid -v /libexec/ld-elf.so.1: Shared object "libcrypt.so.4" not found, required by "squid"

Comment by mic...@samovojski.com, Sep 10, 2010

cd /lib

ln -s libcrypt.so.5 libcrypt.so.4

ln -s libmd.so.5 libmd.so.4

ln -s libutil.so.8 libutil.so.7

add squid_enable="YES" in /etc/defaults/rc.conf

Comment by mic...@samovojski.com, Sep 10, 2010

delay pool not seem working using pfsense gui to config

Comment by roger2...@gmail.com, Sep 28, 2010

hello follow the steps

# From your "PfSense?'s GUI" run this at Diagnostics > Command. other than lusca packages might be old. Use package.sh to use original repository.

fetch http://pfsense-cacheboy.googlecode.com/svn/trunk/script/package.sh && chmod +x package.sh && ./package.sh

# Now you may install lusca-cache at System > Packages

but when I went to install lusca I say that has not been able to connect to chudy.0fees.net

you can help thanks

Comment by project member chudy.fernandez, Oct 8, 2010

pfsense20: captive portal + transparent proxy fix http://forum.pfsense.org/index.php/topic,28211.0.html

Comment by yellowha...@gmail.com, Oct 8, 2010

I have problem with PURGE function. I'm trying to PURGE a site cache but when I use: e.g.: squid -m PURGE http://www.facebook.com it's result: "FATAL: Need to add -DMALLOC_DBG when compiling to use -mX option Squid Cache (Version LUSCA r14756 patched: chudy r12): Terminated abnormally. CPU Usage: 0.002 seconds = 0.000 user + 0.002 sys Maximum Resident Size: 0 KB Page faults with physical i/o: 0 Abort (core dumped)"

So how to use PURGE function in lusca. thanks.

Comment by michel.s...@gmail.com, Nov 9, 2010

you can purge with squidclient -m PURGE URL

Comment by michel.s...@gmail.com, Nov 10, 2010

can you compile the next update with --with-aufs-threads=5 im always getting squidaio_queue_request: WARNING - Queue congestion

thx

Comment by voice.pe...@gmail.com, Nov 27, 2010

Pls make support for pfsense AMD I`m try here the logs

Nov 28 11:37:08 squid5771?: Squid Parent: child process 5774 started Nov 28 11:37:09 squid5774?: The store_rewriter helpers are crashing too rapidly, need help! Nov 28 11:37:09 squid5771?: Squid Parent: child process 5774 exited due to signal 6 Nov 28 11:37:09 kernel: pid 5774 (squid), uid 62: exited on signal 6 Nov 28 11:37:12 squid5771?: Squid Parent: child process 5828 started Nov 28 11:37:12 squid5828?: The store_rewriter helpers are crashing too rapidly, need help! Nov 28 11:37:12 squid5771?: Squid Parent: child process 5828 exited due to signal 6 Nov 28 11:37:12 kernel: pid 5828 (squid), uid 62: exited on signal 6 Nov 28 11:37:15 squid5771?: Squid Parent: child process 5853 started Nov 28 11:37:15 squid5853?: The store_rewriter helpers are crashing too rapidly, need help! Nov 28 11:37:15 squid5771?: Squid Parent: child process 5853 exited due to signal 6 Nov 28 11:37:15 kernel: pid 5853 (squid), uid 62: exited on signal 6 Nov 28 11:37:18 squid5771?: Squid Parent: child process 5947 started Nov 28 11:37:19 squid5947?: The store_rewriter helpers are crashing too rapidly, need help!

Comment by carlosca...@gmail.com, Nov 28, 2010

ichel.samovojski add n_aiops_threads=128 to custom and restart.
oice.pekanbaru try run /usr/local/etc/squid/storeurl.pl and show whats the error.

Comment by portine...@gmail.com, Dec 16, 2010

I have a problem "not been able to connect to chudy.0fees.net " and nothing of lusca install, anybody have a solution to this?

Comment by serangku, Dec 27, 2010

i use latest pf20 64 bit, when use lusca from here its error on install package. i would like to use lusca on pf20 64 bit, so it can handle more than 3gb ram for proxy. is lusca package support on pf 20 64 bit ? thanks sir.

Comment by k.tham...@gmail.com, Jan 17, 2011

yes you can use lusca on pfsense-2.0 BETA5 amd64.

Comment by k.tham...@gmail.com, Jan 20, 2011

Dear Chudy,

I am using the following on icore3 intel board:

$squid -v Squid Cache: Version LUSCA r14829 patched: chudy r14 configure options: '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--enable-auth=basic ntlm digest' '--enable-basic-auth-helpers=DB NCSA PAM MSNT SMB LDAP SASL YP' '--enable-digest-auth-helpers=password ldap' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group ldap_group' '--enable-ntlm-auth-helpers=SMB' '--with-pthreads' '--enable-storeio=aufs null coss' '--enable-delay-pools' '--enable-snmp' '--disable-wccp' '--disable-ident-lookups' '--enable-arp-acl' '--enable-pf-transparent' '--with-large-files' '--enable-large-cache-files' '--enable-err-languages=Armenian Azerbaijani Bulgarian Catalan Czech Danish Dutch English Estonian Finnish French German Greek Hebrew Hungarian Italian Japanese Korean Lithuanian Polish Portuguese Romanian Russian-1251 Russian-koi8-r Serbian Simplify_Chinese? Slovak Spanish Swedish Traditional_Chinese? Turkish' '--enable-default-err-language=English' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=i386-portbld-freebsd8.0' 'build_alias=i386-portbld-freebsd8.0' 'CC=cc' 'CFLAGS=-O2 -pipe -I/usr/local/include -I/usr/local/include -DLDAP_DEPRECATED -fno-strict-aliasing' 'LDFLAGS= -L/usr/local/lib -L/usr/local/lib' 'CPPFLAGS=-I/usr/local/include'

$ uname -a FreeBSD megacachebsd.hyphen-telecom 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #0: Wed Jan 12 22:17:23 EST 2011 root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64

$ squidclient -p 3302 mgr:info HTTP/1.0 200 OK Server: Lusca Date: Tue, 18 Jan 2011 00:11:11 GMT Content-Type: text/plain Expires: Tue, 18 Jan 2011 00:11:11 GMT X-Cache: MISS from BSD.Cache Connection: close

I get the following after executing squidclient output:

Squid Object Cache: Version LUSCA r14829 patched: chudy r14 Start Time: Sun, 16 Jan 2011 17:22:51 GMT Current Time: Tue, 18 Jan 2011 00:11:11 GMT Connection information for squid:

Number of clients accessing cache: 2 Number of HTTP requests received: 34408670 Number of ICP messages received: 5748 Number of ICP messages sent: 5748 Number of queued ICP replies: 0 Request failure ratio: f Average HTTP requests per minute since start: f Average ICP messages per minute since start: f Select loop called: 155668793 times, f ms avg

Cache information for squid:

Request Hit Ratios: 5min: f%, 60min: f% Byte Hit Ratios: 5min: f%, 60min: f% Request Memory Hit Ratios: 5min: f%, 60min: f% Request Disk Hit Ratios: 5min: f%, 60min: f% Storage Swap size: 258278220 KB Storage Mem size: 13368 KB Mean Object Size: f KB Requests given to unlinkd: 0

Median Service Times (seconds) 5 min 60 min:

HTTP Requests (All): f f Cache Misses: f f Cache Hits: f f Near Hits: f f Not-Modified Replies: f f DNS Lookups: f f ICP Queries: f f

Resource usage for squid:

UP Time: f seconds CPU Time: f seconds CPU Usage: f% CPU Usage, 5 minute avg: f% CPU Usage, 60 minute avg: f% Process Data Segment Size via sbrk(): 0 KB Maximum Resident Size: 1504692 KB Page faults with physical i/o: 0

Memory accounted for:

Total accounted: 1008517 KB memPoolAlloc calls: 50461961 memPoolFree calls: 19029370

File descriptor usage for squid:

Maximum number of file descriptors: 65536 Largest file desc currently in use: 20471 Number of file desc currently in use: 6609 Files queued for open: 0 Available number of file descriptors: 58927 Reserved number of file descriptors: 100 Store Disk files open: 212 IO loop method: kqueue

Internal Data Structures:

375620 StoreEntries?

33 StoreEntries? with MemObjects?

36 Hot Object Cache Items

374472 on-disk objects

Regards,

Comment by mha...@gmail.com, Feb 15, 2011

Does chuddy.0fees.net down?

Comment by project member chudy.fernandez, Feb 21, 2011

free but host is unstable with their dns. just retry.

been busy. I have'nt compiled yet lusca for SMP pfsense(even though squid's core is using single threaded ). "f%" in cachemgr as a result.

Comment by serangku, Feb 24, 2011

try with lan & vlan iface on pf 123 & 2, there's no running log when tail -f /var/squid/log/access.log thats will log with original squid package thanks for provide alternatif package

Comment by k.tham...@gmail.com, Feb 27, 2011

Thanks Chudy, will be waiting , great performance on pf 2.0

Comment by alfurqon...@gmail.com, Feb 28, 2011

after installing lusca in pfsense 2.0 rc1

i got a message :

Warning: Invalid argument supplied for foreach() in /usr/local/www/pkg_edit.php on line 415

how to resolve it..

Comment by alfurqon...@gmail.com, Mar 1, 2011

i have an error after lusca installation completed in my pfsense 2.0 rc1

Parse error: syntax error, unexpected $end in /etc/inc/pkg-utils.inc on line 1183

Comment by ens...@gmail.com, Mar 2, 2011

i have the same problem

Comment by k.tham...@gmail.com, Mar 3, 2011

downgrade to pfsense2.0-BETA5 at the moment

Comment by michel.s...@gmail.com, Mar 6, 2011

working fine for me under 2.0 RC1 with the lastest version uploaded

Comment by ens...@gmail.com, Mar 7, 2011

so its updated, since when?

Comment by gallium5...@gmail.com, Mar 8, 2011

Since March 5, 2011. Thanks chudy for the update.

Comment by k.tham...@gmail.com, Mar 10, 2011

Thanks chudy !

Comment by roger2...@gmail.com, Mar 14, 2011

Gracias Chudy.

Comment by hisss...@gmail.com, Mar 18, 2011

Thank you Chudy

Comment by cruxnyi...@gmail.com, Apr 8, 2011

How do i fix ?? pls help me...

code?# pkg_add lusca-cache-r14850.tbz pkg_add: package 'lusca-cache-r14850' or its older version already installed[/code]

Comment by bas...@gmail.com, Apr 14, 2011

Hello Chudy, can you please update perl to the latest 8.1 package for pfsense 2.0? the version in your repository is causing problems with other packages such as ipblocklist, thanks!

Comment by michel.s...@gmail.com, May 3, 2011

Hello Chudy, can you please update perl to the latest 8.1 package for pfsense 2.0? the version in your repository is causing problems with other packages such as ipblocklist, thanks!

same for me =D

Comment by ltet...@gmail.com, May 22, 2011

chudy.0fees.net still offline after I retry for a week.

Comment by dayvidga...@gmail.com, May 24, 2011

won't install on 2.0 T_T

Comment by dayvidga...@gmail.com, May 31, 2011

installed on 2.0 thanks.

Comment by ltet...@gmail.com, Jun 9, 2011

How you install it dayvidga cause I found that the chudy.0fees.net was down and unable to get packages info if enable. Is there a way to install the packages manually. Please help,

Comment by dayvidga...@gmail.com, Jul 25, 2011

why my disk usage stays at 88% its been a month since it reach that percentage

Comment by portine...@gmail.com, Aug 17, 2011

please one tutorial for manual install

Comment by bas...@gmail.com, Aug 25, 2011

I'm getting this on the new RC3 builds..:

"2011/08/25 21:26:57| WARNING: store_rewriter #1 (FD 13) exited 2011/08/25 21:26:57| Too few store_rewriter processes are running FATAL: The store_rewriter helpers are crashing too rapidly, need help!"

Comment by bas...@gmail.com, Sep 11, 2011

Version 2.0-RC3 (i386) built on Fri Sep 9 13:20:27 EDT 2011

2011/09/12 06:18:18| WARNING: store_rewriter #1 (FD 25) exited 2011/09/12 06:18:18| Too few store_rewriter processes are running FATAL: The store_rewriter helpers are crashing too rapidly, need help!

Squid Cache (Version LUSCA r14850 patched: chudy r14): Terminated abnormally.

Comment by bas...@gmail.com, Sep 21, 2011

getting this on 2.0 RELEASE: http://pastebin.com/F0SMdL95

Comment by mor...@gmail.com, Oct 11, 2011

I got this error on Pfsense 2.0 RELEASE "http://sites.google.com/site/chudyfernandez/8/amd64/squid_radius_auth-1.10.tbz ... could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/squid_radius_auth-1.10.tbz. of squid_radius_auth-1.10 failed!"

Comment by nour...@hotmail.com, Nov 8, 2011

Dear Sir, i have the problem when i install the lusca_cache pakges and giveing me the reslt

Beginning package installation for lusca-cache... Downloading package configuration file... done. Saving updated package information... done. Downloading lusca-cache and its dependencies... Checking for package installation...

Downloading http://sites.google.com/site/chudyfernandez/8/amd64/squid_radius_auth-1.10.tbz ... could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/squid_radius_auth-1.10.tbz.

of squid_radius_auth-1.10 failed!

Installation aborted.Backing up libraries... Removing package... Starting package deletion for lusca-cache-r14850...done. Starting package deletion for perl-5.10...done. Starting package deletion for squid_radius_auth-1.10...done. Starting package deletion for libwww-5.4.0_4...done. Removing lusca-cache components... Tabs items... done. Menu items... done. Services... done. Loading package instructions... Include file squid.inc could not be found for inclusion. Deinstall commands... Not executing custom deinstall hook because an include is missing. Removing package instructions...done. Auxiliary files... done. Package XML... done. Configuration... done. Cleaning up... Failed to install package.

Installation halted.

Comment by daftarka...@gmail.com, Nov 19, 2011

LUSCA for PFSENSE is the best!!!! Puas...puas...PUASSSS!!!

Comment by christia...@gmail.com, Nov 26, 2011

Sir. I'm also having the same problem

Downloading http://sites.google.com/site/chudyfernandez/8/amd64/squid_radius_auth-1.10.tbz ... could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/squid_radius_auth-1.10.tbz.

of squid_radius_auth-1.10 failed!

Installation halted.

Im running an amd athlon x2 pc with pfsense 2.0-RELEASE (amd64)

Comment by ltet...@gmail.com, Nov 30, 2011

I am also having the same issue. I have tested it on pfSense 2.0 Release and 2.0 RC3. If I install the missing packages (squid_radius_auth-1.10.tbz and libwww-5.4.0_4.tbz) manually before lusca it may install successfully but it causes lusca unable to start because of missing file (from System Log). And it will cause pfSense CPU usage to rise and render an unusable pfSense.

By the way I do have other pfSense 2.0 with lusca cache installed and running but those were build much earlier.

Comment by hisss...@gmail.com, Nov 30, 2011

I think some donations will get this package updated... looks like he's busy and last update to this project added donation info, get a hint :D

Comment by cbexpre...@gmail.com, Feb 26, 2012

I have had to remove the package youtube will not work with it. Whats the word on an update!!??

Comment by kailste...@gmail.com, Mar 23, 2012

Got same error. please check the site your file is missing

Downloading http://sites.google.com/site/chudyfernandez/8/amd64/squid_radius_auth-1.10.tbz ... could not download from there or http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/squid_radius_auth-1.10.tbz. of squid_radius_auth-1.10 failed!

Comment by kailste...@gmail.com, Mar 23, 2012

i tried this before it work once but not anymore

pkg_add -v http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.2-release/All/squid_radius_auth-1.10.tbz pkg_add -v http://ftp2.FreeBSD.org/pub/FreeBSD/ports/amd64/packages-8.2-release/All/libwww-5.4.0_4.tbz

Comment by ilcamp...@hotmail.com, Apr 8, 2012