Appendix B: Crawling and Indexing

The gsa-crawler begins crawling from the URLs you specify on the Crawl and Index > Crawl URLs page. When a page is crawled, it is automatically indexed. Periodically, the index is checked against all of your serving prerequisites. If the index doesn't satisfy the prerequisites, the serving index rolls back to a previous snapshot, based on the rollback strategy that you chose. In other words, if the index conforms to the prerequisites, then it is served. If the index does not conform to the prerequisites, then the current index continues serving results.

File Size Limits

Some limits are imposed on crawling large HTML and non-HTML files.

For HTML files, the search appliance crawls up to a size of 2.5 MB, then discards the remainder of the file.

For non-HTML files, the search appliance crawls files up to 30 MB in size. Files larger than 30MB are discarded and not crawled. Non-HTML files are converted to HTML. Then the first 2MB of the HTML file are indexed. The remainder is discarded.

Crawling Frames and Framesets

To crawl framesets and their nested frames, the framesets must be well-formed with the <frame> tags occurring within the <frameset> tag. Anchors (links) must also occur within the frameset. Depending on the particular structure of your site, the search results may point to the frameset page itself or to the individual frame pages. There is currently no way to specify which behavior you prefer.

File Formats the Search Appliance Crawls and Indexes

Note: Encrypted, viewable PDF documents are converted to HTML for indexing; however, the cached HTML is not displayed. PDF files scanned with optical character recognition (OCR) software are not supported.

Word Processing Formats
Adobe FrameMaker		mif	Version 6.0
ASCII Text (7 & 8 bit)		txt	All versions
ANSI Text (7 & 8 bit)		ans	All versions
DEC WPS Plus		dx	Versions through 4.0
DEC WPS Plus		wpl	Versions through 4.1
DisplayWrite 2 & 3		txt	All versions
DisplayWrite 4 & 5		doc	Versions through Release 2.0
Enable		wpf	Versions 3.0, 4.0 and 4.5
First Choice		pfc	Versions through 3.0
Framework			Version 3.0
HTML		html, htm	Versions through 3.0 (some limitations)
IBM FFT		fft	All versions
IBM Revisable Form Text		rft	All versions
IBM Writing Assistant		iwa	Version 1.01
JustSystems Ichitaro		jaw, jbw	Versions 5.0, 6.0, 8.0-13.0, 2004
JustWrite		jw	Versions through 3.0
Legacy		leg	Versions through 1.1
Lotus AMI/AMI Professional		sam	Versions through 3.1
Lotus Manuscript		doc	Versions through 2.0
Lotus WordPro (Windows only)		lwp	Versions 96 through Millennium 9.6, text only
Lotus WordPro (Text only on UNIX)		lwp	SmartSuite 97 and Millennium
MacWrite II		mcw, mw, mwii	Version 1.1
MASS11		m11	Versions through 8.0
Microsoft Rich Text Format		rtf	All versions
Microsoft Word for DOS		doc	Versions through 6.0
Microsoft Word for Macintosh		doc	Versions 4.0 through 98
Microsoft Word for Windows		doc	Versions through 2003
Microsoft WordPad		rtf, doc	All versions
Microsoft Works for DOS		wks, wps	Versions through 2.0
Microsoft Works for Macintosh		wks, wps	Versions through 2.0
Microsoft Works for Windows		wks, wpf	Versions through 4.0
Microsoft Write		wri	Versions through 3.0
MultiMate		dox	Versions through 4.0
Navy DIF		dif	All versions
Nota Bene		nb	Version 3.0
Novell Perfect Works			Version 2.0
Novell WordPerfect for DOS			Versions through 6.1
Novell WordPerfect for Mac			Versions 1.02 through 3.0
Novell/Corel WordPerfect for Windows			Versions through 12.0
Office Writer		ow4	Version 4.0 to 6.0
PC-File Letter		ltr	Versions through 5.0
PC-File+ Letter		ltr	Versions through 3.0
PFS:Write		pfb	Versions A, B, and C
Professional Write for DOS		pw	Versions through 2.1
Professional Write Plus		pw, pwp	Version 1.0
Q&A for DOS		qa, qw, dtf	Version 2.0
Q&A Write for Windows		dtf	Version 3.0
Samna Word		sam, sm	Versions through Samna Word IV+
SmartWare II		smt	Version 1.02
Sprint		spr	Version 1.0
Text Mail (MIME)			No specific version
Total Word		tw	Version 1.2
Unicode Text		txt	All versions
Volkswriter 3 & 4		vw	Versions through 1.0
Wang PC		iwp	Versions through 2.6
WML			Version 5.2
WordMARC		wmc	Versions through Composer Plus
WordStar 2000 for DOS		ws1, ws2, ws3	Versions through 3.0
WordStar for DOS		ws	Versions through 7.0
WordStar for Windows		ws, wst, wsd	Version 1.0
XyWrite		xy3, xyp, xyw	Versions through III Plus
Spreadsheet Formats
Enable		300, wpf, ssf, dbf	Versions 3.0, 4.0 and 4.5
First Choice		ss, fol	Versions through 3.0
Framework		fw3	Version 3.0
Lotus 1-2-3 (DOS & Windows)		wku, wk1, wk2, wk3, wk4, wk5, wki, wks	Versions through 5.0
Lotus 1-2-3 for SmartSuite		wku, wk1, wk2, wk3, wk4, wk5, wki, wks	SmartSuite 97and Millennium
Lotus 1-2-3 Charts (DOS & Windows)		wku, wk1, wk2, wk3, wk4, wk5, wki, wks	Versions through 5.0
Lotus 1-2-3 (OS/2)		wku, wk1, wk2	Versions through 2.0
Lotus 1-2-3 Charts (OS/2)		wku, wk1, wk2	Versions through 2.0
Lotus Symphony		wr1	Versions 1.0,1.1 and 2.0
Microsoft Excel for Macintosh		xls	Versions 3.0 through 4.0, 98
Microsoft Excel for Windows		xls, xlw	Versions 2.2 through 2000
Microsoft Excel Charts		xlc	Versions 2.x through 7.0
Microsoft Multiplan		col, cod, mod	Version 4.0
Microsoft Works for Windows		wps, wks	Versions through 4.0
Microsoft Works (DOS)		wps, wks, wdb, wcm	Versions through 2.0
Microsoft Works (Macintosh)		wps, wks, wdb, wcm	Versions through 2.0
Mosaic Twin		wku	Version 2.5
Novell Perfect Works			Version 2.0
PFS:Professional Plan		tid	Version 1.0
QuattroPro for DOS		wkq, wq1	Versions through 5.0
QuattroPro for Windows		wb1, wb2, wk3	Versions through 9.0
SmartWare II		def, smt	Version 1.02
StarOffice Calc for Windows and UNIX		text only	Version 5.2
SuperCalc 5		cal	Version 4.0
VP Planner 3D		np	Version 1.0
Database Formats
DBASE		dbf	Versions through 5.0
DataEase		dba, dbm, dql	Version 4.x
dBXL		dbf	Version 1.3
Enable		300, wpf, ssf, dbf	Versions 3.0, 4.0 and 4.5
First Choice		pfc	Versions through 3.0
FoxBase		fmt, dbt, fox, inx, dbf	Version 2.1
Framework		fwk, fw, fw2, fw3	Version 3.0
Microsoft Works (DOS)		wdb, wks	Versions through 2.0
Microsoft Works (Macintosh)		wdb, wks	Versions through 2. 0
Microsoft Works for Windows		wdb, wks, dbf	Versions through 4.0
Paradox (DOS)		fsl, db, px	Versions through 4.0
Paradox (Windows)		fsl, db, px	Versions through 1.0
Personal R:BASE		rbf	Version 1.0
R:BASE 5000		rbf, dbf	Versions through 3.1
R:BASE System V		rbf	Version 1.0
Reflex		r2d	Version 2.0
Q & A		qa, qw, dtf	Versions through 2.0
SmartWare II		db, def, smt	Version 1.02
Graphics Formats
Adobe FrameMaker Graphics		fmv	Vector/raster through 5.0
Adobe Illustrator File Format		ai	Versions through 7.0, 9.0
Adobe Photoshop File Format		psd	Version 4.0
Adobe Portable Document Format		pdf	Adobe Reader version 1.4, which corresponds to Acrobat 5
Ami Draw Format		sdw	Ami Draw
AutoCAD Interchange and Native Drawing Format		dxf, dwg
AutoCAD Drawing Format		dwg	Versions 2.5-2.6, 9.0-14.0, 2000i and 2002
AutoShade Rendering Format		rnd	Version 2
Binary Group 3 Fax			All Versions
Bitmap Format		bmp, rle, ico, cur, OS/2 dib & warp	Windows
CALS Raster Format		gp4	Type I and Type II
Corel Clipart Format		cmx	Versions 5 through 6
Corel Draw		cdr	Versions 3.x through 8.x
Corel Draw		cdr with tiff header	Versions 2.x through 9.x
Computer Graphics Metafile		cgm	ANSI, CALS NIST versions 3.0
Encapsulated PostScript		eps	tiff header only
GEM Paint		img	No specific version
Graphics Environment Manager		gem	Bitmap and vector
Graphics Interface Format		gif	No specific version
Hewlett Packard Graphics Language		hpgl	Version 2
IBM Graphics Data Format		gdf	Version 1.0
IBM Picture Interchange Format		pif	Version 1.0
Initial Graphics Exchange Specification		iges	Version 5.1
JFIF (jpeg not in tiff format)		jfif	All Versions
JPEG (including EXIF)		jpeg	All versions
Kodak Flash Pix		fpx	All versions
Kodak Photo CD		pcd	Version 1.0
Lotus PIC		pic	All versions
Lotus Snapshot			All versions
Macintosh PICT1 and PICT2		pict	Bitmap only
MacPaint		pntg	No specific version
Micrografx Draw		drw	Versions through 4.0
Micrografx Designer		drw	Versions through 3.1
Micrografx Designer		dsf	Windows 95, Version 6.0
Novell PerfectWorks		draw	Version 2.0
OS/2 PM Metafile		met	Version 3.0
Paint Shop Pro 6		psp	Version 5.0 - 6.0
PC Paintbrush		pcx, dcx	All versions
Portable Bitmap		pbm	All versions
Portable Graymap		pgm	No specific version
Portable Network Graphics		png	Version 1.0
Portable Pixmap		ppm	No specific version
PostScript		ps	Level 2
Progressive JPEG		jpeg	No specific version
Sun Raster		srs	No specific version
TIFF		tiff	Versions through 6
TIFF CCITT Group 3 & 4		tiff	Versions through 6
Truevision TGA		targa	Version 2
Visio (preview)			Version 4
Visio			Version 5, 2000 - 2003
WBMP			No specific version
Windows Enhanced Metafile		emf	No specific version
Windows Metafile		wmf	No specific version
WordPerfect Graphics		wpg, wpg2	Versions through 2.0, 7 and 10
X-Windows Bitmap		xbm	x10 compatible
X-Windows Dump		xdm	x10 compatible
X-Windows Pixmap		xpm	x10 compatible
Presentation Formats
Corel/Novell Presentations		shw	Versions through 12.0
Harvard Graphics for DOS		hgs, cht, ch3, prs	Versions 2.x & 3.x
Harvard Graphics for Windows		hgs, cht, ch3, prs	Windows versions
Freelance for Windows		flw, shw, drw, pre	Versions through Millennium
Freelance for OS/2		flw, shw, drw, pre	Versions through 2.0
Microsoft PowerPoint for Windows		ppt	Versions 3.0 through 2003
Microsoft PowerPoint for Macintosh		ppt	Versions 4.0, 98 through 2004
StarOffice Impress for Windows and UNIX		text only	Versions 5.2
Other Formats
Microsoft Outlook Express		eml	No specific version
Microsoft Outlook Message		msg	Text only
vCard			Version 2.1