Introduction

CIF allows you to do this. It has a feed-parser pulls everything you tell it to (eg: threat feeds, RT feeds, etc) and normalizes all the un-organized data into a massive repository so that over time; you develop a chronological repo of threat (or describing) data about as many observations of data as possible.

The key here, is that it parses various data-formats (xml, rss, text, csv, etc) and normalizes so you can search it, compare it, correlate it across feed sources during an investigation.

A great 3rd party explanation can be found here

the Community

the CommunityRules
How others are leveraging CIF
Join the community
Jump in to IRC on freenode #cif
Feature Requests, Bugs should be logged here, or they will get lost.

Getting Started

Please note: Due to the various content "licenses" for each of the feeds (malwaredomains, zeustracker, etc) We do NOT offer this as a service, we can't give you an api-key to "test with". As an org; you're free to download this data, but we're not able to do it for you, which is why we give you the tools... we teach you how to pish... :)

using the client
Read the FAQ
deploying your own server

Advanced

messing with the REST API
using the Feeds
writing your own cif_feedparser config
ProjectsLab
Potential new feed sources
CookBook

Details

At it's core; it's a simple re-implementation of how social networks appear to store data in a NoSQL-ish kind of way. This framework takes any type of regex-able feed:

Spamhaus DROP list
malwaredomains.com feed
malware.com.br feeds
sshbl.org feed
DRG ssh feed
and more....

The framework:

normalizes the information
indexes it on the fly in temporary index tables for faster searching
allows you to query/generate feeds against these index tables
relate back to the original data messages, usually stored in JSON+IODEF format).

The original idea came from from:

http://bret.appspot.com/entry/how-friendfeed-uses-mysql

This framework pulls in various data-observations from any source; create a series of messages "over time" (eg: reputation). When you query for the data, you'll get back a series of messages chronologically and make decisions much as you would look at an email thread, a series of observations about a particular bad-actor.