(NOTE: the ICCLE1 experiment is now over.

While anyone is free to update this code, please note that new developments are now happening in a daughter code base: http://code.google.com/p/iccle2.)

Welcome to Insightful Code (Common Lisp edition)

"Too many doings, not enough learnings."

When was the last time your code added insight to your business?

For example, how often do your business users interrupt the code demo to phone the CEO?

"Wendy," they shout down the line, "we have to change the western division. The programmers have just shown us a way to optimizing production by 30%."

Then they storm out of the demo, spilling coffee and tipping over chairs, in their race to reorganize the business.

This is a very rare event and this is our fault. We geek too much. After matching brackets for days/ weeks/ years, all our beautiful code achieves only low-level programming goals and not the high-level business goals they are meant to serve.

Insightful Code

We need more insightful code- systems that can:

• can learn from experience.
• that can reveal to business users new and exciting knowledge.

Current data mining methods are often not insightful since they are focus on their internal algorithmic details, rather than their impact on business goals. Also, some of those methods may be needlessly complex. As Janez Demsar observes:

• In many business situations, data miners should not solve the problem. Rather, they should help people solve the problem. And when helping people, more complex methods like neural networks and support vector machines are not as useful as, say, simple symbolic classifiers like classification trees and naive Bayesian classifier with appropriate visualizations.
• No classifier can beat a good scatter plot- just show the data appropriately and let the user decipher what it tells. Forget about the fancy modeling techniques: we are the best hardware with the best software there is. Computers are only good at drawing and at searching through possible visualizations, so let's use them for that.

We therefore seek a simple code library that offers new insights to the business users about their business. To do this, we combine data mining with the cognitive patterns seen at the business-level:

• The business goals are expressed as a cognitive tasks; e.g. monitor, predict, construct, etc. This is the whole story but if contains holes : numerous details not specified in the high level story.
• Data mining is used to fill in the holes; i.e. learn the details of the high-level tasks that tune it to a particular organizational context.

What's Different?

Our library is different from standard data mining toolkits.

• Decades of data mining research has shown us how to build effective data miners using very simple components.
• Here, we exploit that research to build simple components and show how they can service the business users' cognitive tasks.

Insightful coding is:

• Model-light: the high-level cognitive tasks;
• And data-heavy: we assume that an organization has enough data available to tunes its business model. And if this data is absent, we will build a high-level simulator of the business to generate the data needed for this approach.

• Such standard modeling practices are model-heavy: many people, brainstorming complex business models.
• But data-light: very little feedback from the real world to refine, maintain, and optimize those models.

In standard software design, modeling is a mostly-manual early life cycle activity with very little support for model updates. Insightful coding, on the other hand, offers much support for model maintenance. Models are maintained from day one and are continually maintained through-out the lifecycle.