Project Deprecated

This project was very successful, so much so that we expanded it's scope. Bt-Glue turned into RL-Viz, a Java framework that runs on top and is compatible with RL-Glue to help visualize, debug, and experiment with reinforcement learning agents and environments.

Get Updates

If you'd like to receive e-mail updates about bt-glue or ask questions, please subscribe to the bt-glue Google Groups mailing list. The url is http://groups.google.com/group/bt-glue

General Info

This library is something I'm creating as part of my thesis research. It will eventually cooperate and interface with Adam White's RL-Glue.

So far, the code has support for creating environments, agents, sensors, actions, and reward functions. There are a few example projects, including:

Goals

There are several libraries for reinforcement learning agents out there, so what makes this project different? Some ideas:

• A small, minimal (I hope) C++ based framework for creating flexible agents, environments, sensors, and actions
• Focus on getting a wide variety of environments, with all different types of sensors, accessible through a standardized interface so that a single agent can run on any of them, without recompiling or reconfiguring the agent.
• Focus in visualization of environments, value functions, etc. (this is not novel to our project)

Environments

• Discrete State Grid World (agent senses the state via a unique integer label)
• Continuous State Grid World (agent senses the state via a pair of continuously valued variables)
• The previous worlds basically have the functionality of defining reward regions and soft barriers. The agent gets -1 reward per step unless it enters a reward region, which randomly restarts the agent
• Acrobot problem that was used at the reinforcement learning competition at NIPS 2006
• Mountain Car
• Skeleton World (empty environment ready to be customized by you)

Agents

• Tabular Q(0) agent with discrete (up, down, left, right) actions
• Tile Coding Q(0) agent with discrete (up, down, left, right) actions
• Tile Coding Q(0) agent with structured/continuous actions (dx, dy)
• Tile Coding Sarsa Lambda - adapted from the mountain car agent in the RL-Library. Agent is still experimental, but the internals are generic, it can be used without modification on any continuous state discrete action environment.

Visualizers

• Cocoa (OS X) based visualizer of agent running around in the grid worlds
• Visualize agent position and grid world topology
• Visualize agent value function
• Extensible
• Visualizer of agent position in Acrobot. Working on visualizing the 4 dimensional value function.
• Rough visualizer of agent position and value function in mountain car, needs to be cleaned up.
• WANTED Someone to write a very simple QT or Java based visualizer that does the same as the Cocoa one but works cross platform

The project makes use of a library of abstract classes for agents, environments, reward functions, actions, and sensors that allow you to create generic, reusable components.

Oh yeah, right now the code is all for Mac OS X... and is in the form of an Xcode project. The Xcode stuff is very physically separated from the source code though, so someone with more Unix experience can surely help create makefiles.

Special Thanks

Thanks to everyone that's been helping with this project, including:

• Andrew Butcher
• Adam White http://www.cs.ualberta.ca/~awhite/Publications/Home.html
• Brad Joyce http://www.cs.ualberta.ca/people/grad/bjjoyce.php
• Rich Sutton http://www.richsutton.com