My favorites | Sign in
Project Home
READ-ONLY: This project has been archived. For more information see this post.
Search
for
Model_description  
structure of VBA's generative model
Updated Nov 4, 2013 by jean.dau...@gmail.com

This section exposes the structure of generative models that underpin VBA data analysis.

Example: Q-learning model

Reinforcement learning models are typically used to interpret changes in behavioural responses that arise from subject's exposure to reward and/or punishment. Among these, Q-learning models simply assume that subjects update the value of possible actions. In its simplest form, the Q-learning algorithm expresses the change in value from trial to trial as being linearly proportional to the reward prediction error. This yields the following update rule:

where is the reward delivered to the subject at trial , and is the (unknown) learning rate of the subject.

One usually complements Q-learning with a softmax decision rule, i.e. an equation that expresses the probability of the subject to choose action at trial :

where is the (unknown) inverse temperature.

Given a series of experienced reward at each trial, these equations can be used to predict the choices of the subject. Fitting the above Q-learning model to behavioural data means finding estimates of the learning rate , the inverse temperature , and the initial values that best explains the observed choices (see this page for a demonstration).

In fact, although they are not general enough to capture the range of models that the toolbox can deal with, these equations convey the basic structure of models of learning and decision making. This is because:

  • any form of learning (including probabilistic - bayesian - belief update), can be written as an evolution equation, similar in form to the first equation above
  • any form of decision making can be understood as an action emission law, and thus written as an observation mapping (from internal states to actions), similar in form to the second equation above.

More generally, most computational models for neurobiological and behavioural data share the above structure (evolution and/or observation mappings), which captures the response of relevant states (e.g. neural activity, beliefs and preferences, etc...) to experimentally controlled inputs. We will now describe the general structure of these models in finer details.

Nonlinear state-space models

Definitions & Notations

Let us first recall the notations that are used when experimentally observing the behaviour of a subject:

  • : experimentally measured data. These can be categorial or continuous. In the example above, data are composed of the observed choice at each trial.
  • : hidden states at time or trial . These are time-dependent, and their motion is controlled by the evolution function (see below). In the example above, hidden states are the value of each accessible action.
  • : evolution parameters. These determine the evolution function of hidden states. In the example above, the only evolution parameter is the learning rate.
  • : observation parameters. These determine the observation mapping. In the example above, the only observation parameter is the inverse temperature.
  • : experimentally controlled inputs. In the example above, the inputs are the reward delivered to the subject.
  • : so-called generative model. This encompasses all statistical assumptions that subtend the analysis. In the example above, the generative model includes both learning and decision making equations, as well as priors on model parameters (see below).

The goal of the statistical analysis is to derive both the posterior density and the model evidence . The former quantifies the amount of information one possesses about unknown model parameters, and the latter is used for model comparison.

Evolution and observation mappings

We consider so-called "state-space models", which essentially consist of two mappings:

  • The evolution function describes how hidden states change from one time sample to the next:

where is supposed to be iid Gaussian, with mean zero and precision (inverse variance) . In the example above, the evolution function was given by the Q-learning equation, and the state noise precision was infinite (deterministic dynamics, i.e.: ).

  • The observation mapping describes how observed data is generated from hidden states. When dealing with continuous data, the observation equation is given by:

where is iid Gaussian, with mean zero and precision . In any case, the observation mapping specifies the likelihood data . In the example above, the likelihood of observed choices was given by the softmax mapping (categorical data).

The following figure summarizes the model structure:

The plate denotes repetitions over time or trials. Nodes represent variables. Gray nodes represent variables that are known by the experimenter (observed data and controlled inputs). White nodes represent unknown variables (hidden states and parameters of the model). Arrows represent causal dependencies between the variables.

One may have to deal with deterministic systems (). In this case, the trajectory of hidden states through time is controlled by the inputs , the evolution parameters and the initial conditions .

Note that the above class of generative models encompasses static models, i.e. models without hidden states (nor evolution parameters):

This simpler structure is closer to decision making models, whereby subject do not engage in learning (e.g., inter-temporal choices).

In fact, (nonlinear) state-space models with unknown evolution, observation and precision parameters is very general. This is because the form of the evolution and observation mappings is arbitrary (e.g., nonlinear). Their inversion grand-fathers most causal models of the statistical literature, with the exception of "switch" models, i.e. models that include unknown categorical variables.

Prior knowledge

Any data analysis relies upon prior knowledge. For example, the form of the evolution and/or observation mappings is a prior. Within a bayesian framework, the subjective aspect of the inference is made further explicit in the definition of a prior probability distribution over unknown model variables.

Priors can vary in how informative they are. This is important because the more informative they are, the stronger the influence on the posterior they have. Here, the informativeness is related to how tight prior distribution are. For Gaussian densities, this relates to the covariance matrix (informative = low variance, uninformative = high variance). Note: whatever the priors, their contribution to the posterior tends to be null as the size of data tends to infinity.

  • : priors on hidden states are provided through the form of the evolution function, which induces a (gaussian) transition probability density with mean and precision
  • and : priors on evolution and observation parameters are Gaussian distributions that are fully parameterized by their first two moments.
  • and : priors on state and observation noise precisions are Gamma distributions that are fully parameterized by their scale () and shape () hyperparameters. For example, one can inform the VBA toolbox that the system is deterministic by assuming a priori that is 0 with infinite precision ( and ).

In brief, the generative model includes the evolution and observation functions as well as the above priors on evolution, observation and precision parameters. All these are required to perform a bayesian analysis of experimental data.

Comment by project member jean.dau...@gmail.com, Apr 12, 2012

very nice summary!

Powered by Google Project Hosting