, 2006). Note that the sensory prediction errors in predictive coding (Tseng et al., 2007 and Wei and Körding, 2009) have nothing to do with reward prediction errors in optimal control and reinforcement learning learn more (Schultz and Dickinson, 2000 and Gläscher et al., 2010). Sensory prediction errors are required for online state estimation (inference) and optimizing (learning) the forward model. Conversely, reward prediction errors are concerned solely with learning the inverse model, in terms of value functions

or cost-to-go (the path integral of cost under optimal control). Reward prediction errors are generally invoked in the context of reward learning; however, exactly the same errors are required when learning the cost-to-go in motor control. In summary, it is straightforward to cast optimal motor control in terms of predictive coding. In this setting, the forward model is part of a selleckchem generative model mapping from control to sensory consequences. This distinction may be trivial from the perspective of optimal control schemes, but it is important for active inference, as we will see. Figure 2 distinguishes between exteroceptive and proprioceptive prediction errors on sensations caused by (hidden) states in extrinsic and intrinsic frames of reference. Here, the (high-dimensional) intrinsic frame contains the state of the motor plant (e.g.,

muscle fibers). Conversely, the (low-dimensional) extrinsic frame contains movement in extrapersonal space (e.g., a head-centered frame of reference). Intrinsic and extrinsic frames are used in the sense of Kakei et al. (2003) and Shipp (2005): Kakei et al. discuss movement representations in terms of the coordinate transformations that begin with an “extrinsic coordinate frame representing the spatial location of a target and end with an intrinsic coordinate frame describing muscle activation patterns.” In Feldman and Levin

(1995), these frames of reference are considered in terms of physical (intrinsic) and action-perception (extrinsic) frames. The distinction is important because optimal control has to invert a mapping from (1) control signals already to consequences in an intrinsic (muscle-based) frame and then (2) from an intrinsic to an extrinsic (movement-based) frame in which desired movement is defined. In short, the inverse mapping comprises two parts: from an extrinsic to an intrinsic frame and from an intrinsic frame to control signals. The second part of the inversion is easy because there is a simple relationship between motor neuron activity and its consequences (if an alpha motor neuron fires, its extrafusal muscle fibers contract). However, the first part makes inversion difficult because there are many intrinsic degrees of freedom that interact to produce a trajectory in extrinsic coordinates.