Title 1 2 Bayesian learning models of pain a call to action 3 4 Authors 5 6 *Abby Tabor1, Christopher Burr2 7 8 1Centre for Pain Research, Department for Health, University of Bath, Claverton Down Rd, 9 Bath, BA2 7AY. a.tabor@bath.ac.uk 10 11 2Department of Computer Science, University of Bristol, Merchant Venturers Building, 12 Woodland Road, Clifton BS8 1UB. chris.burr@bristol.ac.uk 13 14 15 * Denotes corresponding author 16 17 18 Abstract 19 20 Learning is fundamentally about action, enabling the successful navigation of a 21 changing and uncertain environment. The experience of pain is central to this 22 process, indicating the need for a change in action so as to mitigate potential threat 23 to bodily integrity. This review considers the application of Bayesian models of 24 learning in pain, which inherently accommodate uncertainty and action, which, we 25 shall propose are essential in understanding learning in both acute and persistent 26 cases of pain. 27 28 29 Highlights 30 31 • The experience of pain sits awkwardly in traditional stimulus-response paradigms 32 • Accommodating uncertainty and action is imperative to learning models of pain 33 • Bayesian models provide a normative, probabilistic account of learning in pain 34 • Learning in pain is conceptualised as an ongoing prediction of the consequences of 35 action 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 2 Introduction 55 56 The process of learning is fundamentally about action. In order to successfully navigate our 57 environment, we must continually learn about the ever-changing limits of our body and the 58 constraints that it imposes upon our interaction with the world. The experience of pain is 59 central to this process, indicating the point at which our bodily integrity is potentially 60 compromised through action. 61 62 The interaction between pain and learning can be better understood from an evolutionary 63 perspective, by adopting the concept of the explore-exploit dilemma [1]. When our bodily 64 integrity is threatened, we typically withdraw or rest (exploit) to allow sufficient recovery to 65 within bodily limits, at which point we decide to interact (explore) within our niche. We learn 66 over time when it is best to exploit and when to explore in order to promote adaptive 67 behaviour [2,3]. 68 69 Learning in pain, however, is not straightforward, owing to the complexity that comprises 70 bodily integrity and worldly state. As a consequence, we find ourselves confronted with the 71 reality that in some cases pain persists, seemingly decoupled from acute protection and 72 adaptive behaviour. This necessarily goes beyond responding to and learning about a 73 nociceptive signal, extending to an overall appraisal of the bodily and sociocultural 74 environments in which we exist [4,5]. Adequately accounting for such a rich and diverse set 75 of interactions is the challenge faced in establishing a learning model in pain. 76 77 Current application of learning models in pain 78 79 Over the last 40 years, associative learning models have come to dominate our conception 80 of learning in pain [6]. These accounts are pervasive in different forms across the pain field, 81 from Pavlovian (habitual) to Operant (instrumental) conditioning in behavioural psychology 82 [7,8], extending to reinforcement learning and temporal difference models in computational 83 neuroscience [9–12]. Operationalised through the Rescorla-Wagner model, the heart of 84 associative learning models lies in the concept of an associative weight between stimulus 85 and response, ranging from immediate, reflexive stimulus-response (model-free) to more 86 complicated goal-directed actions, which alter proceeding stimuli (model-based) [13]. 87 88 Through the application of associative learning theory, it is posited that persistent pain 89 reflects the generalisation of pain-related responses and maintained avoidance behaviour 90 [8,17]. This conceptualisation has shaped our understanding of pain in the behavioural 91 sciences, an influence seen from scientific investigation to clinical management. 92 93 Yet, the experience of pain sits awkwardly in these traditional stimulus-response models 94 [21,22]. In light of recent advances across neuroscience and behavioural domains, there is a 95 growing consensus that perceptual experience is a predictive process, in which learners 96 actively seek information to update their prediction of their internal and external environment 97 [23,24]. This is problematic for traditional associative learning models in pain for several 98 reasons. Firstly, pain is classically posited as a stimulus and conflated with nociception, 99 which downplays the significance of pain as an experience and its explanatory role within 100 theories of learning. Secondly, traditional associative learning models the state of the learner 101 as a series of punctate values at any given time [26], which belies the learner's uncertainty 102 [25–28]. Finally, associative models do not adequately accommodate the active nature of 103 the learner (i.e. being able to actively explore and intervene in their environment) [29]. It is 104 proposed that these challenges for traditional learning models may be overcome by taking a 105 Bayesian approach to learning in pain. 106 107 The Bayesian Framework 108 109 3 Bayesian approaches to cognition comprise many distinct models and theories, used in a 110 variety of domains, and spanning distinct levels of explanation. Often, these distinct 111 approaches are grouped under the label 'Bayesian Brain hypothesis' [30,31], despite their 112 many differences. This review will focus on the underlying Bayesian model that informs 113 these approaches, specifying the Bayesian derivative where appropriate. 114 115 To date, the application of Bayesian models in pain has been limited to the description of 116 perceptual experience, presenting pain as part of a probabilistic inference process that is 117 shaped through the optimal integration of informative cues [27]. These models propose a 118 mechanism for determining the hidden (latent) causes of encountered sensory information, 119 summarised in a generative model [32]. In Bayesian terms, this is achieved through the 120 weighted integration of prior experience and current (potentially multisensory) information, 121 represented using probability distributions that reflect the agent's subjective uncertainty-the 122 optimal integration of these probability distributions is given by Bayes' rule [33] (Fig. 1). 123 124 125 126 127 Fig. 1. Generative models: Prediction of bodily threat (i-iv.). A generative model 128 provides the framework from which predictions of the hidden causes of sensory 129 consequences are generated (posterior), these are continually informed by multisensory 130 sensory cues (likelihood) and previous encounters (prior). The relative precision, reflected in 131 the probability density of these elements, influences the prediction. The more precise 132 (narrow probability distribution), the greater the influence on the prediction. Threat panels 133 (Left: i-ii) demonstrate the relative contribution of either a relatively precise prior (i) or precise 134 likelihood (ii), the resultant prediction of threat is drawn toward the more precise source of 135 information. In these cases, the sensory cue (likelihood) is the same in both panels, yet the 136 relative precision of the prior determines the overall prediction of threat. Safety Panels 137 (Right: iii-iv) demonstrate how the same relative precision can influence the prediction of 138 negative threat, or safety. A precise prior, even in the presence of objective threat-based 139 sensory cues, can influence the overall prediction to reflect safety (placebo effect) (iii). In 140 contrast, an imprecise prior has less influence on the posterior (negative threat/safety) (iv). 141 These hypothetical generative models demonstrate the possible decoupling of objective 142 sensory information from experience, by accounting for the precision of the prior, which 143 reflects the ongoing learning of the individual in keeping with previous experiences, 144 homeostatic bounds and sociocultural constraints. 145 146 4 Although not directly about learning, these accounts expose the fundamental elements of 147 the Bayesian approach: a generative model, subjective uncertainty, and variable precision-148 weighting. It is through the inherent encoding of the learner's uncertainty that Bayesian 149 models can shift away from specific associative weighting between variables towards a 150 learning account that is both predictive and active. This is a significant theoretical 151 development [26], which will form the basis of the proceeding review. 152 153 Learning under uncertainty 154 155 In Bayesian approaches, learners are assumed to have only indirect access to the state of 156 their internal and external environment and must, therefore, infer their values on the basis of 157 ambiguous and often incomplete information [34]. In contrast to associative learning models, 158 Bayesian models encode uncertain beliefs about the world as probability distributions [35]. 159 They assume that learners maintain multiple hypotheses (with differing degrees of belief) 160 that reflect a range of candidate predictions about the state of the body and the world. This 161 invokes the notion of a generative model (Fig.1), which can be used to generate the 162 expected sensory consequences that may arise from hidden (latent) states of the 163 environment, and in absence of external stimulation [36,37]. 164 165 According to Bayesian models, learning occurs through the adjustment of the prior 166 distribution (e.g. estimated threat), according to Bayes rule, when new sensory cues are 167 encountered. This asserts that over time a learner attempts to predict, with increasing 168 finesse, the state of the world. Rather than veridical reflections, these predictions are an 169 integration of probability distributions pertaining to the precision of the information. 170 171 An emerging framework, derived from a Bayesian approach, known as predictive processing 172 [23,38–40] casts the inferential process in probabilistic modelling as a matter of prediction-173 error minimisation. According to this view, the learner's generative model gives rise to 174 multiple top-down predictions that are met by incoming sensory information (prediction 175 error). This is a competitive process, where the prediction that best captures the incoming 176 sensory information is selected, and perception arises as a result of successful prediction-177 error minimisation1. 178 179 The concept of prediction error here represents predictions and prediction-errors as 180 probability distributions, thus retaining the inherent encoding of uncertainty of an agent's 181 beliefs that is common to Bayesian approaches. In predictive processing, specifically, this 182 uncertainty is managed by precision-weighting mechanisms, which modulate the variance 183 associated with the respective distributions, in order to contribute to the overall goal of 184 minimising prediction error [41]. From this perspective, the learner's principle motivation is to 185 minimise the discrepancy between their prediction of the world and the sensory 186 consequences of it (prediction error), in order to ensure they maintain an accurate model of 187 their world (Fig.2). At a cortical level, it has been proposed that precision weighting of 188 prediction errors is mediated by dopamine, with the potential to influence both accurate and 189 aberrant learning [42–44]. 190 191 As we shall explore, the learner can minimise prediction error in two ways: by updating the 192 parameters of their generative model in order to better predict the future sensory 193 consequences of action, or by holding the model fixed and altering their action within the 194 world to sample information that better reflects their predictions. These mechanisms are 195 described under the Active Inference framework [45]. 196 197 1 For a non-technical, conceptual introduction to the Predictive Processing framework see [23]. For for an overview of how the free-energy principle applies to the brain see [57]. 5 198 199 Fig.2. Hierarchical Predictive Processing: from safety to threat. Proposed within a 200 neural hierarchy, generative models are shaped over time to reflect the precision weighting 201 of information. There is a continual bidirectional flow of information at each level of the 202 neural hierarchy involving top-down predictions, prediction error, and the precision-weighting 203 of prediction error. Schematically represented over time, an initial generative model 204 encompassing bodily safety entails the prediction of low bodily threat as a consequence of 205 action. Over time, in the presence of prediction error (a deviation from predicted bodily 206 safety or predicted bodily threat), the generative model is updated to reflect an alteration in 207 action consequences, that of threat. It is suggested that the ability to flexibly update this 208 prediction of threat, in the presence of new sensory evidence (e.g. safety cues), is 209 imperative to the resolution of the need to experience pain. 210 211 Active Learning 212 213 The inherent uncertainty encoded in the agent's probability distributions not only satisfies 214 learning paradigms that are typically challenging for associative theories (e.g backward 215 blocking; see [26]), it crucially affords the agent an active role in reducing uncertainty. Active 216 learning under this formulation is not simply the provision of an adequate sample space 217 (spatial and temporal), it rests on the crucial ability of the learner to intervene in their world, 218 sculpting the sensory consequences of their actions according to what is deemed most 219 salient. The consequences of the learner's actions can either support or disconfirm the 220 predictions of the consequences of action, offering multiple means by which to reduce 221 uncertainty [46,47]. These considerations of active learning recognise ecological validity 222 from the perspective of being in, and acting upon, the world, and where actions are taken 223 based on the ongoing (motivational) homeostatic drives of the biological agent. 224 225 Active Inference2-a component of the predictive processing framework-extends these 226 basic commitments and transforms the role of the learner in pain, from a passive processor 227 of information, to a dynamic predictor of the relationship between the external and internal 228 world. A key claim of the active inference model is that embodied action occurs as a result of 229 2 For a review of active inference, which casts it as a process of descending projections (predictions) from motor cortex, see [43]. Other accounts have implicated the dopaminergic system as playing a key role in active decision-making, while also casting this within the framework of ecological psychology [44]. And, more recently, the active inference framework has been extended to incorporate homeostatic control [40, 51]. For a less technical overview, including empirical and theoretical support, see chapter 4 of [23]. 6 an agent predicting (inferring) the outcomes of certain policies (e.g. reaching for a cup), 230 along with their associated precision estimations. The process of predicting future 231 consequences of actions (i.e. associated sensory information) leads to overt behaviour 232 through the activation of classical reflex arcs by downwards projections from motor cortex 233 [46]. An illustrative example would be a policy that controls an agent's task of reaching for a 234 cup. Prior to enacting the reaching behaviour, the predictions associated with grasping the 235 cup will be unfulfilled, and therefore result in error signals. However, instead of updating the 236 generative model, the agent can instead take actions that lead to the fulfilment of error 237 signals by actually reaching to grab the cup. In cases where the agent predicts that a certain 238 policy will also likely lead to the experience of pain (e.g. bending down to pick up a heavy 239 box), the agent may be reluctant to enact the respective behaviour, or choose to avoid it 240 altogether. 241 242 Seth [48] and others [51] have extended the active inference framework to account for 243 autonomic regulation, arguing that similar predictions generated by the AIC are sent to the 244 autonomic system via smooth muscles to activate autonomic reflexes in a similar manner as 245 earlier described in the case of proprioception. By focussing on the embodied nature of the 246 agent, active inference creates an intuitive segue that unites learning about the state of 247 external world (exteroception) with the state of the internal world (interoception). The same 248 predictive mechanisms that are responsible for predicting sensory states of the external 249 environment are also responsible for regulating the internal environment [48–50] and for 250 providing additional sources of information related to motivational drives [51]. Although often 251 separated in traditional theories, perception and action are entwined in active inference, due 252 to their dual-role in minimising uncertainty [15]. 253 254 The proposition that a single underlying mechanism (i.e. precision-weighted prediction-error 255 minimisation) underlies learning about the condition of the body, has provided instrumental 256 guidance for describing the generation of aberrant bodily predictions and the development of 257 persistent pathological conditions [42,43,52–54]. It is suggested that persistent pain can be 258 formulated in such a way [55]. To illustrate this, the experience of pain is mapped onto the 259 'warning light' scenario, proposed by Adams et al, 2013: 260 261 Consider a circumstance in which you are experiencing knee pain, you predict, with high 262 precision, that the consequences of your action in the world will compromise the integrity 263 of your body. Minor fluctuations in your interoceptive sensory cues (prediction errors) are 264 assigned high precision, which serve to confirm the prediction of potential threat and 265 propagate your experience of pain. You decide to visit your doctor who is unable to 266 determine a specific cause for your pain, they even present you with your x-ray that 267 shows "no structural cause for your pain". Your first thought is that your doctor has 268 missed something, that there must be something else going on, or that the x-ray has 269 been misinterpreted. From your perspective all of these are plausible hypotheses that 270 accommodate the evidence that is available to you. However, from the doctor's 271 perspective, without the knowledge that informs your prediction of bodily threat, your 272 suspicions seem irrational. 273 274 This adapted account highlights the consequences of precision weighting of information in 275 the experience of pain. What is suggested is a decoupling between sensory input and 276 subjective experience, where the latter is dependent on the relative precisions afforded to 277 predictions and prediction error (Fig.2). The learner in pain updates the precision weighting 278 of information that reflects their generative model in a changing world, informing whether to 279 exploit or explore3. This places experiences of the body, whether well-defined through 280 3 Some have proposed that precision-weighting may also be responsible for the transient switching between online and offline control [41]-allowing an agent to deliberate about some future policy, prior to taking action within the world. Although generative models play a central role in guiding online 7 disease process or medically unexplained, on a continuum [55]. What distinguishes them is 281 the accuracy with which they account for the underlying physiological condition of the body. 282 283 Persistent pain, from this view, occurs as aconsequence of precision: either via a precise 284 prediction of bodily threat (top-down) or through aberrant precision weighting of sensory 285 information (bottom up). In both cases, the prediction of bodily threat persists, and so with it 286 the experience of pain, detached from veridical evidence of tissue damage and 287 unchallenged by information assigned less precision. Altering the experience of pain this lies 288 in the ability to promote the flexible reassignment of precision weighting, which in turn alters 289 the individual's prediction about their body and the world. 290 291 The description of learning in pain thus becomes one that concerns optimal precision 292 weighting over time. Importantly, under normative models, optimality does not pertain to 293 accuracy. As such, aberrant but precise predictions of bodily threat (e.g high precision-294 weighting of noisy sensory signals), and an accompanying experience of pain, may persist in 295 the absence of an objective reality of threat. No more or less real, all experiences of the 296 body are a reflection of our evolutionary history, sociocultural present and action-oriented 297 future. 298 299 Discussion 300 301 One core pursuit of learning models in pain is to adequately accommodate the phenomena 302 of acute and persistent cases. That is, why do the majority of people experience pain as 303 transitory-an experience that efficiently promotes acute withdrawal, mitigating further 304 harm-while a significant minority continue to experience pain in a way that seemingly 305 contravenes optimal behaviour? 306 307 We have broadly considered Bayesian models and their relevance to learning in pain. It is 308 proposed that in order to accommodate the ecological validity of the learner in pain, the 309 concepts of uncertainty and active learning must be addressed. As such, derivatives of the 310 Bayesian model have been presented, which attempt to re-conceptualise the learner as an 311 action-oriented predictor of their environment. 312 313 An advantage of this approach is that learning in pain is considered under a unifying 314 framework. The experience of pain becomes a problem of precision-weighting, inherently 315 contextualised in relation to previous experience and future endeavour; both the resolution 316 and persistence of pain lies within one's ability to continually update the predictions of bodily 317 state. 318 319 The approaches that are described are not wholly opposed to the concepts present in 320 associative learning accounts (e.g. kalman filter and the Rescorla Wagner model) [26]. 321 However, a probabilistic formulation of learning promotes an account that naturally extends 322 to the body and action [56], and is highly relevant to learning in pain, whereby the active 323 sampling of one's environment is fundamentally altered. 324 325 behaviour (i.e. active inference), by decoupling generative models from the incoming stream of sensory information (prediction errors), through the use of selective modulation of incoming prediction errors (precision-weighting), generative models may also guide deliberative processes such as planning and offline reasoning [41,51]. This flexible switching between offline and online control could be viewed as a type of arbitration mechanism for model-free and model-based forms of behavioural control, albeit one that may be best viewed as more of a continuum of cases, rather than a welldelineated set of options [41]. 8 Bayesian formulations have proffered much, not least a unifying theory of mind [57]. Yet, 326 with such promise comes inevitable pitfalls [58–60], a number of which require consideration 327 here. 328 329 This review has focussed predominantly on the implementation of such models at an 330 instrumental level, describing the macro phenomena in pain-based learning, without delving 331 into the underlying neural architecture that such probabilistic models aim to account for [61]. 332 Although increasing evidence supports the role of such realist applications in perception, 333 [39,42,62], including in pain [63,64], these are yet to mature into adequate models of 334 complex learning scenarios. Initial investigations comparing models of learning in pain, 335 including generic Bayesian models [65], suggest that there is work to be done to outperform 336 temporal difference models in computational neuroscience paradigms [66]. Consequently, 337 some have argued that the Bayesian Brain should be treated as an instrumental theory in 338 lieu of more developed mechanistic explanations [67]. An important question for the future 339 application of probabilistic models relates to the nature of our experimental paradigms in 340 pain. Using a model, designed to reflect an active learner who minimises uncertainty over 341 time, may demand an alteration in traditional stimulus-response protocols. 342 343 Associative learning theories would be considered incomplete without accounting for value, 344 reward or utility in relation to optimal behaviour. Bayesian generalisations of the Resorla-345 Wagner model, embodied in the Kalman filter, assumes that the target of learning is the 346 problem of predicting immediate reward [68]. However, full active inference accounts aim to 347 replace the notions of reward, value or utility, by subsuming them all within the generative 348 model [13,69]. Whether these concepts can therefore be considered redundant, while still 349 accounting for the complexities of learning in pain and pleasure, is yet to be determined. 350 351 Conclusion 352 353 We have presented a broad overview of Bayesian models of learning in pain. From this 354 view, the experience of pain involves the continual prediction of the consequences of action 355 in relation to bodily threat. As such, learning in pain is both predictive and active. Although 356 there still exist many challenges to the full implementation of such probabilistic accounts, we 357 propose that at present, Bayesian derivatives (such as predictive processing and active 358 inference) can provide important considerations for researchers and clinicians alike. 359 360 Acknowledgements 361 362 The authors would like to extend there gratitude and thanks to Dr Max Jones for his 363 insightful discussion in the critique and development of this review. 364 This research did not receive any specific grant from funding agencies in the public, 365 commercial, or non-for-profit sectors. 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 9 381 382 383 References 384 385 1. Sutton RS, Barto AG: Reinforcement learning: An introduction. MIT Press; 1998. 386 2. McNamara JM, Houston AI, Collins EJ: Optimality models in behavioral biology. 387 SIAM Rev 2001, 43:413–466. 388 3. Houston AI, Higginson AD, McNamara JM: Optimal foraging for multiple nutrients 389 in an unpredictable environment. Ecol Lett 2011, 14:1101–1107. 390 4. Vlaeyen JWS, Linton SJ: Fear-avoidance model of chronic musculoskeletal pain: 391 12 years on. Pain 2012, 153:1144–1147. 392 5. Karos K, Williams AC de C, Meulders A, Vlaeyan J: Pain as a threat to the social 393 self: a motivational account. Pain 2018, In Press. 394 6. Fordyce WE: Behavioural science and chronic pain. Postgrad Med J 1984, 395 60:865–868. 396 7. Meulders A, Vansteenwegen D, Vlaeyen JWS: The acquisition of fear of 397 movement-related pain and associative learning: A novel pain-relevant human 398 fear conditioning paradigm. Pain 2011, 152:2460–2469. 399 8. Vlaeyen JWS: Learning to predict and control harmful events: Chronic pain and 400 conditioning. Pain 2015, 156:S86–S93. 401 9. O'Doherty JP, Seymour B, Koltzenburg M, Frackowiak RSJ, Wiech K, Friston KJ, 402 Dolan RJ: Opponent appetitive-aversive neural processes underlie predictive 403 learning of pain relief. Nat Neurosci 2005, 8:1234–40. 404 10. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ: Temporal difference 405 models and reward-related learning in the human brain. Neuron 2003, 38:329–406 337. 407 11. Seymour B, Daw ND, Roiser JP, Dayan P, Dolan R: Serotonin Selectively 408 Modulates Reward Value in Human Decision-Making. J Neurosci 2012, 32:5833–409 5842. 410 12. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, 411 Frackowiak RS: Temporal difference models describe higher-order learning in 412 humans. Nature 2004, 429:664–667. 413 13. Friston KJ, Daunizeau J, Kiebel SJ: Reinforcement learning or active inference? 414 PLoS One 2009, doi:10.1371/journal.pone.0006421. 415 14. Dayan P, Berridge KC: Model-based and model-free Pavlovian reward learning: 416 Revaluation, revision, and revelation. Cogn Affect Behav Neurosci 2014, 14:473–417 492. 418 15. Gershman SJ, Daw ND: Perception, action and utility the tangled skein. Princ 419 Brain Dyn Glob State Interact 2011, 420 16. Daw ND, Niv Y, Dayan P: Uncertainty-based competition between prefrontal and 421 dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704–422 1711. 423 17. Crombez G, Eccleston C, Van Damme S, Vlaeyen JWS, Karoly P: Fear-avoidance 424 model of chronic pain: the next generation. Clin J Pain 2012, 28:475–83. 425 18. Wiech K, Tracey I: Pain, decisions, and actions: a motivational perspective. Front 426 Neurosci 2013, doi:10.3389/fnins.2013.00046. 427 19. Vlaeyen JWS, Crombez G, Linton SJ: The fear-avoidance model of pain. Pain 428 2016, 157:1588–1589. 429 20. Gatzounis R, Schrooten MGS, Crombez G, Vlaeyen JWS: Operant Learning Theory 430 in Pain and Chronic Pain Rehabilitation. Curr Pain Headache Rep 2012, 16:117–431 126. 432 21. Melzack R, Wall P: The Challenge of pain. Viking Penguin; 1988. 433 22. Wall PD: On the relation of injury to pain. The John J. Bonica Lecture. Pain 1979, 434 6:253–264. 435 10 23. Clark A: Surfing uncertainty: Prediction, action, and the embodied mind. Oxford 436 University Press; 2017. 437 24. Hohwy J: The predictive mind. Oxford University Press; 2013. 438 25. Körding KP, Wolpert DM: Bayesian integration in sensorimotor learning. Nature 439 2004, 427:244–247. 440 26. Kruschke JK: Bayesian approaches to associative learning: From passive to 441 active learning. Learn Behav 2008, 36:210–226. 442 27. Tabor A, Thacker MA, Moseley GL, Körding KP: Pain: A Statistical Account. PLoS 443 Comput Biol 2017, 13. 444 28. Vilares I, Kording K: Bayesian models: the structure of the world, uncertainty, 445 behaviour, and then brain. Ann N Y Acad Sci 2011, 1224:22–39. 446 29. Clark A: Embodied Prediction. In Open MIND. Edited by Metzinger T, Windt JM. 447 MIND Group; 2015:7. 448 30. Doya K, Ishii S, Pouget A, Rao RPN: Bayesian Brain: Probabilistic Approaches to 449 Neural Coding. 2007. 450 31. Friston K: The history of the future of the Bayesian brain. Neuroimage 2012, 451 62:1230–1233. 452 32. Griffiths TL, Kemp C, Tenenbaum JB: Bayesian models of cognition. In Cambridge 453 handbook of computational cognitive modeling. 2007:1–49. 454 33. Pouget A, Beck JM, Ma WJ, Latham PE: Probabilistic brains: Knowns and 455 unknowns. Nat Neurosci 2013, 16:1170–1178. 456 34. Trommershauser J, Kording KP, Landy MS: Sensory Cue Integration. Oxford 457 University Press; 2011. 458 35. Kording KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, Shams L: Causal 459 inference in multisensory perception. PLoS One 2007, 2. 460 36. Koller D, Friedman N: Probabilistic graphical models: principles and techniques. MIT 461 Press; 2009. 462 37. Chater N, Oaksford M: The probabilistic mind: Prospects for Bayesian cognitive 463 science. Oxford University Press; 2008. 464 38. Clark A: Whatever next? Predictive brains, situated agents, and the future of 465 cognitive science. Behav Brain Sci 2013, 36:181–204. 466 39. Hohwy J: Priors in perception: Top-down modulation, Bayesian perceptual 467 learning rate, and prediction error minimization. Conscious Cogn 2017, 47:75–85. 468 40. Seth AK: The Cybernetic Bayesian Brain. In Open MIND. . 2015. 469 41. Clark A: The many faces of precision (Replies to commentaries on "Whatever 470 next? Neural prediction, situated agents, and the future of cognitive science." 471 Front Psychol 2013, 4. 472 42. Haarsma J, Fletcher P, Ziauddeen H, Spencer T, Diederen K: Precision weighting 473 of cortical unsigned prediction errors is mediated by dopamine and benefits. 474 bioRxiv 2018, doi:10.1101/288936. 475 43. Adams R a, Stephan KE, Brown HR, Frith CD, Friston KJ: The computational 476 anatomy of psychosis. Front Psychiatry 2013, 4:47. 477 44. Friston KJ, Shiner T, FitzGerald T, Galea JM, Adams R, Brown H, Dolan RJ, Moran 478 R, Stephan KE, Bestmann S: Dopamine, affordance and active inference. PLoS 479 Comput Biol 2012, doi:10.1371/journal.pcbi.1002327. 480 45. *Friston K, FitzGerald T, Rigoli F, Schwartenbeck P, O'Doherty J, Pezzulo G: Active 481 inference and learning. Neurosci Biobehav Rev 2016, 68:862–879. 482 An active inference accoun of learning. The authors discuss model-based and model free 483 learning in the context of belief. Here, the value function is absorbed into a single functional 484 belief, centred around uncertainty. 485 486 46. Friston K, Mattout J, Kilner J: Action understanding and active inference. Biol 487 Cybern 2011, doi:10.1007/s00422-011-0424-z. 488 47. Seth AK, Friston KJ: Active interoceptive inference and the emotional brain. 489 11 Philos Trans R Soc Lond B Biol Sci 2016, 371. 490 48. Seth AK: Interoceptive inference, emotion, and the embodied self. Trends Cogn 491 Sci 2013, 17:565–573. 492 49. *Owens AP, Allen M, Ondobaka S, Friston KJ: Interoceptive inference: From 493 computational neuroscience to clinic. Neurosci Biobehav Rev 2018, 90:174–183. 494 The authors consider predictive processing and active inference as potential frameworks to 495 describe the role of interoceptive inference in health and disease. Building upon the EPIC 496 model put forward by Barrett and Simmons, 2015, they assert that pathology may arise as a 497 consequence of deficits in the attenuation of ascending prediction error. 498 499 50. Pezzulo G: Why do you fear the bogeyman? An embodied predictive coding 500 model of perceptual inference. Cogn Affect Behav Neurosci 2014, 14:902–11. 501 51. Pezzulo G, Rigoli F, Friston K: Active inference, homeostatic regulation and 502 adaptive behavioural control. Prog Neurobiol 2015, 134:17–35. 503 52. Barrett LF, Simmons WK: Interoceptive predictions in the brain. Nat Rev Neurosci 504 2015, doi:10.1038/nrn3950. 505 53. Edwards MJ, Adams RA, Brown H, Pareés I, Friston KJ: A Bayesian account of 506 "hysteria." Brain 2012, 135:3495–3512. 507 54. Brown RJ: Medically unexplained symptoms: a new model. Psychiatry 2006, 508 doi:10.1383/psyt.2006.5.2.43. 509 55. Van den Bergh O, Witthöft M, Petersen S, Brown RJ: Symptoms and the body: 510 Taking the inferential leap. Neurosci Biobehav Rev 2017, 511 doi:10.1016/j.neubiorev.2017.01.015. 512 56. *Clark: Busting out: predictive brains, embodied minds, and the puzzle of the 513 evidentiary veil. Nous 2016, doi:doi: 10.1111/nous.12140. 514 An accessible overview of the predictive brain literature, with a poignant attempt to extend 515 the scope from a neurocentric mind to an extended mind. The brain from this account is just 516 part of the embodied architecture that predicts the consequences of action. 517 518 57. Friston K: The free-energy principle: A unified brain theory? Nat Rev Neurosci 519 2010, doi:10.1038/nrn2787. 520 58. Bowers JS, Davis CJ: Bayesian just-so stories in psychology and neuroscience. 521 Psychol Bull 2012, doi:10.1037/a0026450. 522 59. *Klein C: What do predictive coders want? Synthese 2016, doi:10.1007/s11229-523 016-1250-6. 524 A pessimistic overview of Predictive Coding/Procesing as unifying theories of mind, 525 critiquing the ability of such accounts to accommodate motivation. The use of the dark room 526 problem is central. Klein reconciles the application by providing insights into how such 527 accounts can be enhanced through embodied frameworks. 528 529 60. Jones M, Love BC: Bayesian fundamentalism or enlightenment? on the 530 explanatory status and theoretical contributions of bayesian models of 531 cognition. Behav Brain Sci 2011, 34:169–188. 532 61. Rasmussen D, Eliasmith C: God, the devil, and the details: Fleshing out the 533 predictive processing framework. Behav Brain Sci 2013, 36:223–224. 534 62. Gordon N, Koenig-Robert R, Tsuchiya N, Van Boxtel JJA, Hohwy J: Neural markers 535 of predictive coding under perceptual uncertainty revealed with hierarchical 536 frequency tagging. Elife 2017, 6. 537 63. Buchel C, Geuter S, Sprenger C, Eippert F: Placebo analgesia: a predictive coding 538 perspective. Neuron 2014, 81:1223–1239. 539 64. Anchisi D, Zanon M: A Bayesian perspective on sensory and cognitive 540 integration in pain perception and placebo analgesia. PLoS One 2015, 10:1–20. 541 65. Mathys C: A Bayesian foundation for individual learning under uncertainty. Front 542 Hum Neurosci 2011, doi:10.3389/fnhum.2011.00039. 543 12 66. Zhang S, Mano H, Lee M, Yoshida W, Kawato M, Robbins T, Seymour B: The 544 control of pain by active relief learning. Elife 2018, 7:e31949. 545 67. Colombo M, Series P: Bayes in the brain-on Bayesian modelling in 546 neuroscience. Br J Philos Sci 2012, 63:697–723. 547 68. *Kruschke JK: Bayesian approaches to associative learning: From passive to 548 active learning. Learn Behav 2008, doi:10.3758/LB.36.3.210. 549 A core text expressing the differences and similarities of traditional associative learning 550 models in relation to Bayesian approaches. A demonstration of how uncertainty and active 551 learning are accommodated by a Bayesian model, accounted for backward blocking 552 phenomena. 553 554 69. Friston KJ: What is optimal about motor control? Neuron 2011, 72:488–498. 555 70. Friston KJ, Daunizeau J, Kiebel SJ: Reinforcement learning or active inference? 556 PLoS One 2009, 4. 557