Embodied Sensorimotor Control: Computational Modeling of the Neural Control of Movement.

We review how sensorimotor control is dictated by interacting neural populations, optimal feedback mechanisms, and the biomechanics of bodies. First, we outline the distributed anatomical loops that shuttle sensorimotor signals between cortex, subcortical regions, and spinal cord. We then summarize evidence that neural population activity occupies low-dimensional, dynamically evolving manifolds during planning and execution of movements. Next, we summarize literature explaining motor behavior through the lens of optimal control theory, which clarifies the role of internal models and feedback during motor control. Finally, recent studies on embodied sensorimotor control address gaps within each framework by aiming to elucidate neural population activity through the explicit control of musculoskeletal dynamics. We close by discussing open problems and opportunities: multi-tasking and cognitively rich behavior, multi-regional circuit models, and the level of anatomical detail needed in body and network models. Together, this review and recent advances point towards reaching an integrative account of the neural control of movement.

Article Figures and data Abstract eLife digest Introduction Results Discussion Materials and methods References Decision letter Author response Article and author information Metrics Abstract Grasping requires translating object geometries into appropriate hand shapes. How the brain computes these transformations is currently unclear. We investigated three key areas of the macaque cortical grasping circuit with microelectrode arrays and found cooperative but anatomically separated visual and motor processes. The parietal area AIP operated primarily in a visual mode. Its neuronal population revealed a specialization for shape processing, even for abstract geometries, and processed object features ultimately important for grasping. Premotor area F5 acted as a hub that shared the visual coding of AIP only temporarily and switched to highly dominant motor signals towards movement planning and execution. We visualize these non-discrete premotor signals that drive the primary motor cortex M1 to reflect the movement of the grasping hand. Our results reveal visual and motor features encoded in the grasping circuit and their communication to achieve transformation for grasping. https://doi.org/10.7554/eLife.15278.001 eLife digest In order to grasp and manipulate objects, our brains have to transform information about an object (such as its size, shape and position) into commands about movement that are sent to our hands. Previous work suggests that in primates (including humans and monkeys), this transformation is coordinated in three key brain areas: the parietal cortex, the premotor cortex and the motor cortex. But exactly how these transformations are computed is still not clear. Schaffelhofer and Scherberger attempted to find out how this transformation happens by recording the electrical activity from different brain areas as monkeys reached out to grasp different objects. The specific brain areas studied were the anterior intraparietal (AIP) area of the parietal cortex, a part of the premotor cortex known as F5, and the region of the motor cortex that controls hand movements. The exact movement made by the monkeys' hands was also recorded. Analysing the recorded brain activity revealed that the three brain regions worked together to transform information about an object into commands for the hand, although each region also had its own specific, separate role in this process. Neurons in the AIP area of the parietal cortex mostly dealt with visual information about the object. These neurons specialized in processing information about the shape of an object, including information that was ultimately important for grasping it. In contrast, the premotor area F5 represented visual information about the object only briefly, quickly switching to representing information about the upcoming movement as it was planned and carried out. Finally, the neurons in the primary motor cortex were only active during the actual hand movement, and their activity strongly reflected the action of hand as it grasped the object. Overall, the results presented by Schaffelhofer and Scherberger suggest that grasping movements are generated from visual information about the object via AIP and F5 neurons communicating with each other. The strong links between the premotor and motor cortex also suggest that a common network related to movement executes and refines the prepared plan of movement. Further investigations are now needed to reveal how such networks process the information they receive. https://doi.org/10.7554/eLife.15278.002 Introduction Grasping objects of different shapes and sizes appears trivial in daily life. We can distinguish between thousands of objects (Biederman, 1987) and shape our hands according to their geometry in order to hold and manipulate them (Napier, 1956; Smeets and Brenner, 1999). Although such operations seem to be effortless, their underlying neuronal mechanisms are highly complex and require extensive computational resources (Fagg and Arbib, 1998; Felleman and Van Essen, 1991). The cortical grasping network needs to translate high-dimensional visual information of an object into multidimensional motor signals that control the complex biomechanics of the hand. In the primate brain, these processes are linked to the anterior intraparietal (AIP), the ventral premotor (F5), and the primary motor cortex (M1) (Brochier and Umilta, 2007; Castiello, 2005; Davare et al., 2011; Nelissen and Vanduffel, 2011). Within this network, AIP provides access to the dorsal visual stream that processes vision for action (Culham et al., 2003; Goodale et al., 1994). In fact, neurons in AIP were shown to strongly respond to the presentation of graspable objects or 3D contours (Murata et al., 2000; Taira et al., 1990; Theys et al., 2012b), but could also encode specific grip types (Baumann et al., 2009). This grasp-relevant information processed in AIP is exchanged with F5 via dense reciprocal connections (Borra et al., 2008; Gerbella et al., 2011; Luppino et al., 1999). Accordingly, deactivation of AIP or F5 causes severe deficits in pre-shaping the hand while approaching an object (Fogassi et al., 2001; Gallese et al., 1994). In contrast to AIP, concurrent electrophysiological studies suggest that F5 is primarily encoding objects in motor terms and is storing context-specific grip type information (Fluet et al., 2010; Raos et al., 2006). Connections of the dorsal subdivision of F5 (F5p) to the spinal cord and to M1 provide further evidence for the important role of F5 for grasp movement preparation (Borra et al., 2010; Dum and Strick, 2005). These electrophysiological and anatomical observations lead to our current understanding of the fronto-parietal network as the main circuitry for translating object attributes into motor commands (Jeannerod et al., 1995; Rizzolatti and Luppino, 2001). In detail, it has been suggested that visual features extracted in AIP activate motor prototypes in F5, which store hand configurations according to an object's geometry (Rizzolatti and Luppino, 2001). However, the detailed neural mechanisms of these processes remained unclear. To create a deeper understanding of how visual information is transformed into motor commands, a precise identification and differentiation of visual and motor processes within the grasping network is required. Previous important grasping studies classified visual-dominant, visual-motor or motor-dominant neurons primarily on the phase of their activation [for AIP see Murata et al. (2000) and Sakata et al. (1995), for F5 see Raos et al. (2006); Theys et al. (2012a)], but they could not discriminate between neural coding of visual features of objects or motor features of the hand. A differentiation between visual and motor coding is challenging for multiple reasons. First, the fronto-parietal network is multimodal and can reflect sensory and motor signals simultaneously. Second, visual and motor descriptions of objects and the hand are multidimensional due to the complexity of object geometry and hand physiology. Investigations at the neural level therefore necessitate multidimensional observations from many neurons. Third, the visual and motor spaces are highly linked to each other since the form of an object often defines the shape of the grasping hand. Disassociating both neuronal representations therefore requires highly variable visual stimuli and motor responses. In this study, we took a multidimensional approach to identify and separate visual and motor processes in the grasping network of AIP, F5, and M1. We recorded large populations of neurons simultaneously from the entire network and compared their modulation patters to the visual attributes of highly diverse objects and the kinematic features recorded from the grasping hand. Our data revealed distinct roles of the grasping network in translating visual object attributes (AIP) into planning (F5) and execution signals (M1) and allowed visualizing the propagation of these features for grasping. Results Two macaque monkeys grasped a large set of 49 objects causing highly variable visual stimuli and motor responses (Figure 1a–b, Video 1). During the experiments we recorded hand and arm kinematics from an instrumented glove (Schaffelhofer and Scherberger, 2012) (see Figure 1c, Video 2) in conjunction with neuronal activity from 6 cortical microelectrode arrays (6 x 32 channels) (Figure 1e–g). Figure 1 Download asset Open asset Behavioural design and implantation details. (a–b) Two monkeys were trained to grasp a total of 48 objects presented on a PC-controlled turntable. In addition, monkeys were instructed to perform either precision or power grips on a handle. Each of the 50 grasping conditions was denoted with a double-digit number (ID1, ID2), a colour code, and a symbol to allow easy identification throughout this manuscript. (c) An instrumented glove equipped with electro-magnetic sensors allowed monitoring and recording the monkeys' hand and arm kinematics in 27 DOF. (d) All grasping actions were performed as a delayed grasp-and-hold task consisting of eye-fixation, cue, planning, grasping and hold epochs. (e–g) Neural activity was recorded simultaneously from six floating microelectrode arrays implanted in the cortical areas AIP, F5, and M1. (f) Electrode placement in monkey Z (right hemisphere). Each array consisted of 2 ground and 2 reference electrodes (black), as well as 32 recording channels (white) aligned in a 4x9 matrix. Electrode length for each row increased towards the sulcus from 1.5–7.1 mm. (g) Same for monkey M (left hemisphere). Two arrays were implanted in each area. AIP: toward the lateral end of the intraparietal sulcus (IPS); F5: on the posterior bank of the arcuate sulcus (AS); hand area of M1: on the anterior bank of the central sulcus (CS). https://doi.org/10.7554/eLife.15278.003 Video 1 Download asset This video cannot be played in place because your browser does support HTML5 video. You may still download the video for offline viewing. Download as MPEG-4 Download as WebM Download as Ogg Experimental task. A monkey grasped and held highly variable objects presented on a PC-controlled turntable. Note: For presentation purposes, the video was captured in the light. https://doi.org/10.7554/eLife.15278.004 Video 2 Download asset This video cannot be played in place because your browser does support HTML5 video. You may still download the video for offline viewing. Download as MPEG-4 Download as WebM Download as Ogg Hand and arm tracking. 18 joints of the primate hand were tracked with electromagnetic sensors and used to drive a 3-D primate-specific musculoskeletal model to extract 27 joint angles. Thumb, index, wrist, elbow, and shoulder angles are shown while the monkey is grasping a ring, a ball and a cylinder. The video runs at half speed. https://doi.org/10.7554/eLife.15278.005 Training the monkeys to perform grasping movements in a delayed grasp-to-hold paradigm allowed us to investigate neural activity at several key stages of the task. As shown in Figure 1d, visual responses (i.e., object presentation in cue epoch), planning activity (i.e., motor preparation in planning epoch), and motor execution signals (i.e., grasp and hold epoch) were temporarily distinct and could therefore be explored separately. We analysed data from 20 recording sessions of two macaque monkeys (10 sessions per animal). On average, spiking activity of 202 ± 7 and 355 ± 20 (mean ± s.d.) single and multiunits were collected in each session in monkey Z and M, respectively. Of these units, 29.2% and 25.2% were recorded from AIP, 37.3% and 32.3% from F5, and 33.5% and 42.5% from M1 (monkey Z and M, respectively). Vision for hand action Presenting 3D objects to the monkeys lead to vigorous discharge (Baumann et al., 2009; Murata et al., 2000) of AIP-neurons (Figure 2a–b). The modulated population was not only larger (sliding ANOVA, Figure 3), but also significantly faster appearing after stimulus onset than in F5 (49.7 ms and 54.9 ms, monkey M and Z respectively). Impressively, individual AIP cells were capable of differentiating object shapes at high precision (e.g., Figure 2a). To quantify this attribute, we computed a modulation depth analysis that determined the relative difference in firing rate between all pairs of conditions (objects) during the cue epoch (see Materials and methods). The example cell of Figure 2b revealed a chequered structure caused by the shape-wise order of object conditions 00–76 (for object id, see Figure 1b) and a maximum modulation depth (MD) of 62 Hz. Statistical analysis between all conditions (ANOVA and post-hoc Tukey-Kramer criterion, p<0.01; see Materials and methods) revealed a high encoding capacity of the example neuron that could significantly separate 71% of the 946 condition pairs (44 conditions). Interestingly, the neuron decreased its MD in darkness but maintained its encoding of shape (as also indicated in Figure 2a). Figure 2 with 2 supplements see all Download asset Open asset Visual object processing in area AIP. (a) Example neuron of AIP responding to the presentation of graspable objects (each curve represents one task condition). (b) Modulation depth plot illustrating the absolute firing rate difference in the cue epoch between all condition pairs (conditions 00 – 76 placed on axis in ascending order). Warm colours: high modulation depth, cool colours: low modulation depth. (c) Shape-wise clustering of objects in the AIP population during the cue epoch, as demonstrated by CDA. Arrows indicate a shift in position when big horizontal cylinders (red triangles) were grasped from below instead from above (black triangles). (d) Same as c, but during the grasp epoch. (e–f) Dendrograms illustrating the neural distance between object conditions in the simultaneously recorded AIP population in the cue and grasp epoch (N = 62). Symbols and colour code in a, c-f as in Figure 1b. Percentages in c and d describe how much variance of the data is explained by the shown components (1st, 2nd and 3rd). Note: Video 3 visualizes the N-space of AIP of an additional recording in the same animal (Z). See Figure 2—figure supplement 1–2 for the averaged population results of animal Z and animal M, respectively. https://doi.org/10.7554/eLife.15278.006 Video 3 Download asset This video cannot be played in place because your browser does support HTML5 video. You may still download the video for offline viewing. Download as MPEG-4 Download as WebM Download as Ogg Population coding in AIP. The first three canonical variables of the AIP population are shown in 3D and are animated for presentation purposes. Each symbol represents one trial. Symbols and colours as in Figure 1b. https://doi.org/10.7554/eLife.15278.009 Figure 3 Download asset Open asset Visual processing of object shapes. (a) A set of six 'mixed' objects elicited different visual stimuli and different motor responses. (b) Percentage of tuned neurons of the AIP, F5, and M1 population express the significant modulation with respect to the mixed objects across time (sliding one-way ANOVA). (c) Tuned neurons (shades of red) were mapped to their recording location during the visual (t = 0.16 s after object presentation) and motor phase (t = 0.7 s after movement onset). (d) As a contrast and to elicit pure visual responses, 'abstract' objects caused different visual stimuli but the same grip. (e) Similar to b, but for the abstract objects set. (f) Similar to c, but showing the map of tuned neurons (shades of green) with respect to the abstract object set. For b, e: Data is doubly aligned on cue onset and on the grasp (go) signal. Sliding ANOVA was computed for each session individually and averaged across all 10 recording sessions per animal. Shades represent standard error from mean (s.e.m.) across recording sessions. For c, f: The number of tuned neurons per channel were averaged across all recording sessions and visualized in shades of green and red for the abstract and mixed objects, respectively. Channels without any identified neurons were highlighted in light grey. Map of monkey M is mirrored along vertical axis for better comparison of both animals. https://doi.org/10.7554/eLife.15278.010 To investigate this effect at the neuronal population level, we performed canonical discriminant analysis (CDA; see Materials and methods), which allowed reducing the neuronal state space (N-space) to its most informative dimensions. Figure 2c and 2d show the first three canonical variables during the cue and grasp epoch, respectively. In them, each marker represents the neuronal state of an individual trial in the AIP population (see Figure 1b for symbol and colour code). In this N-space of AIP, we found objects to be separated based on their shape. Independent of the way the objects were grasped, the neural space accurately differentiated horizontal cylinders (black), vertical cylinders (green), rings (magenta), spheres (orange), cubes (blue), and bars (black) (see Video 3 for an animated 3D view of a typical N-space). Video 4 Download asset This video cannot be played in place because your browser does support HTML5 video. You may still download the video for offline viewing. Download as MPEG-4 Download as WebM Download as Ogg Population coding in F5. Joint angles and the population activity of F5 were recorded together and for visual display reduced to their most informative dimensions (component 1 and 2). The video displays the evolving hand kinematics (top, left) and neuronal population activity (bottom, left) during three subsequent grasping actions. Arrows point at these trials in the J- (top, right) and N-space (bottom, right). The audio-track plays the spiking activity of an individual F5 neuron, which is highlighted in the raster plot in blue. https://doi.org/10.7554/eLife.15278.011 To further quantify these findings, we computed the Mahalanobis distance between all pairs of conditions in the complete N-space of AIP (see Materials and methods). Hierarchical cluster analysis (HCA) performed on these distance measures confirmed the findings of the CDA and revealed a clear clustering according to object shape during visual presentation of the object (Figure 2e) that widely persisted during movement execution, although with significantly shorter neural distances (Figure 2f). 100% and 91% of the objects shared their cluster with other objects of the same shape during the cue and grasp epoch, respectively. Importantly, consistent results were observed in both monkeys when performing HCA across all recording sessions (Figure 2—figure supplement 1–2, see Materials and methods). The large number of objects presented in one recording session required separating the 48 objects on different turntables (see Figure 1), often objects of similar shape. This separation created small offsets already in the epoch, but at low An is shown in Figure This on coding in AIP was due to object or to the object presentation order turntables presented However, shape-wise clustering in AIP cannot be explained by the task design for the The offsets in the epoch were small in comparison to the visual observed in AIP when the objects were presented (e.g., see Video The set of 'mixed' objects – presented and grasped in the same – with other objects of the same shape in mixed with other this clear shape processing in AIP. The AIP population also encoded the of objects, but differentiated this at neural compared to object shape. As shown in Figure and the of objects were to objects of similar size, but with significantly shorter neural distances in comparison to shape The role of was since object has significant on the of the grasping hand and 1991). To further the for shape processing, we a mixed set of objects (Figure causing highly variable visual stimuli and motor responses, an abstract object set (Figure that we have to different visual stimuli but the same grip. In a visual area both of since they both provide different visual In contrast, an motor area not show for the abstract objects because they require the same grip. the AIP population highly similar to the presentation epoch) of mixed (Figure and abstract (Figure objects and mixed and abstract objects in monkey Z and M, respectively). the responses to both object strongly the of AIP in processing object shapes. Importantly, AIP remained the most tuned area when the monkeys planned and grasped the abstract objects, as shown in Figure However, the number of significantly tuned cells decreased during these in comparison to the responses by the mixed objects (Figure This reduced could indicate either motor or visual transformations that are both required for First, the abstract objects were grasped with the same hand (see Figure activity could therefore reflect the same motor across the six abstract objects (Fagg and Arbib, 1998; Rizzolatti and Luppino, 2001). Second, the abstract objects have different but have the graspable in common that the same dimensions across all six objects (see Figure A modulation could therefore also represent visual processes that the objects to its for grasping the same geometries of the across the six abstract that the same AIP population separated the complete object set primarily on their features (Figure suggest visual than motor transformations We found further for this when on objects that in contrast to the abstract objects, visual stimuli but different motor responses. To create such a monkeys were trained to perform power or precision grips on the same object, the 00 and Although both conditions were most in the kinematic or space in both monkeys (see Figure and they were to each other in the N-space of AIP (see Figure a visual of the handle. Statistical analysis that both conditions increased their neural distance (Figure towards planning and movement execution, as by an of and of significantly tuned AIP neurons in monkey Z and M (ANOVA in grasp epoch, These observations suggest a visual of the and a further differentiation of its that are for grasping. Figure 4 Download asset Open asset planning and execution in F5. (a) Example neuron of F5, responding to all 50 task conditions code as in Figure (b) Modulation depth (as in Figure in the planning and grasp epoch. (c) kinematics was used to drive a musculoskeletal model that allowed 27 DOF. (d) A of is presented for three and wrist, elbow, and (e) performed on the during the hold epoch allowed visualizing the grip types of all conditions and trials of one recording session and 2nd plot the spiking activity of recorded from a single firing during the grasp epoch (N-space) were transformed with CDA to and visualize the multidimensional of the complete F5 population (N = simultaneously In example trials and are highlighted in epoch in grasp epoch in and with in state space the of the task determined by the CDA. and For visual comparison the N-space was aligned to the Symbols and colours as in Figure symbol to object Figure with 2 supplements see all Download asset Open asset Hierarchical cluster analysis of the F5 (a) of of complete N-space during the plan and the grasp epoch. as in Figure 1b. A of grip types and their objects are In similar motor are highlighted with (see is based on the complete F5 population (N = simultaneously recorded in contrast to its in the reduced neural space in Figure See Figure supplement 1–2 for the averaged population results across all sessions of animal M and animal respectively. the AIP population all horizontal cylinders based on their shape and differentiated the two horizontal cylinders see Figure when they were grasped from or from conditions required a on different of the object as well as different hand configurations Importantly, the neural of both conditions from the same shape cluster in N-space (Figure that further (see in Figure these observations suggest a visual than motor in AIP. planning and execution To grasping visual attributes of objects to be transformed into motor commands they (Jeannerod et al., 1995; Rizzolatti and Luppino, 2001). a motor plan and its execution is with areas F5 and M1 (Murata et al., Raos et al., et al., In a first analysis that compared neuronal population for the mixed (Figure and abstract objects (Figure we found evidence for a primary motor role of F5 and M1. The F5 population was strongly in the motor when the mixed objects were grasped to and tuned neurons in monkey Z and M see Figure and it was when the abstract objects were grasped similar (Figure the M1 population for similar grips (Figure but it was modulated strongly when different hand configurations were to and in monkey Z an M see Figure During movement planning, M1 or preparation activity (Figure F5 revealed a multimodal in the cue epoch, the tuned F5 population decreased from to in monkey and from to in monkey M, when abstract with mixed objects. This is in strong contrast to AIP, which demonstrated population responses for both type of object The reduced F5 modulation suggests motor processes after object However, and Z and M of all F5 neurons remained their modulation when the monkeys observed the abstract at F5 cells objects in visual is that we found significantly different in motor preparation between the F5 recording The visual (Figure and (Figure to movement primarily from the ventral recording to the subdivision et al., 2011; Theys et al., In fact, of the visual and of tuned neurons recorded from F5 were on the ventral (ANOVA all in cue epoch), in with of planning signals from ventral F5 (Schaffelhofer et al., In contrast, the dorsal F5 to during movement execution by a of its tuned population with respect to the cue epoch. coding in area F5 The population response of F5 was confirmed when our analysis to all 49 objects. Neurons were modulated by hand grasping actions and the MD during motor execution. The example neuron shown in Figure demonstrated a maximum MD of while grasping and allowed significantly separating of all condition pairs in this epoch (Figure right). Importantly, the neuron similar motor

Our ability to perform a variety of difficult tasks everything from reasoning about the best chess move, to shooting a free throw, or finely dicing an onion is due to the coordinated activity of populations of neurons throughout the nervoussystem. And yet, we lack an understanding of how the brain generates the activity appropriate for achieving something as simple as pressing an elevator button. In part, this is because we do not know which neural activity patterns the brain is capable of generating, nor how that activity will change with experience. By exploring the structure and constraints on the activity patterns the brain can express, we move closer to understanding how the brain can generate the activity supportive of such a rich variety of behaviors and adaptations. Presently, in studies of arm or eye movements, we typically don't know the causal relationship between neural activity and behavior. Here we use a brain-computer interface (BCI) paradigm to study learning, because the exact relationship between neural activity and behavior is controlled by the experimenter. To generate proficient behavior, the animal must change the activity of the neurons currently being recorded. This provides us with the means to causally relate any observed structure in neural population activity with animals' performance at the task. The focus of this thesis is to characterize the structure and time course of neural population activity during learning. In the first part of this thesis, we note that just as there is more than one way to win a game of chess, the brain has many differentpatterns of neural activity it can produce to drive the same behavior. Which of these redundant options does the brain prefer? We find that the frequency with which animals used different patterns of neural population activity was remarkably similar before and after learning. This suggests that the brain's ability to take advantage of redundancy may be somewhat limited, at least within the span of a few hours. In the second part of this thesis, we asked how internal states such as our arousal, attention, and motivation interact with how we learn new tasks. We identified large, abrupt fluctuations in neural population activity in motor cortex indicative of arousal-like internal state changes, which we term \neural engagement. We find that stereotyped changes in neural engagement during learning were unrelated to goal-seeking behavior, but nevertheless influenced how quickly different tasks were learned. Overall, this thesis characterizes a variety of different constraints and influences on how populations of neurons change their activity during learning.

The motor system routinely generates a multitude of fast, accurate, and elegant movements.In large part, this capacity is enabled by closed-loop feedback control systems in the brain.Brain-machine interfaces (BMIs), which translate neural activity into control signals for drivingprosthetic devices, also engage the brain’s feedback control systems and offer a promisingexperimental paradigm for studying the neural basis of feedback motor control. Here, we addressboth the engineering challenges facing current BMI systems and the basic science opportunitiesafforded by them.Previous studies have demonstrated reliable control of the direction of movement in cursorbasedBMI systems. However, control of movement speed has been notably deficient. We providean explanation for these observed difficulties based on neurophysiological studies of armreaching. These findings inspired our design of a novel BMI decoding algorithm, the speeddampeningKalman filter (SDKF) that automatically slows the cursor upon detecting changesin decoded movement direction. SDKF improved success rates by a factor of 1.7 relative to astandard Kalman filter in a closed-loop BMI task requiring stable stops at targets.Next, we transition toward leveraging the BMI paradigm for basic scientific studies of feedbackmotor control. It is widely believed that the brain employs internal models to describe ourprior beliefs about how an effector responds to motor commands. We developed a statisticalframework for extracting a subject’s internal model from neural population activity. We discoveredthat a mismatch between the actual BMI and the subjects internal model of the BMI explainsroughly 65% of movement errors. We also show that this internal model mismatch limits movementspeed dynamic range and may contribute toward the aforementioned known difficulties incontrol of BMI movement speed.

Dynamics on the manifold: Identifying computational dynamical activity from neural population recordings.

Human motor behavior involves planning and execution of actions, some more frequently. Manipulating probability distribution of a movement through intensive direction-specific repetition causes physiological bias toward that direction, which can be cortically evoked by transcranial magnetic stimulation (TMS). However, because evoked movement has not been used to distinguish movement execution and plan histories to date, it is unclear whether the bias is because of frequently executed movements or recent planning of movement. Here, in a cohort of 40 participants (22 female), we separately manipulate the recent history of movement plans and execution and probe the resulting effects on physiological biases using TMS and on the default plan for goal-directed actions using a timed-response task. Baseline physiological biases shared similar low-level kinematic properties (direction) to a default plan for upcoming movement. However, manipulation of recent execution history via repetitions toward a specific direction significantly affected physiological biases, but not plan-based goal-directed movement. To further determine whether physiological biases reflect ongoing motor planning, we biased plan history by increasing the likelihood of a specific target location and found a significant effect on the default plan for goal-directed movements. However, TMS-evoked movement during preparation did not become biased toward the most frequent plan. This suggests that physiological biases may either provide a readout of the default state of primary motor cortex population activity in the movement-related space, but not ongoing neural activation in the planning-related space, or that practice induces sensitization of neurons involved in the practiced movement, calling into question the relevance of cortically evoked physiological biases to voluntary movements.SIGNIFICANCE STATEMENT Human motor performance depends not only on ability to make movements relevant to the environment/body's current state, but also on recent action history. One emerging approach to study recent movement history effects on the brain is via physiological biases in cortically-evoked involuntary movements. However, because prior movement execution and plan histories were indistinguishable to date, to what extent physiological biases are due to pure execution-dependent history, or to prior planning of the most probable action, remains unclear. Here, we show that physiological biases are profoundly affected by recent movement execution history, but not ongoing movement planning. Evoked movement, therefore, provides a readout of the default state within the movement space, but not of ongoing activation related to voluntary movement planning.

Many aspects of perception and cognition are supported by activity in neural populations that are tuned to different stimulus features (e.g., orientation, spatial location, color). Goal-directed behavior, such as sustained attention, requires a mechanism for the selective prioritization of contextually appropriate representations. A candidate mechanism of sustained spatial attention is neural activity in the alpha band (8-13 Hz), whose power in the human EEG covaries with the focus of covert attention. Here, we applied an inverted encoding model to assess whether spatially selective neural responses could be recovered from the topography of alpha-band oscillations during spatial attention. Participants were cued to covertly attend to one of six spatial locations arranged concentrically around fixation while EEG was recorded. A linear classifier applied to EEG data during sustained attention demonstrated successful classification of the attended location from the topography of alpha power, although not from other frequency bands. We next sought to reconstruct the focus of spatial attention over time by applying inverted encoding models to the topography of alpha power and phase. Alpha power, but not phase, allowed for robust reconstructions of the specific attended location beginning around 450 msec postcue, an onset earlier than previous reports. These results demonstrate that posterior alpha-band oscillations can be used to track activity in feature-selective neural populations with high temporal precision during the deployment of covert spatial attention.

To successfully guide limb movements, the brain takes in sensory information about the limb, internally tracks the state of the limb, and produces appropriate motor commands. It is widely believed that this process uses an internal model, which describes our prior beliefs about how the limb responds to motor commands. Here, we leveraged a brain-machine interface (BMI) paradigm in rhesus monkeys and novel statistical analyses of neural population activity to gain insight into moment-by-moment internal model computations. We discovered that a mismatch between subjects’ internal models and the actual BMI explains roughly 65% of movement errors, as well as long-standing deficiencies in BMI speed control. We then used the internal models to characterize how the neural population activity changes during BMI learning. More broadly, this work provides an approach for interpreting neural population activity in the context of how prior beliefs guide the transformation of sensory input to motor output.DOI:http://dx.doi.org/10.7554/eLife.10015.001

To successfully guide limb movements, the brain takes in sensory information about the limb, internally tracks the state of the limb, and produces appropriate motor commands. It is widely believed that this process uses an internal model, which describes our prior beliefs about how the limb responds to motor commands. Here, we leveraged a brain-machine interface (BMI) paradigm in rhesus monkeys and novel statistical analyses of neural population activity to gain insight into moment-by-moment internal model computations. We discovered that a mismatch between subjects' internal models and the actual BMI explains roughly 65% of movement errors, as well as long-standing deficiencies in BMI speed control. We then used the internal models to characterize how the neural population activity changes during BMI learning. More broadly, this work provides an approach for interpreting neural population activity in the context of how prior beliefs guide the transformation of sensory input to motor output.

Considerable progress has been made over the last decades in characterizing the neural coding of hand shape, but grasp force has been largely ignored. We trained two macaque monkeys (Macaca mulatta) on a delayed grasping task where grip type and grip force were instructed. Neural population activity was recorded from areas relevant for grasp planning and execution: the anterior intraparietal area (AIP), F5 of the ventral premotor cortex, and the hand area of the primary motor cortex (M1). Grasp force was strongly encoded by neural populations of all three areas, thereby demonstrating for the first time the coding of grasp force in single- and multi-units of AIP. Neural coding of intended grasp force was most strongly represented in area F5. In addition to tuning analysis, a dimensionality reduction method revealed low-dimensional responses to grip type and grip force. Additionally, this method revealed a high correlation between latent variables of the neural population representing grasp force and the corresponding latent variables of electromyographic forearm muscle activity. Our results therefore suggest an important role of the cortical areas AIP, F5, and M1 in coding grasp force during movement execution as well as of F5 for coding intended grasp force.

Movements are defining characteristics of all behaviors. Animals walk around, move their eyes to explore the world or touch structures to learn more about them. So far we only have some basic understanding of how the brain generates movements, especially when we want to understand how different areas of the brain interact with each other. In this study we investigated the influence of sensory object information on grasp planning in four different brain areas involved in vision, touch, movement planning, and movement generation in the parietal, somatosensory, premotor and motor cortex. We trained one monkey to grasp objects that he either saw or touched beforehand while continuously recording neural spiking activity with chronically implanted floating multi-electrode arrays. The animal was instructed to sit in the dark and either look at a shortly illuminated object or reach out and explore the object with his hand in the dark before lifting it up. In a first analysis we confirmed that the animal not only memorizes the object in both tasks, but also applies an object-specific grip type, independent of the sensory modality. In the neuronal population, we found a significant difference in the number of tuned units for sensory modalities during grasp planning that persisted into grasp execution. These differences were sufficient to enable a classifier to decode the object and sensory modality in a single trial exclusively from neural population activity. These results give valuable insights in how different brain areas contribute to the preparation of grasp movement and how different sensory streams can lead to distinct neural activity while still resulting in the same action execution.

The unfolding of neural population activity can be described as a dynamical system. Stability in the latent dynamics that characterize neural population activity has been linked with consistency in animal behavior, such as motor control or value-based decision-making. However, whether such characteristics of neural dynamics can explain visual perceptual behavior is not well understood. To study this, we recorded V4 populations in two male monkeys engaged in a non-match-to-sample visual change-detection task that required sustained engagement. We measured how the stability in the latent dynamics in V4 might affect monkeys' perceptual behavior. Specifically, we reasoned that unstable sensory neural activity around dynamic attractor boundaries may make animals susceptible to taking incorrect actions when withholding action would have been correct ("false alarms"). We made three key discoveries: (1) greater stability was associated with longer trial sequences; (2) false alarm rate decreased (and response times slowed) when neural dynamics were more stable; and (3) low stability predicted false alarms on a single-trial level, and this relationship depended on the position of the neural activity within the state space, consistent with the latent neural state approaching an attractor boundary. Our results suggest the same outward false alarm behavior can be attributed to two different potential strategies that can be disambiguated by examining neural stability: (1) premeditated false alarms that might lead to greater stability in population dynamics and faster response time and (2) false alarms due to unstable sensory activity consistent with misperception.

The manner in which neural activity unfolds over time is thought to be central to sensory, motor and cognitive functions in the brain. Network models have long posited that the brain’s computations involve time courses of activity that are shaped by the underlying network. A prediction from this view is that the activity time courses should be difficult to violate. We leveraged a brain–computer interface to challenge monkeys to violate the naturally occurring time courses of neural population activity that we observed in the motor cortex. This included challenging animals to traverse the natural time course of neural activity in a time-reversed manner. Animals were unable to violate the natural time courses of neural activity when directly challenged to do so. These results provide empirical support for the view that activity time courses observed in the brain indeed reflect the underlying network-level computational mechanisms that they are believed to implement.

Traditionally, movement-related behavior is estimated using activity from motor regions in the brain. This predictive capability of interpreting neural signals into tangible outputs has led to the emergence of Brain-Computer Interface (BCI) systems. However, nonmotor regions can play a significant role in shaping how movements are executed. Our goal was to explore the contribution of nonmotor brain regions to movement using a unique experimental paradigm in which local field potential recordings of several cortical and subcortical regions were obtained from eight epilepsy patients implanted with depth electrodes as they performed goal-directed reaching movements. The instruction of the task was to move a cursor with a robotic arm to the indicated target with a specific speed, where correct trials were ones in which the subject achieved the instructed speed. We constructed subject-specific models that predict the speed error of each trial from neural activity in nonmotor regions. Neural features were found by averaging spectral power of activity in multiple frequency bands produced during the planning or execution of movement. Features with high predictive power were selected using a forward selection greedy search. Using our modeling framework, we were able to identify networks of regions related to attention that significantly contributed to predicting trial errors. Our results suggest that nonmotor brain regions contain relevant information about upcoming movements and should be further studied.

Rapid categorization of visual objects is critical for comprehending our complex visual world. The role of individual cortical neurons and neural populations in categorizing visual objects during passive vision has previously been studied. However, it is unclear whether and how perceptually guided behaviors affect the encoding of stimulus categories by neural population activity in the higher visual cortex. Here we studied the activity of the inferior temporal (IT) cortical neurons in macaque monkeys during both passive viewing and categorization of ambiguous body and object images. We found enhanced category information in the IT neural population activity during the correct, but not wrong, trials of the categorization task compared to the passive task. This encoding enhancement was task difficulty dependent with progressively larger values in trials with more ambiguous stimuli. Enhancement of IT neural population information for behaviorally relevant stimulus features suggests IT neural networks' involvement in perceptual decision-making behavior.