Computational Methods for Behavior Analysis

Abstract

Behavioral scientists strive to decode the functional relationship between sensory input and motor output of the brain, which requires quantitive measurement of animal behavior. Artificial intelligence researchers aim to build intelligent systems, capable of understanding, predicting, and generating behavior. Our research lies on the intersection of the two fields; our goal is to automate measurement of animal behavior and to model their sensory-motor relationship using machine learning. We have developed a tool that tracks the pose of multiple fruit flies and aims to maintain their identity throughout a video. It outputs motion trajectories that can be used to quantify behavioral differences between individuals, for example by comparing histograms of velocities and wing angles. We show that the tool also works well on non-fly-like animals such as zebrafish larvae. Embedded in these motion trajectories are temporal patterns that constitute actions. We developed two supervised learning frameworks for action detection: a sliding window framework and a structured output framework. Both frameworks learn to classify actions from motion trajectories and expert annotated action intervals. Our results show that the simpler sliding window framework achievers better results in spite of being much faster to train, reaching 90% of human performance. Supervised learning requires a lot of training data which involves time consuming and painstaking annotation. To alleviate that we have built a semi-supervised neural network framework that, in addition to classifying actions, learns to predict how an animal will move next given its motion and sensory inputs so far. Our model archives as good results as its supervised counterpart with only half of the expert labels. In addition, we show that motion prediction can be used to generate convincing simulations of fruit fly behavior and handwritten text, and that our model learns to represent high level information, such as identity, when trained unsupervised. Although developed for animal behavior, our methods are general and could be applied to other motion data. We hope that this thesis demonstrates the value of studying animal behavior for the development of artificial intelligence.