Using ultrasound for point-of-care lung assessment is becoming more and more relevant. Today, the outbreak of the coronavirus disease 2019 (COVID-19) has spread over the world at a very high rate. The most severe cases of COVID-19 are associated with lung damage such as ground-glass opacities and areas of lung consolidation, leading to acute respiratory distress. During the COVID-19 pandemic the need to detect and monitor the lung state is critical. Changes in the COVID-19 lung structure modify the way ultrasound propagates in the lung and are reflected by changes in the appearance of lung ultrasound images. Vertical artifacts known as B-lines appear and can evolve into white lung patterns in the more severe cases. Currently, these artifacts are assessed by trained physicians and sonographers, and the diagnosis is qualitative and operator dependent. We propose an automatic segmentation method using a convolutional neural network, to automatically stage the progression of the disease and predict the severity of the lung damage. By classifying the images based on illness severity we can define different scores—from healthy lung to most severe case—and produce a reliable tool to establish severity of COVID-19.