The construction sector is one of the largest consumers of natural resources, but also a producer of a considerable amount of waste. Construction and Demolition Waste can be transformed into recycled aggregates and used as a substitute for natural aggregates, either in road construction or concrete, which is one way to reduce the environmental impacts of the construction industry. In order to increase the use of recycled aggregates in high value-added materials like concrete, it is important to guarantee the quality of the produced aggregates. The recycling industry therefore needs new methods to automate the characterization of recycled aggregates. In a previous work we showed that deep convolutional networks can classify the different constituents of recycled aggregates, achieving an average accuracy of 97%. In this work, we propose a novel network architecture called RACNET designed to estimate the mass, the class and the binary mask of recycled aggregates from only 2D images. This could replace advantageously manual sorting tests as well as other geometric characterization tests (like particles size distribution), allowing for a real-time monitoring of recycled aggregates composition. We also present a prototype which preform automatic characterization of a flow of aggregates in real conditions, showing that our approach could be used in real industrial environments