Many post-disaster and post-conflict regions do not have sufficient data on their transportation infrastructure assets, hindering both mobility and reconstruction. In particular, as the number of ageing and deteriorating bridges increases, it is necessary to quantify their load characteristics in order to inform maintenance and asset databases. The load carrying capacity and the design load are considered as the main aspects of any civil structures. Human examination can be costly and slow when expertise is lacking in challenging scenarios. In this paper, we propose to employ deep learning as a method to estimate the load carrying capacity from crowdsourced images. A convolutional neural network architecture is trained on data from over 6000 bridges, which will benefit future research and applications. We observe significant variations in the dataset (e.g. class interval, image completion, image colour) and quantify their impact on the prediction accuracy, precision, recall and F1 score. Finally, practical optimization is performed by converting multiclass classification into binary classification to achieve a promising field use performance.