Asbestos is known to be an independent risk factor for lung and pleural cancers. However, to date, little attention has been paid to the psychological effects of asbestos exposure among exposed subjects. The objectives of this study were to estimate the prevalence of anxious and depressive symptoms among >2000 French participants of the Asbestos-Related Diseases Cohort (ARDCO), 6 years after their inclusion, to identify the risk factors associated with those anxious and depressive symptoms and to evaluate the impact of the asbestos-risk perception. The ARDCO was constituted in four regions of France between October 2003 and December 2005, by including former asbestos workers. Between 2011 and 2012, participants of the ARDCO program were invited to undergo another chest CT scan 6 years after the previous scan. Participants were asked to complete questionnaires including asbestos exposure assessment, Hospital Anxiety and Depression Scale (HADS), asbestos-risk perception and self-perception of asbestos-related diseases. Among the 2225 participants, 2210 fully completed questionnaires were collected and analyzed. The prevalence of symptoms of probable anxiety and probable depression was 19.7% and 9.9%, respectively. The risk of anxious and depressive symptoms was independently associated with self-perception of the intensity of asbestos exposure, asbestos-risk perception and self-perception of asbestos-related diseases. The results obtained in this large study confirm that previously asbestos-exposed subjects are likely to develop anxious and depressive symptoms. Finally, implications related to the prevention of anxiety and depression among asbestos-exposed workers is discussed.