Accurately evaluating weed density in paddy fields is crucial for achieving high-quality control of rice weeds, as it enables precise application of herbicides or matching of mechanical weeding intensity. However, visual methods alone may not provide ideal classification of weed density in specialized paddy field environments due to complex backgrounds, environmental interference, and visual occlusion. In this study, we propose an innovative method that combines vision and tactile features to achieve accurate weed density evaluation. We first obtained tactile data containing weed density information using a self-made tactile sensor, while simultaneously capturing corresponding visual images. We then utilized the kernel canonical correlation analysis (KCCA) algorithm to maximize the correlation of discriminating features extracted from the tactile data and visual images, and obtained fusion eigenvectors that characterize the weed density. These eigenvectors were used as input for a broad learning system (BLS), in which the random feature mapping was replaced by a cascade feature node to construct a KCCA-based cascade feature broad learning system (KCCA-CFBLS). Our method achieved accurate evaluation of weed density in low, medium, and high weed conditions using a novel weed dataset from a specialized paddy field environment. We analyzed the accuracy and running time, and found that KCCA-CFBLS outperformed methods based on SVM and YOLOv5-Lite, achieving 11.56% and 7.56% higher accuracy, respectively, with less time consumption. Experimental results demonstrated significant advantages of our method in terms of accuracy and real-time performance compared to visual methods, providing a decision-making basis for the implementation of intelligent chemical and mechanical weeding in specialized paddy field environments.