Disease assessment involves two stages of decision-making process. The first process is to infer the development trend of the disease through the existing judgment conditions, which is to make a positive estimation (prediction) decision. The second process is to update the judgment conditions for different disease states and optimize the clinical assessment evidence by fusing disease progression data, that is, to make a reverse evaluation (optimization) decision by reviewing historical diagnostic and treatment data to summarize the best judgment conditions for the disease characteristics. Therefore, we construct a three-way bidirectional decision method for multi-modal incomplete labeling information, considering the mixed data features with no labels and labels, and the bidirectional decision features of positive prediction and reverse optimization. Specifically, the method first introduces semi-supervised learning (SSL) into the rough sets, constructs a rough sets model based on cosine similarity relation, generates pseudo-labels for incomplete information, and explains the positive prediction decision process principle. Second, the gradient space is constructed on the predictive information system by the three-way decision-making principle, the loss function formed by the combination of KL divergence and cross entropy loss function is used to find the direction of the fastest decline in attribute information value, and the optimal decision rule and the automatic updating strategy of the threshold value are obtained. At the same time, the principle of the reverse optimization decision process is explained. Finally, multimodal information from pathological tissue images is used to evaluate clinical progress problem. A pathological tissue images dataset includes 2000 colorectal cancer and 300 kidney disease cases used for simulation experiments. The proposed three-way bidirectional decision-making model can be widely applied to actual decision-making scenarios for evaluating clinical progress of different diseases. Compared with previous studies, this method can realize objective calculation and automatic update of threshold and loss values and effectively solve complex prediction and optimization decision-making problems.