The explosive growth of the Internet has elevated social networks to a pivotal role in information propagation, reshaping conventional paradigms of information distribution. Utilizing the vast amount of data available online to model and predict this diffusion is of great importance in various fields. Existing approaches for information cascade prediction fall into three main categories: feature-driven methods, point process-based methods, and deep learning-based methods. Among them, deep learning-based methods, characterized by their superior learning and representation capabilities, mitigate the shortcomings inherent in the other methods.However, current deep learning methods still face several persistent challenges. In particular, accurate representation of user attributes remains problematic due to factors such as fake followers and complex network configurations. Previous algorithms that focused on the sequential order of user activations often neglected the rich insights offered by activation timing. Furthermore, these techniques often fail to holistically integrate temporal and structural aspects, thus missing the nuanced propagation trends inherent in information cascades.To address these issues, we propose the Cross-Domain Information Fusion Framework (CasCIFF), which is tailored for information cascade prediction. This framework exploits multi-hop neighborhood information to make user embeddings robust. When embedding cascades, the framework intentionally incorporates timestamps, endowing it with the ability to capture evolving patterns of information diffusion. In particular, CasCIFF seamlessly integrates the tasks of user classification and cascade prediction into a consolidated framework, thereby allowing the extraction of common features that prove useful for all tasks, a strategy anchored in the principles of multi-task learning. After extensive experiments conducted on publicly available datasets, the results demonstrate CasCIFF's superiority over established baseline methods in terms of prediction accuracy.