Brain aging is a complex and heterogeneous process characterized by both structural and functional decline. This study aimed to establish a novel deep learning (DL) method for predicting brain age by utilizing structural and metabolic imaging data. The dataset comprised participants from both the Universal Medical Imaging Diagnostic Center (UMIDC) and the Alzheimer's Disease Neuroimaging Initiative (ADNI). The former recruited 395 normal control (NC) subjects, while the latter included 438 NC subjects, 51 mild cognitive impairment (MCI) subjects, and 56 Alzheimer's disease (AD) subjects. We developed a novel dual-pathway, 3D simple fully convolutional network (Dual-SFCNeXt) to estimate brain age using [18F]fluorodeoxyglucose positron emission tomography ([18F]FDG PET) and structural magnetic resonance imaging (sMRI) images of NC subjects as input. Several prevailing DL models were trained and tested using either MRI or PET data for comparison. Model accuracies were evaluated using mean absolute error (MAE) and Pearson's correlation coefficient (r). Brain age gap (BAG), deviations of brain age from chronologic age, was correlated with cognitive assessments in MCI and AD subjects. Both PET- and MRI-based models achieved high prediction accuracy. The leading model was the SFCNeXt (the single-pathway version) for PET (MAE = 2.92, r = 0.96) and MRI (MAE = 3.23, r = 0.95) on all samples. By integrating both PET and MRI images, the Dual-SFCNeXt demonstrated significantly improved accuracy (MAE = 2.37, r = 0.97) compared to all single-modality models. Significantly higher BAG was observed in both the AD (P < 0.0001) and MCI (P < 0.0001) groups compared to the NC group. BAG correlated significantly with Mini-Mental State Examination (MMSE) scores (r=-0.390 for AD, r=-0.436 for MCI) and the Clinical Dementia Rating Scale Sum of Boxes (CDR-SB) scores (r = 0.333 for AD, r = 0.372 for MCI). The integration of [18F]FDG PET with structural MRI enhances the accuracy of brain age prediction, potentially introducing a new avenue for related multimodal brain age prediction studies.