3D Medical Images Research Articles

Enhancing Medical Image Classification with Unified Model Agnostic Computation and Explainable AI

Background: Advances in medical image classification have recently benefited from general augmentation techniques. However, these methods often fall short in performance and interpretability. Objective: This paper applies the Unified Model Agnostic Computation (UMAC) framework specifically to the medical domain to demonstrate its utility in this critical area. Methods: UMAC is a model-agnostic methodology designed to develop machine learning approaches that integrate seamlessly with various paradigms, including self-supervised, semi-supervised, and supervised learning. By unifying and standardizing computational models and algorithms, UMAC ensures adaptability across different data types and computational environments while incorporating state-of-the-art methodologies. In this study, we integrate UMAC as a plug-and-play module within convolutional neural networks (CNNs) and Transformer architectures, enabling the generation of high-quality representations even with minimal data. Results: Our experiments across nine diverse 2D medical image datasets show that UMAC consistently outperforms traditional data augmentation methods, achieving a 1.89% improvement in classification accuracy. Conclusion: Additionally, by incorporating explainable AI (XAI) techniques, we enhance model transparency and reliability in decision-making. This study highlights UMAC’s potential as a powerful tool for improving both the performance and interpretability of medical image classification models.

Temporomandibular joint CBCT image segmentation via multi-view ensemble learning network.

Accurate segmentation of the temporomandibular joint (TMJ) from cone beam CT (CBCT) images holds significant clinical value for diagnosing temporomandibular joint osteoarthrosis (TMJOA) and related conditions. Convolutional neural network-based medical image segmentation methods have achieved state-of-the-art performance in various segmentation tasks. However, 3D medical images segmentation requires substantial global context and rich spatial semantic information, demanding much more GPU memory and computational resources. To address these challenges in 3D medical image segmentation, we propose a novel network- the MVEL-Net (Multi-view Ensemble Learning Network) for TMJ CBCT image segmentation. By resampling images along three dimensions, we generate multiple weak learners with different spatial semantic information. A subsequent strong learning network effectively integrates the outputs from these weak learners to achieve more accurate segmentation results. We evaluated our network model using a clinical dataset comprising 88 subjects with TMJ CBCT images. The average Dice similarity coefficient (DSC) was 0.9817 ± 0.0049, the average surface distance was 0.0540 ± 0.0179mm, and the 95% Hausdorff distance was 0.1743 ± 0.0550mm. Our proposed MVEL-Net demonstrates excellent segmentation performance on TMJ from CBCT images, while using fewer GPU memory resources compared to other 3D networks. The effectiveness of this method in capturing spatial context could be leveraged for tasks like organ segmentation from volumetric scans. This may facilitate wider adoption of AI-based solutions for automated analysis of 3D medical images.

Fast‐MedNeXt: Accelerating the MedNeXt Architecture to Improve Brain Tumour Segmentation Efficiency

ABSTRACTWith the rapid development of medical imaging technology, 3D image segmentation technology has gradually become a mainstream method, especially in brain tumour detection and diagnosis showing its unique advantages. The technique makes full use of 3D spatial information to locate and analyze tumours more accurately, thus playing an important role in improving diagnostic accuracy, optimising treatment planning and promoting research. However, it also suffers from significant computational expenditure and delayed processing pace. In this paper, we propose an innovative optimisation scheme to address this problem. We thoroughly investigate the MedNeXt network and propose Fast‐MedNeXt, which aims to increase the processing speed while maintaining accuracy. First, we introduce the partial convolution (PConv) technique, which replaces the deep convolutional layers in the network. This improvement effectively reduces computation and memory requirements while maintaining efficient feature extraction. Second, based on PConv, we propose PConv‐Down and PConv‐Up modules, which are applied to the up‐sampling and down‐sampling modules to further optimise the network structure and improve efficiency. To confirm the efficacy of the approach, we carried out a sequence of tests in the multimodal brain tumour segmentation challenge 2021 (BraTS2021). By comparing with the MedNeXt series network, the Fast‐MedNeXt reduced the latency by 22.1%, 20.5%, 15.8%, and 11.4% respectively, while the average accuracy also increased by 0.475% and 0.2% respectively. These significant performance improvements demonstrate the effectiveness of Fast‐MedNeXt in 3D medical image segmentation tasks and provide a new and more efficient solution for the field.

CQformer: Learning Dynamics Across Slices in Medical Image Segmentation.

Prevalent studies on deep learning-based 3D medical image segmentation capture the continuous variation across 2D slices mainly via convolution, Transformer, inter-slice interaction, and time series models. In this work, via modeling this variation by an ordinary differential equation (ODE), we propose a cross instance query-guided Transformer architecture (CQformer) that leverages features from preceding slices to improve the segmentation performance of subsequent slices. Its key components include a cross-attention mechanism in an ODE formulation, which bridges the features of contiguous 2D slices of the 3D volumetric data. In addition, a regression head is employed to shorten the gap between the bottleneck and the prediction layer. Extensive experiments on 7 datasets with various modalities (CT, MRI) and tasks (organ, tissue, and lesion) demonstrate that CQformer outperforms previous state-of-the-art segmentation algorithms on 6 datasets by 0.44%-2.45%, and achieves the second highest performance of 88.30% on the BTCV dataset. The code will be publicly available after acceptance.

Hybrid 3D Medical Image Segmentation Using CNN and Frequency Transformer Fusion

Hybrid 3D Medical Image Segmentation Using CNN and Frequency Transformer Fusion

An Efficient and Robust 3D Medical Image Classification Approach Based on 3D CNN, Time‐Distributed 2D CNN‐BLSTM Models, and mRMR Feature Selection

ABSTRACTThe advent of 3D medical imaging has been a turning point in the diagnosis of various diseases, as voxel information from adjacent slices helps radiologists better understand complex anatomical relationships. However, the interpretation of medical images by radiologists with different levels of expertise can vary and is also time‐consuming. In the last decades, artificial intelligence‐based computer‐aided systems have provided fast and more reliable diagnostic insights with great potential for various clinical purposes. This paper proposes a significant deep learning based 3D medical image diagnosis method. The method classifies MedMNIST3D, which consists of six 3D biomedical datasets obtained from CT, MRA, and electron microscopy modalities. The proposed method concatenates 3D image features extracted from three independent networks, a 3D CNN, and two time‐distributed ResNet BLSTM structures. The ultimate discriminative features are selected via the minimum redundancy maximum relevance (mRMR) feature selection method. Those features are then classified by a neural network model. Experiments adhere to the rules of the official splits and evaluation metrics of the MedMNIST3D datasets. The results reveal that the proposed approach outperforms similar studies in terms of accuracy and AUC.

MedLSAM: Localize and segment anything model for 3D CT images

Recent advancements in foundation models have shown significant potential in medical image analysis. However, there is still a gap in models specifically designed for medical image localization. To address this, we introduce MedLAM, a 3D medical foundation localization model that accurately identifies any anatomical part within the body using only a few template scans. MedLAM employs two self-supervision tasks: unified anatomical mapping (UAM) and multi-scale similarity (MSS) across a comprehensive dataset of 14,012 CT scans. Furthermore, we developed MedLSAM by integrating MedLAM with the Segment Anything Model (SAM). This innovative framework requires extreme point annotations across three directions on several templates to enable MedLAM to locate the target anatomical structure in the image, with SAM performing the segmentation. It significantly reduces the amount of manual annotation required by SAM in 3D medical imaging scenarios. We conducted extensive experiments on two 3D datasets covering 38 distinct organs. Our findings are twofold: (1) MedLAM can directly localize anatomical structures using just a few template scans, achieving performance comparable to fully supervised models; (2) MedLSAM closely matches the performance of SAM and its specialized medical adaptations with manual prompts, while minimizing the need for extensive point annotations across the entire dataset. Moreover, MedLAM has the potential to be seamlessly integrated with future 3D SAM models, paving the way for enhanced segmentation performance. Our code is public at https://github.com/openmedlab/MedLSAM.

Translation Consistent Semi-supervised Segmentation for 3D Medical Images.

3D medical image segmentation methods have been successful, but their dependence on large amounts of voxel-level annotated data is a disadvantage that needs to be addressed given the high cost to obtain such annotation. Semi-supervised learning (SSL) solves this issue by training models with a large unlabelled and a small labelled dataset. The most successful SSL approaches are based on consistency learning that minimises the distance between model responses obtained from perturbed views of the unlabelled data. These perturbations usually keep the spatial input context between views fairly consistent, which may cause the model to learn segmentation patterns from the spatial input contexts instead of the foreground objects. In this paper, we introduce the Translation Consistent Co-training (TraCoCo) which is a consistency learning SSL method that perturbs the input data views by varying their spatial input context, allowing the model to learn segmentation patterns from foreground objects. Furthermore, we propose a new Confident Regional Cross entropy (CRC) loss, which improves training convergence and keeps the robustness to co-training pseudo-labelling mistakes. Our method yields state-of-the-art (SOTA) results for several 3D data benchmarks, such as the Left Atrium (LA), Pancreas-CT (Pancreas), and Brain Tumor Segmentation (BraTS19). Our method also attains best results on a 2D-slice benchmark, namely the Automated Cardiac Diagnosis Challenge (ACDC), further demonstrating its effectiveness. Our code, training logs and checkpoints are available at https://github.com/yyliu01/ TraCoCo.

HEDN: multi-oriented hierarchical extraction and dual-frequency decoupling network for 3D medical image segmentation.

Previous 3D encoder-decoder segmentation architectures struggled with fine-grained feature decomposition, resulting in unclear feature hierarchies when fused across layers. Furthermore, the blurred nature of contour boundaries in medical imaging limits the focus on high-frequency contour features. To address these challenges, we propose a Multi-oriented Hierarchical Extraction and Dual-frequency Decoupling Network (HEDN), which consists of three modules: Encoder-Decoder Module (E-DM), Multi-oriented Hierarchical Extraction Module (Multi-HEM), and Dual-frequency Decoupling Module (Dual-DM). The E-DM performs the basic encoding and decoding tasks, while Multi-HEM decomposes and fuses spatial and slice-level features in 3D, enriching the feature hierarchy by weighting them through 3D fusion. Dual-DM separates high-frequency features from the reconstructed network using self-supervision. Finally, the self-supervised high-frequency features separated by Dual-DM are inserted into the process following Multi-HEM, enhancing interactions and complementarities between contour features and hierarchical features, thereby mutually reinforcing both aspects. On the Synapse dataset, HEDN outperforms existing methods, boosting Dice Similarity Score (DSC) by 1.38% and decreasing 95% Hausdorff Distance (HD95) by 1.03 mm. Likewise, on the Automatic Cardiac Diagnosis Challenge (ACDC) dataset, HEDN achieves 0.5% performance gains across all categories.

TPAFNet: Transformer-Driven Pyramid Attention Fusion Network for 3D Medical Image Segmentation.

The field of 3D medical image segmentation is witnessing a growing trend in the utilization of combined networks that integrate convolutional neural networks and transformers. Nevertheless, prevailing hybrid networks are confronted with limitations in their straightforward serial or parallel combination methods and lack an effective mechanism to fuse channel and spatial feature attention. To address these limitations, we present a robust multi-scale 3D medical image segmentation network, the Transformer-Driven Pyramid Attention Fusion Network, which is denoted as TPAFNet, leveraging a hybrid structure of CNN and transformer. Within this framework, we exploit the characteristics of atrous convolution to extract multi-scale information effectively, thereby enhancing the encoding results of the transformer. Furthermore, we introduce the TPAF block in the encoder to seamlessly fuse channel and spatial feature attention from multi-scale feature inputs. In contrast to conventional skip connections that simply concatenate or add features, our decoder is enriched with a TPAF connection, elevating the integration of feature attention between low-level and high-level features. Additionally, we propose a low-level encoding shortcut from the original input to the decoder output, preserving more original image features and contributing to enhanced results. Finally, the deep supervision is implemented using a novel CNN-based voxel-wise classifier to facilitate better network convergence. Experimental results demonstrate that TPAFNet significantly outperforms other state-of-the-art networks on two public datasets, indicating that our research can effectively improve the accuracy of medical image segmentation, thereby assisting doctors in making more precise diagnoses.

Dual Channel‐Spatial Self‐Attention Transformer and CNN synergy network for 3D medical image segmentation

Even though the Vision Transformer leverages the self-attention mechanism to capture long-range dependencies, showing significant potential in medical image segmentation, the limited annotations in the image dataset make it difficult for the Transformer model to extract different global features, resulting in attention collapse and generating similar or identical attention maps. Previous studies have attempted to solve the problem by integrating convolutional neural layers into Transformer-based architectures. However, improper integration may lead to the inability of the model to effectively capture local and global information in both spatial and channel dimensions. To address the above issue, we propose a hybrid architecture using the Dual Channel-Spatial Self-Attention Transformer and CNN Synergy Network (DTC-SUNETR) for medical image segmentation. Specifically, we redesigned the self-attention mechanism. A novel Channel-Spatial Self-Attention (CSSA) block is introduced to integrate the enhanced channel and spatial self-attention mechanism to capture the global relationship and local structure among image features. This helps the model to more comprehensively understand the inter-dependencies between different channels and capture the relationships between different pixels, thus enhancing the feature representation of the corresponding dimensions. Simultaneously, it also improves the overall computational efficiency of the network. Extensive experiments on four different medical image segmentation datasets, including Synapse, ACDC, Brain Tumor, and Lung Tumor, demonstrate the superiority of the proposed DTC-SUNETR over state-of-the-art methods.

Skeleton-guided 3D convolutional neural network for tubular structure segmentation.

Accurate segmentation of tubular structures is crucial for clinical diagnosis and treatment but is challenging due to their complex branching structures and volume imbalance. The purpose of this study is to propose a 3D deep learning network that incorporates skeleton information to enhance segmentation accuracy in these tubular structures. Our approach employs a 3D convolutional network to extract 3D tubular structures from medical images such as CT volumetric images. We introduce a skeleton-guided module that operates on extracted features to capture and preserve the skeleton information in the segmentation results. Additionally, to effectively train our deep model in leveraging skeleton information, we propose a sigmoid-adaptive Tversky loss function which is specifically designed for skeleton segmentation. We conducted experiments on two distinct 3D medical image datasets. The first dataset consisted of 90 cases of chest CT volumetric images, while the second dataset comprised 35 cases of abdominal CT volumetric images. Comparative analysis with previous segmentation approaches demonstrated the superior performance of our method. For the airway segmentation task, our method achieved an average tree length rate of 93.0%, a branch detection rate of 91.5%, and a precision rate of 90.0%. In the case of abdominal artery segmentation, our method attained an average precision rate of 97.7%, a recall rate of 91.7%, and an F-measure of 94.6%. We present a skeleton-guided 3D convolutional network to segment tubular structures from 3D medical images. Our skeleton-guided 3D convolutional network could effectively segment small tubular structures, outperforming previous methods.

CvTMorph: Improving Local Feature Extraction in Medical Image Registration for Respiratory Motion Modeling with Convolutional Vision Transformer.

Accurately modeling respiratory motion in medical images is crucial for various applications, including radiation therapy planning. However, existing registration methods often struggle to extract local features effectively, limiting their performance. In this paper, we aimed to propose a new framework called CvTMorph, which utilizes a Convolutional vision Transformer (CvT) and Convolutional Neural Networks (CNN) to improve local feature extraction. CvTMorph integrates CvT and CNN to construct a hybrid model that combines the strengths of both approaches. Additionally, scaling and square layers are added to enhance the registration performance. We have evaluated the performance of CvTMorph on the 4D-Lung and DIR-Lab datasets and compared it with state-of-the-art methods to demonstrate its effectiveness. The experimental results have demonstrated CvTMorph to outperform the existing methods in terms of accuracy and robustness for respiratory motion modeling in 4D images. The incorporation of the convolutional vision transformer has significantly improved the registration performance and enhanced the representation of local structures. CvTMorph offers a promising solution for accurately modeling respiratory motion in 4D medical images. The hybrid model, leveraging convolutional vision transformer and convolutional neural networks, has proven effective in extracting local features and improving registration performance. The results have highlighted the potential of CvTMorph for various applications, such as radiation therapy planning, and provided a basis for further research in this field.

FPL+: Filtered Pseudo Label-Based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation.

Adapting a medical image segmentation model to a new domain is important for improving its cross-domain transferability, and due to the expensive annotation process, Unsupervised Domain Adaptation (UDA) is appealing where only unlabeled images are needed for the adaptation. Existing UDA methods are mainly based on image or feature alignment with adversarial training for regularization, and they are limited by insufficient supervision in the target domain. In this paper, we propose an enhanced Filtered Pseudo Label (FPL+)-based UDA method for 3D medical image segmentation. It first uses cross-domain data augmentation to translate labeled images in the source domain to a dual-domain training set consisting of a pseudo source-domain set and a pseudo target-domain set. To leverage the dual-domain augmented images to train a pseudo label generator, domain-specific batch normalization layers are used to deal with the domain shift while learning the domain-invariant structure features, generating high-quality pseudo labels for target-domain images. We then combine labeled source-domain images and target-domain images with pseudo labels to train a final segmentor, where image-level weighting based on uncertainty estimation and pixel-level weighting based on dual-domain consensus are proposed to mitigate the adverse effect of noisy pseudo labels. Experiments on three public multi-modal datasets for Vestibular Schwannoma, brain tumor and whole heart segmentation show that our method surpassed ten state-of-the-art UDA methods, and it even achieved better results than fully supervised learning in the target domain in some cases.

Multimodal Image Confidence: A Novel Method for Tumor and Organ Boundary Representation

BackgroundThe ambiguous boundaries of tumors and organs at risk (OARs) seen in medical images pose challenges in treatment planning and other tasks in radiotherapy. MethodsThis study introduces an innovative analytical algorithm, Multi-Modal Image Confidence (MMC), which exploits the collective strengths of complementary multi-modal medical images to determine a confidence measure for each voxel belonging to the region of interest (ROI). MMC facilitates the creation of modality-specific ROI-enhanced images, enabling a detailed representation of the ROI's boundaries and internal features. By employing an interpretable mathematical model that propagates voxel confidence based on inter-voxel correlations, MMC avoids the need for model training, distinguishing it from deep learning (DL)-based methods. ResultsThe performance of the proposed algorithm was qualitatively and quantitatively evaluated using 156 nasopharyngeal carcinoma cases and 1251 glioma cases. Qualitative assessments underscored the accuracy of MMC and ROI-enhanced images in estimating lesion boundaries and capturing internal tumor characteristics. Quantitative analyses revealed strong agreement between MMC and manual delineations. ConclusionThis paper introduces a novel analytical algorithm to identify and depict ROI boundaries based on complementary multi-modal 3D medical images. The applicability of the proposed method can extend to both targets and OARs at diverse anatomical sites across multiple image modalities, amplifying its potential for augmenting radiotherapy-related tasks.

UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation.

Owing to the success of transformer models, recent works study their applicability in 3D medical segmentation tasks. Within the transformer models, the self-attention mechanism is one of the main building blocks that strives to capture long-range dependencies, compared to the local convolutional-based design. However, the self-attention operation has quadratic complexity which proves to be a computational bottleneck, especially in volumetric medical imaging, where the inputs are 3D with numerous slices. In this paper, we propose a 3D medical image segmentation approach, named UNETR++, that offers both high-quality segmentation masks as well as efficiency in terms of parameters, compute cost, and inference speed. The core of our design is the introduction of a novel efficient paired attention (EPA) block that efficiently learns spatial and channel-wise discriminative features using a pair of inter-dependent branches based on spatial and channel attention. Our spatial attention formulation is efficient and has linear complexity with respect to the input. To enable communication between spatial and channel-focused branches, we share the weights of query and key mapping functions that provide a complimentary benefit (paired attention), while also reducing the complexity. Our extensive evaluations on five benchmarks, Synapse, BTCV, ACDC, BraTS, and Decathlon-Lung, reveal the effectiveness of our contributions in terms of both efficiency and accuracy. On Synapse, our UNETR++ sets a new state-of-the-art with a Dice Score of 87.2%, while significantly reducing parameters and FLOPs by over 71%, compared to the best method in the literature. Our code and models are available at: https://tinyurl.com/2p87x5xn.

Development of a virtual surgical plan for reverse shoulder arthroplasty as a treatment for complex proximal humerus fracture in an elderly patient

IntroductionFracture reduction, implant selection and tuberosity reattachment propose significant challenges in reverse shoulder arthroplasty (RSA) for proximal humerus fractures (PHFs). It is hypothesised that preoperative virtual surgical planning (VSP) can improve implant positioning and tuberosity reattachment, resulting in better functional outcomes, reduced surgery time, and decreased complication risk. Therefore, this study aims to develop, implement, and evaluate a VSP protocol for RSA in complex PHFs. MethodsThe VSP protocol is developed using 3D medical image processing software based on CT scans from four previously treated patients with RSA for complex PHF. Next, the VSP was applied prospectively in a patient with a PHF planned for RSA. Evaluation was done using an expert opinion questionnaire evaluating surgical confidence, a comparison between implant positioning on postoperative radiographs and the VSP, functional outcomes and the Oxford Shoulder Score (OSS). ResultsThe study showed that VSP is a useful preoperative tool to optimize clinical outcome by digital fracture reduction, determination of surgical cuts, calculating implant size and height, and optimising tuberosity shape and reattachment. After two and six months, the patient showed good functional outcomes (OSS: 48/48 points), healing of tuberosities, and had minimal pain complaints. ConclusionVSP for RSA in PHF might create a more thorough preoperative understanding of the fracture pattern and implant and fracture fragment placement. Further investigative study would be required to demonstrate this.

Vasculature segmentation in 3D hierarchical phase-contrast tomography images of human kidneys.

Efficient algorithms are needed to segment vasculature in new three-dimensional (3D) medical imaging datasets at scale for a wide range of research and clinical applications. Manual segmentation of vessels in images is time-consuming and expensive. Computational approaches are more scalable but have limitations in accuracy. We organized a global machine learning competition, engaging 1,401 participants, to help develop new deep learning methods for 3D blood vessel segmentation. This paper presents a detailed analysis of the top-performing solutions using manually curated 3D Hierarchical Phase-Contrast Tomography datasets of the human kidney, focusing on the segmentation accuracy and morphological analysis, thereby establishing a benchmark for future studies in blood vessel segmentation within phase-contrast tomography imaging.

VSmTrans: A hybrid paradigm integrating self-attention and convolution for 3D medical image segmentation

PurposeVision Transformers recently achieved a competitive performance compared with CNNs due to their excellent capability of learning global representation. However, there are two major challenges when applying them to 3D image segmentation: i) Because of the large size of 3D medical images, comprehensive global information is hard to capture due to the enormous computational costs. ii) Insufficient local inductive bias in Transformers affects the ability to segment detailed features such as ambiguous and subtly defined boundaries. Hence, to apply the Vision Transformer mechanism in the medical image segmentation field, the above challenges need to be overcome adequately. MethodsWe propose a hybrid paradigm, called Variable-Shape Mixed Transformer (VSmTrans), that integrates self-attention and convolution and can enjoy the benefits of free learning of both complex relationships from the self-attention mechanism and the local prior knowledge from convolution. Specifically, we designed a Variable-Shape self-attention mechanism, which can rapidly expand the receptive field without extra computing cost and achieve a good trade-off between global awareness and local details. In addition, the parallel convolution paradigm introduces strong local inductive bias to facilitate the ability to excavate details. Meanwhile, a pair of learnable parameters can automatically adjust the importance of the above two paradigms. Extensive experiments were conducted on two public medical image datasets with different modalities: the AMOS CT dataset and the BraTS2021 MRI dataset. ResultsOur method achieves the best average Dice scores of 88.3 % and 89.7 % on these datasets, which are superior to the previous state-of-the-art Swin Transformer-based and CNN-based architectures. A series of ablation experiments were also conducted to verify the efficiency of the proposed hybrid mechanism and the components and explore the effectiveness of those key parameters in VSmTrans. ConclusionsThe proposed hybrid Transformer-based backbone network for 3D medical image segmentation can tightly integrate self-attention and convolution to exploit the advantages of these two paradigms. The experimental results demonstrate our method's superiority compared to other state-of-the-art methods. The hybrid paradigm seems to be most appropriate to the medical image segmentation field. The ablation experiments also demonstrate that the proposed hybrid mechanism can effectively balance large receptive fields with local inductive biases, resulting in highly accurate segmentation results, especially in capturing details. Our code is available at https://github.com/qingze-bai/VSmTrans.

MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation

The Segment Anything Model (SAM), a foundation model for general image segmentation, has demonstrated impressive zero-shot performance across numerous natural image segmentation tasks. However, SAM’s performance significantly declines when applied to medical images, primarily due to the substantial disparity between natural and medical image domains. To effectively adapt SAM to medical images, it is important to incorporate critical third-dimensional information, i.e., volumetric or temporal knowledge, during fine-tuning. Simultaneously, we aim to harness SAM’s pre-trained weights within its original 2D backbone to the fullest extent. In this paper, we introduce a modality-agnostic SAM adaptation framework, named as MA-SAM, that is applicable to various volumetric and video medical data. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments while preserving the majority of SAM’s pre-trained weights. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data. We comprehensively evaluate our method on five medical image segmentation tasks, by using 11 public datasets across CT, MRI, and surgical video data. Remarkably, without using any prompt, our method consistently outperforms various state-of-the-art 3D approaches, surpassing nnU-Net by 0.9%, 2.6%, and 9.9% in Dice for CT multi-organ segmentation, MRI prostate segmentation, and surgical scene segmentation respectively. Our model also demonstrates strong generalization, and excels in challenging tumor segmentation when prompts are used. Our code is available at: https://github.com/cchen-cc/MA-SAM.