2025
4.
Wei, Xiwen; Li, Guihong; Marculescu, Radu
Online-LoRA: Task-Free Online Continual Learning via Low Rank Adaptation Conference
2025 Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2025), 2025.
Abstract | Links | BibTeX | Tags: Vision Transformer
@conference{ONLINE_LORA_WACV_2025,
title = {Online-LoRA: Task-Free Online Continual Learning via Low Rank Adaptation},
author = {Xiwen Wei and Guihong Li and Radu Marculescu},
url = {https://ieeexplore.ieee.org/document/10943985},
year = {2025},
date = {2025-03-06},
urldate = {2025-01-06},
booktitle = {2025 Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2025)},
abstract = {Catastrophic forgetting is a significant challenge in online continual learning (OCL), especially for non-stationary data streams that do not have well-defined task boundaries. This challenge is exacerbated by the memory constraints and privacy concerns inherent in rehearsal buffers. To tackle catastrophic forgetting, in this paper, we introduce Online-LoRA, a novel framework for task-free OCL. Online-LoRA allows to finetune pre-trained Vision Transformer (ViT) models in real-time to address the limitations of rehearsal buffers and leverage pre-trained models' performance benefits. As the main contribution, our approach features a novel online weight regularization strategy to identify and consolidate important model parameters. Moreover, Online-LoRA leverages the training dynamics of loss values to enable the automatic recognition of the data distribution shifts. Extensive experiments across many task-free OCL scenarios and benchmark datasets (including CIFAR-100, ImageNet-R, ImageNet-S, CUB-200 and CORe50) demonstrate that Online-LoRA can be robustly adapted to various ViT architectures, while achieving better performance compared to SOTA methods.},
keywords = {Vision Transformer},
pubstate = {published},
tppubtype = {conference}
}
Catastrophic forgetting is a significant challenge in online continual learning (OCL), especially for non-stationary data streams that do not have well-defined task boundaries. This challenge is exacerbated by the memory constraints and privacy concerns inherent in rehearsal buffers. To tackle catastrophic forgetting, in this paper, we introduce Online-LoRA, a novel framework for task-free OCL. Online-LoRA allows to finetune pre-trained Vision Transformer (ViT) models in real-time to address the limitations of rehearsal buffers and leverage pre-trained models' performance benefits. As the main contribution, our approach features a novel online weight regularization strategy to identify and consolidate important model parameters. Moreover, Online-LoRA leverages the training dynamics of loss values to enable the automatic recognition of the data distribution shifts. Extensive experiments across many task-free OCL scenarios and benchmark datasets (including CIFAR-100, ImageNet-R, ImageNet-S, CUB-200 and CORe50) demonstrate that Online-LoRA can be robustly adapted to various ViT architectures, while achieving better performance compared to SOTA methods.
2024
3.
Rahman, Md Mostafijur; Munir, Mustafa; Marculescu, Radu
EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation Conference
Proceedings of the IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2024.
Abstract | Links | BibTeX | Tags: Deep Learning Architecture, Efficient Decoder, Medical Image Segmentation, Multi-scale Depth-wise Convolutions, Vision Transformer
@conference{EMCAD_CVPR_2024,
title = {EMCAD: Efficient Multi-scale Convolutional Attention Decoding for Medical Image Segmentation},
author = {Md Mostafijur Rahman and Mustafa Munir and Radu Marculescu },
url = {https://openaccess.thecvf.com/content/CVPR2024/papers/Rahman_EMCAD_Efficient_Multi-scale_Convolutional_Attention_Decoding_for_Medical_Image_Segmentation_CVPR_2024_paper.pdf},
year = {2024},
date = {2024-06-17},
urldate = {2024-06-17},
publisher = {Proceedings of the IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR)},
abstract = {An efficient and effective decoding mechanism is crucial in medical image segmentation, especially in scenarios with limited computational resources. However, these decoding mechanisms usually come with high computational costs. To address this concern, we introduce EMCAD, a new efficient multi-scale convolutional attention decoder, designed to optimize both performance and computational efficiency. EMCAD leverages a unique multi-scale depth-wise convolution block, significantly enhancing feature maps through multi-scale convolutions. EMCAD also employs channel, spatial, and grouped (large-kernel) gated attention mechanisms, which are highly effective at capturing intricate spatial relationships while focusing on salient regions. By employing group and depth-wise convolution, EMCAD is very efficient and scales well (e.g., only 1.91M parameters and 0.381G FLOPs are needed when using a standard encoder). Our rigorous evaluations across 12 datasets that belong to six medical image segmentation tasks reveal that EMCAD achieves state-of-the-art (SOTA) performance with 79.4% and 80.3% reduction in #Params and #FLOPs, respectively. Moreover, EMCAD’s adaptability to different encoders and versatility across segmentation tasks further establish EMCAD as a promising tool, advancing the field towards more efficient and accurate medical image analysis. Our implementation is available at https://github.com/SLDGroup/EMCAD.},
keywords = {Deep Learning Architecture, Efficient Decoder, Medical Image Segmentation, Multi-scale Depth-wise Convolutions, Vision Transformer},
pubstate = {published},
tppubtype = {conference}
}
An efficient and effective decoding mechanism is crucial in medical image segmentation, especially in scenarios with limited computational resources. However, these decoding mechanisms usually come with high computational costs. To address this concern, we introduce EMCAD, a new efficient multi-scale convolutional attention decoder, designed to optimize both performance and computational efficiency. EMCAD leverages a unique multi-scale depth-wise convolution block, significantly enhancing feature maps through multi-scale convolutions. EMCAD also employs channel, spatial, and grouped (large-kernel) gated attention mechanisms, which are highly effective at capturing intricate spatial relationships while focusing on salient regions. By employing group and depth-wise convolution, EMCAD is very efficient and scales well (e.g., only 1.91M parameters and 0.381G FLOPs are needed when using a standard encoder). Our rigorous evaluations across 12 datasets that belong to six medical image segmentation tasks reveal that EMCAD achieves state-of-the-art (SOTA) performance with 79.4% and 80.3% reduction in #Params and #FLOPs, respectively. Moreover, EMCAD’s adaptability to different encoders and versatility across segmentation tasks further establish EMCAD as a promising tool, advancing the field towards more efficient and accurate medical image analysis. Our implementation is available at https://github.com/SLDGroup/EMCAD.
2023
2.
Yang, Yuedong; Chiang, Hung-Yueh; Li, Guihong; Marculescu, Diana; Marculescu, Radu
Efficient Low-rank Backpropagation for Vision Transformer Adaptation Conference
Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) 2023, 2023.
Abstract | Links | BibTeX | Tags: On-device Training, Vision Transformer
@conference{nokey,
title = {Efficient Low-rank Backpropagation for Vision Transformer Adaptation},
author = {Yuedong Yang and Hung-Yueh Chiang and Guihong Li and Diana Marculescu and Radu Marculescu},
url = {https://radum.ece.utexas.edu/wp-content/uploads/2023/09/LBP_WHT.pdf},
year = {2023},
date = {2023-12-11},
urldate = {2023-12-11},
publisher = {Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) 2023},
abstract = {The increasing scale of vision transformers (ViT) has made the efficient fine-tuning of these large models for specific needs a significant challenge in various applications. This issue originates from the computationally demanding matrix multiplications required during the backpropagation process through linear layers in ViT. In this paper, we tackle this problem by proposing a new Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method. Intuitively, LBP-WHT projects the gradient into a low-rank space and carries out backpropagation. This approach substantially reduces the computation needed for adapting ViT, as matrix multiplication in the low-rank space is far less resource-intensive. We conduct extensive experiments with different models (ViT, hybrid convolution-ViT model) on multiple datasets to demonstrate the effectiveness of our method. For instance, when adapting an EfficientFormer-L1 model on CIFAR100, our LBP-WHT achieves 10.4% higher accuracy than the state-of-the-art baseline, while requiring 9 MFLOPs less computation. As the first work to accelerate ViT adaptation with low-rank backpropagation, our LBP-WHT method is complementary to many prior efforts and can be combined with them for better performance.},
keywords = {On-device Training, Vision Transformer},
pubstate = {published},
tppubtype = {conference}
}
The increasing scale of vision transformers (ViT) has made the efficient fine-tuning of these large models for specific needs a significant challenge in various applications. This issue originates from the computationally demanding matrix multiplications required during the backpropagation process through linear layers in ViT. In this paper, we tackle this problem by proposing a new Low-rank BackPropagation via Walsh-Hadamard Transformation (LBP-WHT) method. Intuitively, LBP-WHT projects the gradient into a low-rank space and carries out backpropagation. This approach substantially reduces the computation needed for adapting ViT, as matrix multiplication in the low-rank space is far less resource-intensive. We conduct extensive experiments with different models (ViT, hybrid convolution-ViT model) on multiple datasets to demonstrate the effectiveness of our method. For instance, when adapting an EfficientFormer-L1 model on CIFAR100, our LBP-WHT achieves 10.4% higher accuracy than the state-of-the-art baseline, while requiring 9 MFLOPs less computation. As the first work to accelerate ViT adaptation with low-rank backpropagation, our LBP-WHT method is complementary to many prior efforts and can be combined with them for better performance.
1.
Rahman, Md Mostafijur; Marculescu, Radu
Medical Imaging with Deep Learning, 2023.
Links | BibTeX | Tags: Deep Learning Architecture, Medical Image Segmentation, Vision Transformer
@conference{MIDL2023,
title = {Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation},
author = {Rahman, Md Mostafijur and Marculescu, Radu},
url = {https://arxiv.org/abs/2303.16892},
year = {2023},
date = {2023-07-10},
urldate = {2023-07-10},
booktitle = {Medical Imaging with Deep Learning},
keywords = {Deep Learning Architecture, Medical Image Segmentation, Vision Transformer},
pubstate = {published},
tppubtype = {conference}
}
