WorksComputer Science
A note on data-efficient finetuning of vision foundation models
Abstract
Adapting foundation models to a new task within or beyond the training domain is the de facto standard for building specialized machine learning models. Recently, Visual Prompting (VP) has emerged as a promising technique for efficient knowledge transfer. As new foundation model families~(like Mamba) get introduced and VP pipelines such as AutoVP reach greater maturity, we find a growing need for a systematic evaluation of current approaches. In this work, we assess the performance of the latest models, comparing them to earlier architectures and alternative fine-tuning methods, to better understand the progress, challenges and opportunities in the field of efficient fine-tuning under resource limitations. Towards this goal, this paper provides a concise empirical overview of the interactions among foundation model families (Attention-, Convolution-, and Mamba-based) and transfer paradigms: VP, Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA). Our work builds up on previous findings by broadening the selection of evaluated models and techniques while tuning hyperparameters. In the interest of delivering practical guidelines for the user, we also explore application of prevalent regularization techniques to boost performance in the context of VP.
For a review of these methods, please refer to § 2. Given this rapid expansion, it is critical to systematically and regularly assess how different architectural choices interact with various transfer learning methods. In this work, we present an empirical evaluation of VP across several prominent model architectures, analyze how performance correlates with pretraining accuracy, and discuss potential performance and compute tradeoffs when comparing it to other fine-tuning methods on challenging tasks.
Compared to earlier results ([24] for ILM-VP [4]), we found that vision transformers and Mamba-based ((Gu & Dao, 2023)) models perform similarly – with only a slight edge for the former – under latest VP practice ([31]). However, when selecting a model for VP, ConvNext appears to be the best choice from our selection (Figure 1). References and a short description for these models are presented in § 3.
Figure 1. ImageNet1k pretrain and average VP top-1 accuracy over the tasks described in § 3. We follow the recipe of Tsao et al. [31]. All models are tuned to best. Dotted line is linear regression, full results in Table 1.
On DTD, LoRA achieves higher reported accuracy than FFT for five of the six evaluated models while training fewer parameters. State-of-the-art VP practice can underperform (overfit) on data scarce tasks compared to LP, FFT and LoRA – see Table 2. In settings with restricted computational resources, LP seems to be dominant w.r.t. VP.
| VP | LP | FFT | LoRA | |
| MAMBAVISION | 65.64 | 70.21 | 75.16 | 73.88 |
| CONVNEXT | 70.77 | 70.90 | 73.18 | 76.60 |
| VMAMBA | 67.87 | 67.45 | 73.30 | 75.00 |
| DEIT | 66.54 | 68.56 | 72.98 | 74.68 |
| VIT | 62.13 | 71.38 | 75.32 | 77.02 |
| SWIN | 68.47 | 77.07 | 79.63 | 81.33 |
Table 2. Table 2: Top 1 test-performance in % of various base-models on DTD using different Transfer Learning paradigms. Learning rate, rank (LoRA) and weight decay tuned. Base-size used for all models.
Introduction
Efficient transfer of pretraining knowledge to new tasks and domains is an essential requirement for successful downstream applicability of foundation models. As the AI landscape continues to grow – spanning diverse architectures and design choices – so do options for fine-tuning, including new pipelines for Visual Prompting (VP), Linear Probing (LP), Full Fine-tuning (FFT) and Low Rank Adaptation (LoRA).
| CIFAR10 | CIFAR100 | FLOWERS102 | GTSRB | DTD | EUROsat | FOOD101 | OXFORDPETS | PRETRAIN ACC. | ||
| VMAMBA-T | 94.89 | 80.53 | 91.67 | 86.75 | 67.17 | 95.72 | 69.25 | 88.07 | 84.25 | 82.60 |
| MAMBAOUT-T | 95.07 | 81.36 | 89.58 | 88.27 | 67.73 | 95.70 | 71.26 | 89.14 | 84.76 | 82.70 |
| MAMBAVISION-T | 95.44 | 80.89 | 92.55 | 87.74 | 66.26 | 96.56 | 71.56 | 90.91 | 85.24 | 82.30 |
| SWIN-T | 94.28 | 80.32 | 92.34 | 88.95 | 66.37 | 96.31 | 72.99 | 88.83 | 85.05 | 81.50 |
| MAMBAVISION-S | 95.97 | 82.37 | 90.71 | 86.93 | 66.58 | 96.62 | 74.18 | 91.57 | 85.61 | 83.30 |
| MAMBAVISION-B | 96.54 | 83.88 | 91.79 | 88.79 | 65.64 | 96.63 | 75.65 | 91.22 | 86.27 | 84.20 |
| CONVNEXT-B | 96.42 | 84.44 | 94.15 | 90.28 | 70.77 | 96.61 | 75.10 | 92.04 | 87.48 | 84.10 |
| VMAMBA-B | 96.58 | 76.90 | 93.67 | 90.56 | 67.87 | 94.14 | 75.52 | 91.35 | 85.82 | 83.90 |
| DEIT-B | 96.52 | 83.59 | 93.51 | 91.57 | 66.54 | 96.85 | 72.13 | 90.01 | 86.34 | 81.80 |
| VIT-B | 95.52 | 78.18 | 90.74 | 89.54 | 62.13 | 96.80 | 70.84 | 89.97 | 84.21 | 81.10 |
| SWIN-B | 95.45 | 81.55 | 93.35 | 86.19 | 68.47 | 96.59 | 74.26 | 89.95 | 85.73 | 84.50 |
| CORR. IMAGENET | 0.49 | 0.31 | 0.35 | -0.23 | 0.66 | -0.23 | 0.73 | 0.53 | 0.60 |
caption: tbl:1: Top-1 accuracy after training with AutoVP using fully connected label mapping. denotes the mean fine-tuning performance. The last column represents the ImageNet pretraining accuracy. The bottom row reports the correlation coefficient between the corresponding column and the ImageNet pretraining accuracy.
It is natural to ask at this point if VP performance can be improved via regularization. We found that prevalent regularization techniques can boost generalization, whilst still underperforming LoRA. We noticed that most gains can be observed when applying regularization to the label mapping weights, as opposed to the prompt.
ambiguous to interpret. In this work, we evaluate a selection of ImageNet pretrained foundation models (§ 3) using AutoVP (Tsao et al. [31]) on a common set of downstream image-classification tasks and in addition to performance (Tab. 1 and Figure 1) also investigate on the behavior of models w.r.t. the resizing of the prompt (Table 6 and Figure 2).
Figure 2. Mean prompt size of the best performing run, with the initial prompt scaling being 1. Mean and standard deviation calculated over the tasks introduced in § 3.
| CIFAR10 | CIFAR100 | FLOWERS102 | GTSRB | DTD | EUROSAT | FOOD101 | OXFORDPETS | |||
| VMAMBA-T | 0.990 | 0.999 | 0.996 | 1.047 | 0.999 | 1.083 | 1.000 | 0.999 | 1.014 | 0.031 |
| MAMBAOUT-T | 1.000 | 0.999 | 0.999 | 3.674 | 0.997 | 0.691 | 1.254 | 2.275 | 1.486 | 0.938 |
| MAMBAVISION-T | 0.995 | 0.998 | 1.401 | 3.111 | 2.195 | 0.927 | 1.182 | 2.342 | 1.644 | 0.756 |
| SWINT-T | 0.986 | 1.001 | 0.999 | 0.995 | 0.999 | 0.950 | 1.000 | 1.196 | 1.016 | 0.070 |
| MAMBAVISION-S | 0.927 | 1.006 | 2.930 | 1.105 | 0.997 | 1.006 | 1.238 | 1.942 | 1.394 | 0.656 |
| MAMBAVISION-B | 1.003 | 1.006 | 1.002 | 3.912 | 2.362 | 0.994 | 1.386 | 1.574 | 1.655 | 0.960 |
| CONVNEXT-B | 0.994 | 1.001 | 0.999 | 0.991 | 0.998 | 0.985 | 1.000 | 1.001 | 0.996 | 0.005 |
| VMAMBA-B | 1.004 | 1.021 | 0.997 | 0.863 | 0.999 | 0.927 | 1.000 | 0.998 | 0.976 | 0.050 |
| DEIT-B | 1.011 | 1.000 | 0.998 | 0.792 | 1.000 | 0.569 | 1.000 | 1.562 | 0.991 | 0.261 |
| VIT-B | 1.004 | 0.856 | 1.090 | 0.998 | 1.349 | 1.000 | 2.332 | 2.190 | 1.352 | 0.542 |
| SWIN-B | 1.003 | 0.997 | 1.001 | 4.798 | 0.999 | 1.003 | 1.214 | 1.542 | 1.569 | 1.233 |
Table 6. Prompt scaling of the best run for the respective model-task combination
Background
Benchmarking Visual Prompting. VP, first introduced as Adversarial Reprogramming (Elsayed et al., [9]), leverages the idea of utilizing a universal input perturbation to transfer a pretrained model to a new domain or task. The term Prompting originates from language modeling (Lester et al., [17]) and was later adapted to the vision domain (Bahng et al., [2]). VP keeps the pretrained source model entirely frozen and therefore differs from other techniques (see next paragraph) that may require adapting source model weights. Black-box variants of reprogramming can also work with API-only access (Tsai et al., [30]).
With AutoVP, Tsao et al. [31] introduced improvements of the VP methodology as well as a unified benchmark for VP. The introduced improvements include the possibility of resizing the prompt as well as a new Label Mapping (LM) technique: Fully Connected Label Mapping (FCLM). FCLM is realized by a fully connected layer of dimensions (of source classes ; # of target classes ).
While there exists research on the combination of new foundation models and VP ([4, 31, 24]), these works are either limited in model selection or didn’t tune hyperparameters, making the results
Full fine-tuning and linear probing. Based on the improvements of AutoVP, we conduct a comparison of FFT, LP, LoRA (described next) and VP on the data scarce DTD dataset (Cimpoi et al., [5]), which showed to be the most challenging task in the benchmark. We investigate a subset of the models evaluated in Tab. 1, while tuning learning rate and weight decay to ensure optimal performance for each method. LP freezes the pretrained model, whilst only replacing and training a new classifier with the idea of leveraging the existing feature representations at the penultimate layer of a model (Alain & Bengio, [1]). In contrast to full fine-tuning, LP drastically reduces the amount of trainable parameters as well as hardware requirements. Last, in our implementation of FFT, we account for the new number of output classes by training a new classifier from scratch ([4]). In contrast to LP or VP, no model parameters are frozen. While this unlocks stronger results, it also requires larger compute and memory (Table 3) compared to the other approaches.
| # PARAM. | GPU-MEM (GB) | EPOCHTIME (S) | |
| VP | 197,576 | 12.5 | 39.8 |
| LP | 48,175 | 11.2 | 36.1 |
| FFT | 87,601,967 | 72.3 | 49.2 |
| LoRA | 740,911 | 67.5 | 45.0 |
Table 3. VP refers to AutoVP with fully connected label mapping and adjustable prompt size. GPU-Mem describes the peak allocated GPU Memory. Reported Runtime is the median over all runs. All reported numbers refer to the use of VMamba-B on the DTD dataset.
LoRA. Low-Rank Adaptation (LoRA) ((Hu et al., 2022)) is one of the most prominent lines of PEFT research. Instead of updating the weight matrix directly, LoRA parametrizes the update as a product , of low-rank , and hyperparameter . Only and are trained, which drastically reduces the number of trainable parameters. Since the LoRA modules contain only these low-rank updates and not the base model’s weights, they are lightweight and can be swapped depending on the task, while the base model remains frozen and unchanged.
Regularizing Visual Prompting. We noticed models showed signs of overfitting when being transferred to DTD using VP – as testified by Table 2, where all models reached training accuracy despite suboptimal test performance compared to LP, FFT or LoRA. This poses the question on how to effectively regularize VP. This task is particularly interesting, as common regularization methods are motivated by a different learning paradigm. We investigated a set of common regularization approaches while also ablating on targeting regularizing only to a subset of the optimized parameters in VP: either prompt or Label Mapping parameters. Specifically, we evaluated performance of Dropout ([14]), L1-/L2-regularization, Sharpness-aware minimization (SAM) ([10]) and Autoaugment ([6]).
Experimental setup
The tasks are selected in accordance with earlier evaluations on VP ([4, 31]), consisting of CIFAR10/100 ([16]), Flowers102 ([25]), GTSRB ([27]), DTD ([5]), EuroSAT ([13]), Food101 ([3]), OxfordPets ([26]).
The models we use are VMamba1 ([19]), MambaOut2 ((Yu & Wang, 2025)), MambaVision3 ([12]), SwinT4 ([20]), ConvNext4 ([21]), ViT4 ([8]) and DeiT5 ([29]). In all cases, the ImageNet ([7]) pretrained weights are utilized.
Our initial set of experiments (Table 1 and § 4.1) involves training the previously introduced models – belonging to drastically different architecture families – on the AutoVP Benchmark ([31]). For detailed information regarding hyperparameter tuning, please refer to Appendix § 7.
Our second series of experiments (§ 4.2) entails a comparative evaluation of LP, VP, FFT and LoRA on a subset of the introduced models. These fine-tuning paradigms are compared on the DTD dataset, as it appears to be the most challenging task with limited training data.
Our third set of experiments (§ 4.3) focuses on the regularization of VP. To assess regularization strategies on VP, we evaluate VMamba-B on the DTD Dataset (with Vanilla-VP performance obtainable from Table 1), utilizing a fixed learning rate of 0.001. Dropout and L1/L2 regularization are implemented both across the entire model or specifically targeted to either the mapping or the prompt parameters.
All runs from the first and second set of experiments are using a single Nvidia A100 GPU. The remaining experiments are utilizing either Nvidia Tesla V100 or Nvidia Quadro RTX 6000 GPUs.
The evaluation of different fine-tuning techniques must consider the distinct computational requirements for training models of significant size. The precise calculation of trainable model parameters for the different approaches is contingent on numerous factors, including the selected model, input dimensions, and the number of target classes. For this comparison we assume VMamba-B being applied to the DTD dataset with 47 target classes. For VP, inputs are resized using a learnable prompt size, while for LP, FFT and LoRA they are resized to the input size 224 × 224. Furthermore, while the trainable parameters of all approaches depend on the number of target classes, VP is also reliant on the input dimensions and the number of source classes in the pretrained model while LP also depends on the feature dimension of the penultimate layer. For LoRA, a rank of 2 was used to compute the reported number of parameters, as the different ranks all yielded similar results and therefore rank 2 showed to be the most lightweight variant reaching good performance. It is noteworthy that the implementation for LoRA in combination with VMamba was probably less optimized compared to the implementation for transformer based models. Consequently, the reported numbers need to be interpreted within the context of the specific choice of pretrained model and task.
Results
VP levels model performances
Tab. 1 presents the model performances alongside the correlation between model performance and pretraining ImageNet accuracy. In general, raw ImageNet pretraining accuracy presents itself to be a determining factor for model performance, as validated by the reported correlation coefficient. This indicates an external validity of ImageNet-performance [28]. Nonetheless, certain tasks exhibit weak or even negative correlation.
ConvNext achieved the highest accuracy on four of the eight tasks. Yet, some tasks are more sensitive to model choice than others: while on CIFAR10 and EuroSAT performance differences are marginal, on DTD, the right selection of pretrained model is crucial. Our last investigation concerns the comparison with VP methods prior to Tsao et al. [31], and in particular with ILM-VP [4], tested also by Misra et al. [24]. In contrast to ILM-VP, the variance in
performance between different model architectures is significantly reduced. Using AutoVP on the Flowers102 dataset increases the performance of VMamba by 54.77 percentage points in comparison to ILM-VP, where it only reaches 38.9%. In comparison, ViT-B already achieves 83.1% under ILM-VP and is outperformed by VMamba under AutoVP.
For VMamba-T on Flowers102, our ablation separates the effects of label mapping and prompt resizing and identifies fully connected label mapping as the main source of the improvement over ILM-VP (Figure 3). As depicted in Figure 2, the capability to resize the prompt is not exploited uniformly across models. ConvNext, which on average performs best, retains the initial configuration, while Swin-B exhibits considerable variation in prompt size. Models differing only by size (e.g., MambaVision, VMamba) tend to have similar behavior, while Swin Transformer behaves as an outlier with Swin-T maintaining the initial prompt size and Swin-B showing variation. Detailed results can be found in Appendix § 7.
Figure 3. Flowers102 performance of VMamba-T for combinations of ILM and FCLM with and without enabled resizing.
Comparison training paradigms
As showcased in Table 3, for the specific setting described above, LP proves to be the most lightweight fine-tuning method, with VP being only slightly more hardware demanding. By requiring only about 15% of the GPU-memory required for FFT, the field of application is drastically broadened by, in contrast to FFT, being able to e.g. run on a Nvidia Tesla V100 GPU. LoRA manages to outperform FFT on 5 of the 6 evaluated models, surprisingly not showing substantial differences between different rank configurations (See tab. 4).
| RANK | 1 | 2 | 4 | 8 | 16 | |
| M-VISION | 73.09 | 73.78 | 72.77 | 73.88 | 73.94 | |
| CONVNEXT | 76.22 | 76.39 | 76.49 | 76.54 | 76.12 | |
| VMAMBA | 73.78 | 75.00 | 74.41 | 74.57 | 74.15 | |
| DEIT | 73.09 | 73.94 | 74.57 | 73.46 | 73.67 | |
| VIT | 76.70 | 76.44 | 76.60 | 75.85 | 77.02 | |
| SWIN | 80.00 | 81.33 | 81.17 | 80.90 | 80.00 |
Table 4. Top 1 test-performance in % for different rank configurations.
It is also noteworthy, that whilst the majority of research conducted on LoRA is in combination with Transformer-based architectures, even a naive implementation on Mamba- and Convolution-based architectures shows to be an efficient way of fine-tuning, making it thereby easily applicable for a variety of tasks. Whilst LoRA outperforming FFT is generally not a novel observation on language modeling [18], the transfer to image classification offers great potential for downstream applicability to further domains.
Regularization of VP
Applying L1 regularization to either the prompt generation and the Label Mapping leads to almost similar enhancements. Conversely, for L2 regularization, regularizing the Label Mapping is substantially more effective than regularizing the prompt. The augmentation schemes from Cubuk et al. [6], which are derived from ImageNet, SVHN, and CIFAR10, increased performance, however to the smallest degree. VMamba also benefits from the utilization of SAM as well as from applying Dropout to the Label Mapping.
| Reference | 67.87 |
| Dropout | 68.73 |
| L1 Prompt | 69.42 |
| L1 Mapping | 69.37 |
| L2 Prompt | 68.84 |
| L2 Mapping | 70.31 |
| AutoAugment | 68.65 |
| SAM | 69.31 |
Table 5. AutoVP performances of VMamba-B using different regularization mechanics. First row reports the unregularized VP performance.
For VMamba-B on DTD, the tested regularizers improve VP accuracy but leave a gap to LoRA. Also, no single regularization method can completely close the performance gap between VMamba and ConvNext.
Conclusion
The VMamba-T ablation on Flowers102 shows that the choice of label mapping can strongly affect visual prompting performance, highlighting the importance of the adaptation procedure when assessing a pretrained model. When employing VP, our results show that the selection should consider both its pretraining accuracy and the model architecture, with ConvNext being the best option in our setup. In particular on data scarce tasks, LoRA seems to be the method of choice, outperforming FFT while being lightweight.
Acknowledgments
The authors acknowledge the financial support of the Hector Foundation.
Benchmarking Visual Prompting
The Experiments from 4.1 followed the AutoVP recipe (Tsao et al., [31]) with:
Learning rates of [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3]
2 different seeds and no regularization applied.
Initial image-scale of 1.0 with prompt scaling enabled
50 epochs
The reported accuracy is the maximum test accuracy over the run. Runs comparing the different Mapping approaches were conducted with learning rates [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3] with one seed.
Table 6 provides detailed results regarding the Prompt resizing for different Model-task combinations. The reported number is the prompt size after the last training epoch w.r.t. the initial instantiation.
An interesting observation is that while some tasks generally lead to a stronger resizing (e.g. GTSRB results in an average prompt size of 2.026 times the initial calibration), 5 out of the 11 models observed also reduce the prompt size.
Comparing Knowledge Transfer Methods
LP and FFT were implemented by adapting https://github.com/kuangliu/pytorch-cifar. The results for VP were obtained using the setup described in A.
The models were all evaluated on FFT and LP using the combinations of the learning rates [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3] and weight decays [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3]. All models were trained for 200 epochs on one seed.
LoRA was implemented using the peft (Mangrulkar et al., [23]) library.
RandomRotation((10)), Colorjitter(brightness=0.05,contrast=0.05,saturation=0.05) and RandomAffine(degrees=5,translate=(0.1,0.1),scale=(0.9,1.1)) were used as input perturbations.
For LP and FFT, the linear layer of the classification head was replaced to account for the correct number of output classes. For LP, the rest of the model was frozen and only the new classifier was trained.
Regularizing Visual Prompting
The Experiments from 4.3 used the best performing learning rate of 0.001.
Dropout was applied with p-values of [0.05, 0.1, 0.2, 0.4] on the Label Mapping.
L1 and L2 regularization were both run with -values of [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3] on either the prompt or the Label Mapping.
Autoaugment was used by applying the precomputed schemes for ImageNet, SVHN and CIFAR10.
For SAM, of [0.001, 0.003, 0.005, 0.008 and 0.01] were used.
References
- [1]Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes, 2018.
- [2]Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022.arxiv.org/abs/2203.17274
- [3]Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision, 2014.DOI
- [4]Chen, A., Yao, Y., Chen, P.-Y., Zhang, Y., and Liu, S. Understanding and improving visual prompting: A label-mapping perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19133–19143, 2023.arxiv.org/abs/2211.11635
- [5]Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606–3613, 2014. doi: 10.1109/CVPR.2014.461.DOI
- [6]Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 113–123, 2019.DOI
- [7]Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
- [8]Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.arxiv.org/abs/2010.11929
- [9]Elsayed, G. F., Goodfellow, I. J., and Sohl-Dickstein, J. Adversarial reprogramming of neural networks. CoRR, abs/1806.11146, 2018.arxiv.org/abs/1806.11146
- [10]Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.arxiv.org/abs/2010.01412
- [12]Hatamizadeh, A. and Kautz, J. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083, 2024.arxiv.org/abs/2407.08083
- [13]Helber, P., Bischke, B., Dengel, A., and Borth, D. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. doi: 10.1109/JSTARS.2019.2918242.DOI
- [14]Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.arxiv.org/abs/1207.0580
- [16]Krizhevsky, A. et al. Learning multiple layers of features from tiny images. 2009.
- [17]Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.arxiv.org/abs/2104.08691
- [18]Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024a.arxiv.org/abs/2402.09353
- [19]Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024b.arxiv.org/abs/2401.10166
- [20]Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.arxiv.org/abs/2103.14030
- [21]Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11966–11976, 2022. doi: 10.1109/CVPR52688.2022.01167.DOI
- [23]Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022.
- [24]Misra, D., Gala, J., and Orvieto, A. On the low-shot transfer-ability of [v]-mamba. arXiv preprint arXiv:2403.10696, 2024.arxiv.org/abs/2403.10696
- [25]Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722–729, 2008.DOI
- [26]Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498–3505, 2012.
- [27]Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks, pp. 1453–1460. IEEE, 2011.DOI
- [28]Torralba, A. and Efros, A. A. Unbiased look at dataset bias. In CVPR 2011, pp. 1521–1528. IEEE, 2011.DOI
- [29]Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347–10357. PMLR, 2021.
- [30]Tsai, Y.-Y., Chen, P.-Y., and Ho, T.-Y. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International Conference on Machine Learning, pp. 9614–9624. PMLR, 2020.arxiv.org/abs/2007.08714
- [31]Tsao, H.-A., Hsiung, L., Chen, P.-Y., Liu, S., and Ho, T.-Y. AutoVP: An Automated Visual Prompting Framework and Benchmark. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.08381
- [32]Wightman, R. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019.


