betaFeedback

WorksComputer Science

Learn, Freeze, and Transfer an Activation Function Across Architectures

Abstract

Transfer learning typically reuses knowledge acquired by a source network through its learned weights. We investigate whether a learned activation function can also be transferred and remain useful in networks trained from scratch. We study this complementary form of transfer through a learn-freeze-transfer protocol, in which the activation is optimized jointly with a source network in a single training run, and is then frozen and reused in target networks trained from scratch. A pointwise activation with globally shared parameters is independent of layer dimensions and can therefore be incorporated into networks of different architectures trained from scratch. We instantiate this protocol with FAct, a five-parameter activation represented by a two-harmonic truncated Fourier series, initialized from a Fourier approximation of GELU and learned in a Vision Transformer (ViT) on ImageNet-1K. When reused in a ViT of the same architecture trained from scratch on ImageNet-1K, the frozen activation (Frozen FAct) improves top-1 accuracy by 1.46 percentage points over GELU. Across four smaller target architectures (standard ViT, Swin Transformer, linear-attention ViT, and MLP-Mixer) and four image-classification datasets, it achieves the highest mean accuracy among eleven activation functions in 15 of 16 settings under a common training configuration, with improvements of up to 1.71 percentage points over the best conventional activation. When the learning rate and weight decay are tuned separately for each activation under the same search budget, Frozen FAct still ranks first in 11 of the 16 settings, second in four, and fourth in one. These results provide empirical evidence that a learned activation function can retain its usefulness when transferred across vision architectures, independently of the weights with which it was learned.

Introduction

Transfer learning typically reuses knowledge acquired by a source network through its learned weights. We investigate whether an activation function learned jointly with a source network in a single training run can also be transferred to target networks and remain useful. To this end, we use a parameterized activation function whose coefficients are shared globally across all activation sites of the network. Global sharing makes the number of activation parameters independent of network depth and width, so the learned function can be reused in networks of different architectures. This yields a complementary form of transfer, in which the learned activation, rather than the source-network weights, is retained and reused.

Learnable activation functions allow the nonlinear transformation to adapt during training. Existing parameterizations include adaptive slopes [10, 14], customized activation functions [23], rational

Code

https://github.com/AI4Bharat/LeFT-activation-transfer

Paper with interactive figures

https://arunprakash-a.github.io/papers/learn-freeze-transfer/ functions [26] and kernel-based representations [30]. Their parameters may be assigned locally to individual neurons or layers [1, 10, 15, 23, 26] or shared across multiple layers [14, 5]. These approaches study how activation functions adapt within a network. In contrast, we examine whether a learned activation can be retained and reused independently of the network in which it was optimized.

Prior work has reused activation functions beyond the setting in which they were obtained, but mainly through automated search [29, 2, 32] or meta-learning [34]. To our knowledge, transferring an activation learned in a single training run across architectures has not been studied. In our setting, the activation is learned through end-to-end training with a single source network, without meta-learning or a separate search. After source training, the activation is frozen and transferred to target networks trained from scratch. Only its learned coefficients are transferred; the source-network weights are not reused, and the activation remains fixed throughout target training (Figure 1). This protocol allows us to examine whether a nonlinearity learned within one network remains useful across different architectures and tasks.

Figure 1. Overview of the learn–freeze–transfer framework. (a) A five-parameter FAct activation, shared across all MLP blocks, is learned jointly with a depth-6 ViT on ImageNet-1K. (b) The learned activation coefficients are frozen after source training. (c) Only these coefficients, not the source-network weights, are transferred to independently initialized target networks. Evaluation includes same-task transfer to a freshly initialized copy of the source architecture (ViT-d6) and cross-architecture transfer to depth-2 targets (ViT-d2, Swin-d2, LinViT-d2, and MLP-Mixer), with the activation coefficients fixed throughout target training. Transfer performance for these settings is reported in Table 1, Table 2, Table 3, Table 4, and Table 6.

ActivationTop-1 accuracy (%)Δ\Delta vs GELU (pp)
GELU62.45 ±\pm 0.13—
Frozen FAct63.91 ±\pm 0.15+1.46

Table 1. Same-task transfer on ImageNet-1K using a freshly initialized ViT-d6. Results report top-1 accuracy (%) on the ILSVRC validation set as mean ±\pm sample standard deviation across five target runs. Frozen FAct uses the activation coefficients learned by the source network; source-network weights are not transferred, and activation coefficients remain fixed. Δ\Delta denotes the difference in mean accuracy relative to GELU, in percentage points.

Table 4: Transfer performance on LinViT-d2 with linear attention under the common training configuration. Results are mean test accuracy (%) ± sample standard deviation across five seeds. Bold and underlining indicate the highest and second-highest mean accuracies, respectively.

ActivationFashion-MNISTCIFAR-10CIFAR-100Food-101
Frozen Fact90.67 ± 0.2475.15 ± 0.3848.05 ± 0.1936.12 ± 0.34
ReLU90.29 ± 0.1673.94 ± 0.4846.57 ± 0.3634.40 ± 0.27
LeakyReLU90.18 ± 0.2373.89 ± 0.3046.35 ± 0.2934.50 ± 0.29
GELU89.07 ± 0.2272.13 ± 0.3145.47 ± 0.3233.88 ± 0.63
SiLU87.89 ± 0.5270.05 ± 0.3143.29 ± 0.4632.49 ± 0.13
Mish88.06 ± 0.2570.23 ± 0.2843.66 ± 0.6332.70 ± 0.17
HardSwish87.15 ± 0.4068.35 ± 0.4741.16 ± 0.6231.35 ± 0.31
Softplus85.48 ± 0.2664.30 ± 0.9838.58 ± 0.4831.47 ± 0.28
ELU86.02 ± 0.2465.63 ± 0.8737.09 ± 0.2429.24 ± 0.34
SELU85.95 ± 0.4365.07 ± 0.6235.56 ± 0.5527.91 ± 0.38
Tanh86.28 ± 0.5064.85 ± 0.8835.57 ± 0.5128.04 ± 0.11

Table 4.

We instantiate this protocol with FAct (Fourier Activation), a learnable activation represented by a two-harmonic truncated Fourier series with five coefficients. A single set of coefficients is shared across all MLP blocks of the network. The coefficients are initialized from a two-harmonic Fourier approximation of GELU [11] and optimized jointly with the source-network weights. We refer to the activation after source training, with its coefficients fixed, as Frozen FAct.

We evaluate Frozen FAct in two transfer settings. In the same-task setting, the activation learned by a depth-6 Vision Transformer (ViT) [7] on ImageNet-1K [6] is reused in a ViT of the same architecture, trained from scratch on the same task. In the cross-architecture setting, the same five coefficients are used in four depth-2 target architectures: a ViT with global self-attention, a Swin Transformer with shifted-window self-attention [22], a ViT with linear attention [16], and an attention-free MLP-Mixer [35]. These targets contain approximately 100K parameters each and are trained on four image-classification datasets: Fashion-MNIST [36], CIFAR-10, CIFAR-100 [20], and Food-101 [3].

In the same-task setting, Frozen FAct improves ImageNet-1K top-1 accuracy by 1.46 percentage points over GELU. In the cross-architecture setting, it achieves the highest mean test accuracy among eleven activation functions in 15 of the 16 architecture–dataset combinations under a common training configuration, with improvements of up to 1.71 percentage points over the best conventional activation. When the learning rate and weight decay are tuned separately for each activation under the same search budget, Frozen FAct still ranks first in eleven combinations, second in four, and fourth in one. Finally, on Swin targets of one to six blocks, a frozen, untrained Fourier approximation of GELU does not reproduce these gains, indicating that the benefit comes from the learned activation rather than the Fourier parameterization alone. In contrast to the SPLASH ablation of Tavakoli et al. [33], these results show that a frozen learned activation can retain its benefit after transfer.

This work makes three contributions:

  1. A learn–freeze–transfer protocol for activation functions. We formulate a complementary form of transfer in which a globally shared activation is learned jointly with a source network in a single training run, then frozen and reused in target networks trained from scratch, without transferring source-network weights.

  1. A globally shared activation suited for transfer. We instantiate the protocol with FAct, whose five coefficients are shared across all MLP blocks, so that the learned activation can be placed in networks of different architectures.

  1. An empirical evaluation of activation transferability. We evaluate same-task and cross-architecture transfer using three transformer architectures and an attention-free MLP-Mixer on four image-classification datasets. We compare Frozen FAct with ten conventional activation functions under common and activation-specific hyperparameter configurations, and we include a control showing that the gains come from the learned activation rather than its parameterization.

Related Work

Transfer Learning

Transfer learning reuses knowledge acquired on a source task to improve learning on a target task [28]. In deep networks, this is typically done by reusing learned weights, either to initialize a target network that is then fine-tuned or as fixed feature extractors [37]. Knowledge distillation transfers knowledge between networks of different architectures, but it requires the source network during target training to provide supervision [12]. In both cases, the transferred knowledge is carried by the source network’s weights or outputs. We study a complementary setting in which only a learned activation function is transferred, and the source network is not used during target training.

Learnable and Globally Shared Activation Functions

Learnable activation functions introduce parameterized nonlinear transformations whose parameters are optimized jointly with the network weights. PReLU learns the slope of the negative input region [10], and ACON provides a more flexible parametric formulation [23]. Padé activation units represent the nonlinearity as a ratio of polynomials [26], while other approaches use adaptive piecewise-linear functions [1] or kernel-based representations [30]. Learnable Fourier-type activations have also been studied, with coefficients learned within the network in which they are used [17, 19, 31].

Activation parameters also differ in how they are shared. They may be assigned to individual neurons or layers, or shared across multiple layers. Jagtap et al. [14] introduced globally adaptive activation functions for deep and physics-informed neural networks, using a single trainable scaling parameter that adjusts the activation slope across the network. Delfosse et al. [5] shared a common set of rational-function parameters across layers to regulate activation plasticity in deep reinforcement learning. These approaches show that sharing activation parameters within a network can be useful. However, sharing alone does not show whether a learned activation can be separated from its source network and reused in another network.

Reuse and Transfer of Activation Functions

Several studies have reused activation functions beyond the setting in which they were obtained. Ramachandran et al. [29] discovered Swish by automated search on a small network and showed that it transfers to larger models and other tasks. Bingham and Miikkulainen [2] used surrogate-based search to find activation functions that transfer from small search tasks to larger models, including ResNet-50 on ImageNet. Strack et al. [32] used gradient-based search and transferred the resulting activations, with their constants fixed, to larger models of the same type and to new datasets. Teney et al. [34] meta-learned activation functions through bilevel optimization across task episodes, demonstrating transfer across algorithmic tasks and reuse in networks of different widths.

Reusing a learned shape does not always preserve its benefit. Tavakoli et al. [33] found that a fixed activation mimicking the final shape of trained SPLASH units performed only on par with ReLU, and concluded that the evolution of the shape during training was important. Although this was not a transfer study, it suggests that freezing a learned activation may discard part of its benefit.

The present study differs from these works in two respects. First, the activation is learned in a single conventional training run, without search or meta-learning. Second, the frozen coefficients are transferred across architecture families, from a ViT to Swin, linear-attention, and MLP-Mixer targets, without reusing source-network weights or updating the activation during target training.

FAct: Learning and Transferring a Shared Nonlinearity

Global Activation Sharing

Let zℓz_\ell denote the pre-activation at the ℓ\ell-th activation site in a network, and let ϕ(z;θ)\phi(z;\theta) denote a parameterized activation function. In FAct, a single parameter vector θ\theta is shared across all MLP blocks of the network:

hℓ=ϕ(zℓ;θ),ℓ=1,…,L,(1)h_\ell= \phi(z_\ell;\theta), \quad\ell= 1,\ldots,L, \tag*{(1)}

where LL denotes the number of activation sites and hℓh_\ell is the corresponding activation output. The same parameter vector is used at every site, without replication across layers or hidden units.

Because the activation parameters are shared, their gradients aggregate contributions from all activation sites during training:

∂J∂θ=∑ℓ=1L∂J∂hℓ∂ϕ(zℓ;θ)∂θ,(2)\frac{\partial\mathcal{J}}{\partial\theta} = \sum_{\ell=1}^{L} \frac{\partial\mathcal{J}}{\partial h_\ell} \frac{\partial\phi(z_\ell;\theta)}{\partial\theta}, \tag*{(2)}

where J\mathcal{J} denotes the network training objective. Each term represents the gradient contribution propagated through an individual activation site. Consequently, the shared activation parameters are optimized using signals from all MLP blocks rather than through independently learned, layer-specific parameter sets.

Global sharing defines a single parameterized nonlinear function that can be retained independently of the source-network weights. We refer to FAct while its parameters are jointly optimized with the source-network weights as Learnable FAct. After source training, the parameters are frozen, and the resulting activation is referred to as Frozen FAct.

Fourier Parameterization and Initialization

We parameterize FAct using a truncated Fourier series:

ϕ(z;θ)=a0+∑k=1K[akcos⁡(kωz)+bksin⁡(kωz)],(3)\phi(z;\theta) = a_0 + \sum_{k=1}^{K} [a_k \cos(k\omega z) + b_k \sin(k\omega z)], \tag*{(3)}

where a0a_0, aka_k, and bkb_k are learnable coefficients, and KK denotes the number of harmonics. We fix the angular frequency to ω=1\omega= 1, giving the activation a period of 2π2\pi. The parameter vector is defined as

θ=(a0,a1,…,aK,b1,…,bK),(4)\theta= (a_0,a_1,\ldots,a_K,b_1,\ldots,b_K), \tag*{(4)}

containing 2K+12K + 1 coefficients. In the main experiments, we set K=2K = 2, resulting in five learnable parameters shared across the entire network. Consequently, the number of activation parameters is independent of network depth and width.

The Fourier representation provides a compact and smooth parameterization whose shape can adapt during training. Depending on the learned coefficients, the resulting function can exhibit monotonic or non-monotonic behaviour over the input range of interest. Furthermore, its output is bounded for any finite coefficient values:

∣ϕ(z;θ)∣≤∣a0∣+∑k=1K(∣ak∣+∣bk∣).(5)|\phi(z;\theta)| \le|a_0| + \sum_{k=1}^{K} (|a_k| + |b_k|). \tag*{(5)}

This bound follows from the bounded magnitudes of the sine and cosine functions.

We initialize FAct using a two-harmonic Fourier approximation of GELU [11]. For a reference activation function σ\sigma, the Fourier coefficients are obtained by projection over one period, [−π,π][-\pi, \pi]:

a0=12π∫−ππσ(z) dz,(6)a_0 = \frac{1}{2\pi} \int_{-\pi}^{\pi} \sigma(z)\,\mathrm{d}z, \tag*{(6)}
ak=1π∫−ππσ(z)cos⁡(kωz) dz,k=1,…,K,(7)a_k = \frac{1}{\pi} \int_{-\pi}^{\pi} \sigma(z)\cos(k\omega z)\,\mathrm{d}z,\quad k=1,\ldots,K, \tag*{(7)}
bk=1π∫−ππσ(z)sin⁡(kωz) dz,k=1,…,K.(8)b_k = \frac{1}{\pi} \int_{-\pi}^{\pi} \sigma(z)\sin(k\omega z)\,\mathrm{d}z,\quad k=1,\ldots,K. \tag*{(8)}

For σ=GELU\sigma= \mathrm{GELU}, K=2K = 2, and ω=1\omega= 1, the resulting initialization is

θ0=(0.7061, −0.7049, 0.0261, 1.0000, −0.5000).(9)\theta_0 = (0.7061,\,-0.7049,\,0.0261,\,1.0000,\,-0.5000). \tag*{(9)}

This initialization provides a reference nonlinear transformation while allowing all five coefficients to adapt during source-network training.

Learn–Freeze–Transfer Protocol

The globally shared parameterization allows the learned activation to be separated from the source network and reused independently. During source training, the source-network weights and shared activation parameters are jointly optimized:

(Ws∗,θ∗)=arg⁡min⁡Ws,θJsrc(Ws,θ),(10)(W_s^*,\theta^*) = \arg\min_{W_s,\theta} \mathcal{J}_{\mathrm{src}}(W_s,\theta), \tag*{(10)}

where WsW_s denotes the source-network weights, θ\theta represents the shared activation parameters, and Jsrc\mathcal{J}_{\mathrm{src}} is the source-task training objective. The resulting activation is

ϕ∗(z)=ϕ(z;θ∗).(11)\phi^*(z) = \phi(z;\theta^*). \tag*{(11)}

After source training, the learned activation parameters θ∗\theta^* are frozen. Only these parameters are transferred to a target network; the source-network weights Ws∗W_s^* are not transferred. Let WtW_t denote the weights of an independently initialized target network, and let Jtgt\mathcal{J}_{\mathrm{tgt}} denote its training objective. Target training then optimizes

Wt∗=arg⁡min⁡WtJtgt(Wt;θ∗),θ∗ fixed.(12)W_t^* = \arg\min_{W_t} \mathcal{J}_{\mathrm{tgt}}(W_t;\theta^*),\qquad\theta^* \text{ fixed}. \tag*{(12)}

Thus, the target-network weights are optimized for the target task while the transferred activation parameters remain unchanged throughout training. No source-network weights are used to initialize the target model, and target-task gradients do not update the transferred activation.

This protocol isolates the reuse of the learned nonlinearity from the transfer of network weights. It therefore enables us to examine whether an activation learned jointly with one source network remains useful when incorporated into independently initialized networks, without further adaptation of the activation itself. Figure 1 illustrates the three stages of this process.

Experimental Evaluation

This section evaluates whether a globally shared activation learned jointly with a source network remains useful when transferred to independently initialized target networks. We first describe the source and target architectures, datasets, and training protocol. We then examine the evolution of the source-learned activation, its transfer to a network with the same architecture and task, and its transfer across four target architectures with different attention and token-mixing mechanisms. Frozen FAct is compared with ten conventional activation functions to assess its performance under a common training configuration.

Experimental Setup

Source and target networks. We learn FAct in a depth-6 Vision Transformer (ViT-d6; throughout, the suffix -dk denotes a network of depth kk) trained on ImageNet-1K [6]. The source network contains 3,048,232 parameters, and learning FAct adds five trainable activation coefficients, resulting in 3,048,237 trainable parameters. The same five coefficients are shared across all MLP blocks of the source network. We train the source network with five random seeds (seeds 1–5); each run jointly optimizes the source-network weights and the shared activation from the same GELU-based initialization. Unless otherwise stated, the transfer experiments reuse the activation learned by the seed-1 source checkpoint; the five source runs converge to nearly identical activations, as quantified in § 4.2, so this choice is representative rather than specific to one source run.

After source training, the learned activation is transferred to four independently initialized target architectures: a depth-2 Vision Transformer (ViT-d2) with global self-attention, a depth-2 Swin Transformer (Swin-d2) with shifted-window self-attention [22], a depth-2 Vision Transformer with linear attention (LinViT-d2) [16], and a depth-2 MLP-Mixer [35]. Each target network contains approximately 100K parameters, with exact configurations provided in Appendix § 6. No source-network weights are transferred, and the activation coefficients remain fixed throughout target training.

Datasets and transferred activation. Each target architecture is evaluated on Fashion-MNIST [36], CIFAR-10, CIFAR-100 [20], and Food-101 [3]. All target experiments use the same frozen activation coefficients obtained from the seed-1 source checkpoint: a0=0.2182a_0 = 0.2182, a1=0.1193a_1 = 0.1193, a2=−0.4237a_2 = -0.4237, b1=0.8315b_1 = 0.8315, and b2=−0.5918b_2 = -0.5918. Thus, the same source-learned activation is used across all target architectures and datasets.

Training and evaluation protocol. All target experiments use five random seeds, with the same seed configurations across architectures, datasets, and activation choices. Models are trained for 100 epochs using AdamW, with a learning rate of 10−310^{-3}, weight decay of 0.05, and a batch size of 256. The learning-rate schedule consists of five epochs of linear warm-up followed by cosine decay. Label smoothing and dropout are both set to 0.1. All activation choices use identical dataset-specific preprocessing and augmentation, without test-time augmentation.

Frozen FAct is compared with ten conventional activation functions: ReLU [27], LeakyReLU [24], GELU [11], SiLU [9], Mish [25], HardSwish [13], Softplus [8], ELU [4], SELU [18], and Tanh [21]. For each run, the checkpoint with the highest validation accuracy is selected and evaluated once on the held-out test set. Results are reported as mean and sample standard deviation across five seeds. Activation-specific learning-rate and weight-decay tuning results are provided in Appendix § 7.

Evolution of the Source-Learned Activation

FAct is initialized using a two-harmonic Fourier approximation of GELU, with all five coefficients subsequently optimized jointly with the source-network weights. Figure 2 illustrates the evolution of the shared activation during ImageNet-1K training. The activation progressively departs from its initial GELU approximation and develops a non-monotonic shape over the course of training. The final learned function, obtained after 100 epochs, differs substantially from its initialization and is the activation frozen and transferred to all target networks in the subsequent experiments.

Figure 2. Evolution of the FAct activation during source-network training. (a) GELU and its two-harmonic Fourier approximation used to initialize FAct. (b) Evolution of FAct at selected training epochs. (c) Final learned activation after 100 epochs compared with GELU. The final curve corresponds to the seed-1 source checkpoint used in all transfer experiments. The shaded region indicates the central 98% of source-network pre-activations, pooled across all six MLP blocks on 512 ImageNet-1K validation images.

Consistency across source runs. Although the transfer experiments reuse the activation from a single source checkpoint, the activation that FAct learns is highly consistent across independent source runs. Training the source network with five random seeds yields nearly identical coefficients, with mean ±\pm sample standard deviation across seeds of a0=0.2155±0.0030a_0 = 0.2155 \pm0.0030, a1=0.1181±0.0038a_1 = 0.1181 \pm0.0038, a2=−0.4205±0.0021a_2 = -0.4205 \pm0.0021, b1=0.8457±0.0126b_1 = 0.8457 \pm0.0126, and b2=−0.5990±0.0096b_2 = -0.5990 \pm0.0096.

Same-Task Transfer at Source Scale

To examine whether the learned activation remains useful when transferred to a network with the same architecture and task, we train a freshly initialized ViT-d6 on ImageNet-1K using the frozen FAct activation learned by the source network. The five activation coefficients are obtained from the seed-1 source checkpoint and remain fixed throughout target training. No source-network weights are transferred. We conduct five independent target runs, all using the same frozen activation, so the reported variation reflects differences in target initialization and training rather than variability across independently learned activations.

Table 1 compares Frozen FAct with GELU under the same architecture and training configuration. Frozen FAct achieves a mean top-1 accuracy of 63.91% ±\pm 0.15, compared with 62.45% ±\pm 0.13 for GELU, an improvement of 1.46 percentage points. This result provides evidence that the source-learned activation retains useful properties even when separated from its original network weights and reused in an independently initialized network performing the same task.

Transfer Across Transformer Architectures

Tables 2–4 report the test accuracy of the eleven activation choices on ViT-d2, Swin-d2, and LinViT-d2, respectively, under the common training configuration. Table 5 summarizes the performance difference between Frozen FAct and the best-performing conventional activation for each target–dataset combination. Figure 3 illustrates the corresponding accuracy margins in its three transformer panels, alongside those of the MLP-Mixer target of Section 4.5.

Across the three transformer architectures and four datasets, Frozen FAct achieves the highest mean test accuracy in all twelve architecture–dataset combinations. Its improvement over the best-performing conventional activation ranges from 0.38 percentage points on LinViT-d2 with Fashion-MNIST to 1.71 percentage points on ViT-d2 with Food-101. The strongest conventional baseline varies across architectures: ReLU or LeakyReLU provides the highest mean accuracy among conventional activations for ViT-d2 and LinViT-d2, whereas GELU performs best across all four Swin-d2 datasets. Importantly, the same five activation coefficients learned by the source ViT-d6 are used in all target experiments, without transferring source-network weights or updating the activation during target training. These results provide empirical evidence that a globally shared nonlinearity learned within one network can retain its usefulness across independently initialized transformer architectures with different attention configurations and target tasks.

For reference, we also compare Frozen FAct with FAct learned directly in each ViT-d2 target under the same common configuration (Appendix § 9). The transferred activation performs comparably to one learned on the target task: mean differences are within 0.14 percentage points on Fashion-MNIST and CIFAR-10, and Frozen FAct achieves higher mean accuracy on CIFAR-100 and Food-101.

Additional analyses investigate the effect of the GELU-based initialization and of target-network depth in Swin targets (Appendix § 8).

Transfer to an MLP-Mixer Architecture

To examine whether the learned activation transfers beyond attention-based architectures, we evaluate Frozen FAct in an independently initialized depth-2 MLP-Mixer [35]. Unlike the transformer architectures, MLP-Mixer uses separate MLPs for token mixing and channel mixing, without self-attention. We transfer the same five source-learned activation coefficients to both types of MLP blocks and keep them fixed throughout target training. The comparison includes the same ten conventional activation functions under the common training configuration.

Frozen FAct achieves the highest mean test accuracy on Fashion-MNIST, CIFAR-10, and CIFAR-100, outperforming the best-performing conventional activation by 0.62, 0.48, and 0.29 percentage points, respectively (Table 6 and the MLP-Mixer panel of Figure 3). On Food-101, GELU performs better by approximately 1.05 percentage points; this is the only one of the sixteen target–dataset combinations in which Frozen FAct does not achieve the highest mean test accuracy. These results provide evidence that a source-learned activation can also be reused in an attention-free architecture

Table 6: Transfer performance on MLP-Mixer under the common training configuration. Mean test accuracy (%) ± sample s.d. across five seeds. Bold indicates the highest mean test accuracy for each dataset, and underlining indicates the second-highest. The comparison includes Frozen FAct and the same ten conventional activation functions as Tables 2–4.

ActivationFashion-MNISTCIFAR-10CIFAR-100Food-101
Frozen Fact91.42 ± 0.1875.99 ± 0.1647.49 ± 0.3835.09 ± 0.22
ReLU90.80 ± 0.2275.47 ± 0.3547.08 ± 0.4634.92 ± 0.35
LeakyReLU90.64 ± 0.39<u>75.51 ± 0.13</u><u>47.20 ± 0.48</u>34.94 ± 0.40
GELU89.84 ± 0.1375.32 ± 0.4845.99 ± 0.6036.15 ± 0.47
SiLU88.90 ± 0.2973.75 ± 0.3045.07 ± 0.8335.15 ± 0.30
Mish88.94 ± 0.2574.03 ± 0.3045.17 ± 0.58<u>35.25 ± 0.25</u>
HardSwish87.81 ± 0.4172.78 ± 0.4244.05 ± 0.7834.39 ± 0.26
Softplus86.98 ± 0.4268.26 ± 1.0635.33 ± 1.8432.70 ± 0.12
ELU86.52 ± 0.2270.19 ± 0.5641.27 ± 0.7830.82 ± 0.38
SELU86.02 ± 0.2264.97 ± 0.2236.07 ± 0.4624.90 ± 0.25
Tanh86.53 ± 0.1865.42 ± 0.5636.19 ± 0.4329.97 ± 0.29

Table 6.

Figure 3. Accuracy margin of Frozen FAct over the best-performing conventional activation across the sixteen architecture–dataset combinations. Each panel represents a target architecture. Each point denotes the mean seed-matched difference in test accuracy (percentage points) across five seeds. Error bars represent one sample standard deviation of the paired differences. The conventional activation selected for each comparison is indicated below the corresponding dataset.

Table 5: Frozen FAct compared with the best-performing conventional activation across the twelve transformer architecture–dataset combinations under the common training configuration. The best conventional activation is selected from the ten baseline functions, excluding Frozen FAct. Accuracy values are mean test accuracy (%). Δ denotes the seed-matched mean accuracy difference between Frozen FAct and the selected conventional activation, in percentage points, across five seeds.

TargetDatasetFrozen FActBest-performing conventional activationΔ (pp)
ViT-d2Fashion-MNIST90.94ReLU (90.40)+0.53
CIFAR-1075.56LeakyReLU (74.51)+1.04
CIFAR-10047.67ReLU (46.95)+0.72
Food-10135.83LeakyReLU (34.12)+1.71
Swin-d2Fashion-MNIST89.33GELU (88.29)+1.05
CIFAR-1069.47GELU (67.82)+1.65
CIFAR-10038.34GELU (37.25)+1.09
Food-10141.32GELU (40.43)+0.88
LinViT-d2Fashion-MNIST90.67ReLU (90.29)+0.38
CIFAR-1075.15ReLU (73.94)+1.21
CIFAR-10048.05ReLU (46.57)+1.48
Food-10136.12LeakyReLU (34.50)+1.62

Table 5.

without transferring source-network weights or updating its coefficients, although its performance advantage varies across datasets. After activation-specific hyperparameter tuning, Frozen FAct retains its advantage on Fashion-MNIST and CIFAR-10, whereas HardSwish performs better on CIFAR-100 and Food-101.

Conclusion

This work investigated whether a globally shared activation learned jointly with a source network can be reused independently of the network weights with which it was optimized. We introduced a learn–freeze–transfer framework and instantiated it using FAct, a five-parameter activation represented by a two-harmonic truncated Fourier series. Initialized using a Fourier approximation of GELU, its coefficients are optimized jointly with a source Vision Transformer and subsequently frozen for transfer to independently initialized target networks.

We evaluated the proposed approach through same-task and cross-architecture transfer experiments. In the same-task setting, Frozen FAct improved ImageNet-1K top-1 accuracy by 1.46 percentage points over GELU when used in a freshly initialized ViT-d6. Across four target architectures and four image-classification datasets, Frozen FAct achieved the highest mean test accuracy among eleven activation choices in 15 of the 16 architecture–dataset combinations under a common training configuration. This included all twelve combinations involving the three transformer architectures and three of four MLP-Mixer combinations. Following activation-specific hyperparameter tuning, Frozen FAct ranked first in eleven combinations, second in four, and fourth in the remaining combination. Throughout these experiments, only the five learned activation coefficients were transferred; source-network weights were not reused, and the activation remained fixed during target training.

These findings provide empirical evidence that a globally shared nonlinearity learned within one network can retain its usefulness when transferred to independently initialized networks with different architectures and target tasks. However, the evaluation uses a single source-learned activation obtained from one Vision Transformer trained on ImageNet-1K, and the cross-architecture experiments are restricted to relatively small target networks and image-classification tasks. Further investigation is needed to establish how activation transferability varies across source architectures, source training datasets, model scales, and target tasks. Extending this evaluation to convolutional and other network architectures would help determine the broader applicability of transferable learned nonlinearities. The mechanism behind the gains also remains open.

Experimental Configuration

Table 7 summarizes the architectural configurations and parameter counts of the source and target networks. The MLP-Mixer consists of two blocks, each comprising a pre-normalized residual token-mixing MLP (hidden width 38) followed by a pre-normalized residual channel-mixing MLP (hidden width 304), with an embedding dimension of 76. Classification is performed using globally average-pooled tokens, without a class token or positional embeddings. All MLPs use a Linear–activation–dropout–Linear structure, with a single activation module shared across the MLPs within each network.

SourceTargets
ViT-d6ViT-d2Swin-d2LinViT-d2MLP-Mixer
Token mixingglobal softmaxglobal softmaxshifted window (4)linear, elu+1MLP
Depth / width / heads6 / 192 / 62 / 64 / 42 / 64 / 42 / 64 / 42 / 76 / –
MLP hidden width76825625625638 token / 304 channel
Patch size164 (Food-101: 8)44 (Food-101: 8)4 (Food-101: 8)
Parameters3,048,232105,098–123,237102,354–110,317105,098–123,237103,620–126,313
DataImageNet-1K, 2242224^2Fashion-MNIST 28228^2, CIFAR-10/100 32232^2, Food-101 64264^2
Validation split10,000 train images5,000 train images per dataset, fixed
Test splitILSVRC val (50,000)official test set

Table 7. Architectural and dataset configurations of the source and target networks. Parameter ranges reflect differences across the four target datasets.

The target datasets comprise Fashion-MNIST (28×2828 \times28), CIFAR-10 and CIFAR-100 (32×3232 \times32), and Food-101 (resized to 64×6464 \times64). For each dataset, 5,000 training images are held out using a fixed random seed to form a validation set shared across experiments. The official test sets are retained for final evaluation.

Activation-Specific Hyperparameter Tuning

To examine whether the results under the common training configuration depend on the choice of learning rate and weight decay, we conduct activation-specific hyperparameter tuning. For each activation–architecture–dataset combination, we evaluate 15 configurations obtained from five learning rates (3×10−43 \times10^{-4}, 10−310^{-3}, 3×10−33 \times10^{-3}, 10−210^{-2}, and 3×10−23 \times10^{-2}) and three weight-decay values (0, 0.05, and 0.2). Each configuration is evaluated using a 30-epoch, single-seed proxy run. The selected configuration is then trained for 100 epochs using five random seeds. The same search budget is applied to all eleven activation functions across the four target architectures and four datasets.

Table 8 compares Frozen FAct with the best-performing conventional activation under the tuned configurations. Frozen FAct achieves the highest mean accuracy in 11 of the 16 architecture–dataset combinations, ranks second in four, and fourth in the remaining combination. Across the three transformer architectures, it ranks first in nine of the twelve combinations, with GELU achieving higher accuracy in the other three. For the MLP-Mixer, Frozen FAct retains the highest accuracy on Fashion-MNIST and CIFAR-10, whereas HardSwish performs better on CIFAR-100 and Food-101.

TargetDatasetFrozen FActBest-performing conventional ActivationAccuracyΔ\Delta (pp)
ViT-d2Fashion-MNIST91.17±0.1791.17 \pm0.17LeakyReLU90.69±0.3190.69 \pm0.31+0.48
CIFAR-1076.34±0.6976.34 \pm0.69GELU74.91±0.1374.91 \pm0.13+1.43
CIFAR-10048.33±0.4848.33 \pm0.48GELU47.06±0.3047.06 \pm0.30+1.27
Food-10139.15±0.4939.15 \pm0.49GELU37.13±0.4137.13 \pm0.41+2.01
Swin-d2Fashion-MNIST90.51±0.2390.51 \pm0.23GELU90.13±0.1790.13 \pm0.17+0.37
CIFAR-1072.57±0.5772.57 \pm0.57GELU73.07±0.2273.07 \pm0.22−0.50
CIFAR-10044.13±0.5444.13 \pm0.54LeakyReLU42.18±0.5942.18 \pm0.59+1.95
Food-10146.51±0.4046.51 \pm0.40GELU46.60±0.8646.60 \pm0.86−0.09
LinViT-d2Fashion-MNIST91.39±0.1591.39 \pm0.15ReLU90.76±0.2790.76 \pm0.27+0.63
CIFAR-1076.38±0.4576.38 \pm0.45GELU74.44±0.4474.44 \pm0.44+1.95
CIFAR-10048.51±0.4148.51 \pm0.41GELU48.18±0.3648.18 \pm0.36+0.33
Food-10138.34±0.2538.34 \pm0.25GELU38.70±0.6438.70 \pm0.64−0.36
MLP-MixerFashion-MNIST91.69±0.1591.69 \pm0.15GELU90.93±0.2990.93 \pm0.29+0.76
CIFAR-1077.89±0.6677.89 \pm0.66SiLU77.70±0.4477.70 \pm0.44+0.19
CIFAR-10049.01±0.5649.01 \pm0.56HardSwish51.48±0.9151.48 \pm0.91−2.47
Food-10142.48±0.1042.48 \pm0.10HardSwish43.27±1.2143.27 \pm1.21−0.79

Table 8. Frozen FAct compared with the best-performing conventional activation under activation-specific hyperparameter tuning. Results are mean test accuracy (%) ±\pm sample standard deviation across five seeds for both Frozen FAct and the best-performing conventional activation. Δ\Delta denotes the difference in mean accuracy between Frozen FAct and the selected conventional activation, in percentage points.

Figure 4 illustrates the changes in activation rankings between the common and tuned configurations. Although activation-specific tuning changes the relative rankings, Frozen FAct retains the highest mean accuracy in most of the evaluated combinations. These results indicate that its observed advantage is not eliminated by activation-specific hyperparameter optimization, although its relative performance remains dependent on the target architecture and dataset.

Figure 4. Activation rankings before and after activation-specific hyperparameter tuning. Each panel shows the ranking of eleven activation choices based on mean test accuracy across five seeds. The left and right positions represent rankings under the common and tuned configurations, respectively (rank 1 = highest accuracy). Lines connect the rankings of the same activation across the two configurations. Frozen FAct and GELU are highlighted; the remaining nine conventional activations are shown in grey.

Depth Sensitivity and the Effect of GELU-Based Initialization

This experiment examines whether the performance of Frozen FAct can be attributed to the learned activation coefficients rather than merely to the Fourier approximation of GELU, and how its transfer performance varies with target-network depth. We use Swin Transformer targets because GELU is the strongest conventional activation across all four datasets in the main experiments (Table 3), providing a consistent reference for both comparisons. The findings are therefore specific to the evaluated Swin configurations.

ActivationFashion-MNISTCIFAR-10CIFAR-100Food-101
Frozen FAct89.33 ± 0.1169.47 ± 0.3138.34 ± 0.3441.32 ± 0.26
ReLU88.28 ± 0.2867.31 ± 0.3336.87 ± 0.4439.20 ± 0.55
LeakyReLU88.27 ± 0.1567.56 ± 0.4136.85 ± 0.3139.20 ± 0.37
GELU88.29 ± 0.2267.82 ± 0.3137.25 ± 0.2740.43 ± 0.38
SiLU87.69 ± 0.1866.21 ± 0.3536.48 ± 0.4439.01 ± 0.54
Mish87.60 ± 0.0966.44 ± 0.3236.97 ± 0.6839.34 ± 0.32
HardSwish87.33 ± 0.2465.60 ± 0.2736.14 ± 0.3238.17 ± 0.75
Softplus85.73 ± 0.2762.61 ± 0.2632.39 ± 0.4035.67 ± 0.82
ELU85.55 ± 0.3961.03 ± 0.6531.55 ± 0.3532.66 ± 0.58
SELU83.27 ± 0.4054.81 ± 0.6825.71 ± 0.2928.75 ± 0.43
Tanh85.69 ± 0.3757.63 ± 0.9028.06 ± 0.4130.35 ± 0.43

Table 3. Table 3: Transfer performance on Swin-d2 with shifted-window self-attention under the common training configuration. Results are mean test accuracy (%) ± sample standard deviation across five seeds. Bold and underlining indicate the highest and second-highest mean accuracies, respectively.

Experimental Design

We vary the depth of the target Swin Transformer from one to six blocks while keeping the other architectural settings fixed. All models use a single stage with an embedding dimension of 64, four attention heads, an MLP ratio of 4, a patch size of 4, and a window size of 4. Blocks alternate between regular-window and shifted-window attention, beginning with a regular-window block followed by a shifted-window block with a shift of two tokens. No patch merging is applied, keeping the token grid unchanged across depths. Thus, the primary architectural variable is the number of blocks and the corresponding number of activation application sites.

We compare three activation configurations: (1) GELU; (2) Frozen GELU-init, the two-harmonic Fourier approximation of GELU obtained using the initialization in Eq. (9), with its coefficients fixed throughout training; and (3) Frozen FAct, using the coefficients learned by the ImageNet-1K source ViT and transferred without further updates. All target networks are independently initialized and trained using five random seeds under the common training configuration described in § 4.1. Neither frozen configuration uses source-network weights.

Results

Effect of the GELU-Based Fourier Initialization

We first compare Frozen GELU-init with GELU to examine whether the fixed Fourier approximation alone provides an advantage (Table 9). The results show that its performance is dataset-dependent rather than consistently better than GELU. Although Frozen GELU-init achieves higher mean accuracy at all six depths on Fashion-MNIST, its advantage is inconsistent on CIFAR-10 and absent on CIFAR-100 and Food-101. Thus, replacing GELU with its fixed Fourier approximation does not, by itself, explain the improvements observed with Frozen FAct.

DatasetDepthGELUFrozen GELU-initFrozen FAct
Fashion-MNIST183.93 ± 0.3584.10 ± 0.3285.03 ± 0.30
288.28 ± 0.1888.41 ± 0.2089.30 ± 0.06
389.65 ± 0.2690.07 ± 0.2490.71 ± 0.13
490.06 ± 0.0690.49 ± 0.1890.98 ± 0.26
590.71 ± 0.2491.11 ± 0.0791.83 ± 0.23
690.66 ± 0.3391.15 ± 0.1391.92 ± 0.13
CIFAR-10161.19 ± 0.6761.27 ± 0.3962.96 ± 0.35
267.83 ± 0.1967.47 ± 0.3569.31 ± 0.34
373.01 ± 0.6272.30 ± 0.5274.98 ± 0.28
474.05 ± 0.3973.53 ± 0.6275.82 ± 0.47
575.94 ± 0.2876.43 ± 0.3878.89 ± 0.48
675.53 ± 0.6276.68 ± 0.2578.60 ± 0.34
CIFAR-100130.72 ± 0.4230.71 ± 0.1830.80 ± 0.48
237.61 ± 0.3537.00 ± 0.1738.78 ± 0.40
344.09 ± 0.3743.31 ± 0.3144.87 ± 0.19
444.63 ± 0.2143.91 ± 0.2945.94 ± 0.31
547.64 ± 0.2746.96 ± 0.5149.58 ± 0.36
647.78 ± 0.4747.54 ± 0.4249.25 ± 0.40
Food-101131.93 ± 0.2430.43 ± 0.5131.74 ± 0.33
240.79 ± 0.3439.35 ± 0.3541.41 ± 0.34
346.85 ± 0.1645.20 ± 0.1447.95 ± 0.55
449.57 ± 0.3948.16 ± 0.4451.03 ± 0.43
551.26 ± 0.3049.85 ± 0.4453.63 ± 0.39
652.27 ± 0.4150.98 ± 0.2654.92 ± 0.19

Table 9. Test accuracy across Swin Transformer target depths. Results are mean accuracy (%) ± sample standard deviation across five seeds. The three configurations are GELU, Frozen GELU-init and Frozen FAct. Bold indicates the highest mean accuracy at each dataset–depth combination.

Sensitivity to Target-Network Depth

We next examine how the performance of the transferred activation varies with target-network depth (Table 9 and Figure 5). Frozen FAct achieves the highest mean accuracy among the three configurations at every depth on Fashion-MNIST, CIFAR-10, and CIFAR-100, and at five of the six depths on Food-101.

Figure 5. Effect of target-network depth on activation performance. Test accuracy of GELU, Frozen GELU-init and Frozen FAct across Swin Transformer target depths from one to six blocks on Fashion-MNIST, CIFAR-10, CIFAR-100 and Food-101. Points represent mean test accuracy across five seeds, and error bars indicate one sample standard deviation. All target networks are independently initialized and trained with their respective activation configurations.

Notably, Frozen FAct outperforms Frozen GELU-init across all 24 dataset–depth combinations, with improvements ranging from 0.09 to 3.94 percentage points. This indicates that the gains are not explained solely by the Fourier representation of GELU; the source-learned coefficients contribute additional benefits under the evaluated settings. The advantage over GELU also varies with dataset and depth, becoming more pronounced at greater depths on CIFAR-10, CIFAR-100, and Food-101.

Overall, these results show that the transferred activation remains effective across the evaluated Swin target depths, while its performance advantage depends on the target dataset and network depth.

Comparison with Activation Learning in the Target Network

To compare activation transfer with learning an activation directly for the target task, we train FAct jointly with each ViT-d2 target network (Learnable FAct), using the same Fourier parameterization and GELU-based initialization as the source network. We compare this configuration with Frozen FAct transferred from the ImageNet-1K source ViT. Both configurations use the common training protocol and five identical random seeds. The Frozen FAct results correspond to the same runs reported in Table 2.

ActivationFashion-MNISTCIFAR-10CIFAR-100Food-101
Frozen FAct90.94 ± 0.1075.56 ± 0.3547.67 ± 0.4835.83 ± 0.36
ReLU90.40 ± 0.4674.50 ± 0.4646.95 ± 0.6334.07 ± 0.22
LeakyReLU90.27 ± 0.2774.51 ± 0.2646.80 ± 0.2134.12 ± 0.21
GELU89.09 ± 0.2472.46 ± 0.2946.30 ± 0.2434.10 ± 0.39
SiLU87.96 ± 0.5770.47 ± 0.3944.60 ± 0.3432.51 ± 0.54
Mish87.91 ± 0.3570.62 ± 0.4944.85 ± 0.3132.69 ± 0.32
HardSwish87.53 ± 0.5868.01 ± 0.1542.13 ± 0.3331.10 ± 0.72
Softplus86.27 ± 0.3768.51 ± 0.4140.44 ± 0.8031.84 ± 0.22
ELU86.32 ± 0.0966.25 ± 0.1938.44 ± 0.6829.36 ± 0.57
SELU86.24 ± 0.5165.29 ± 0.5037.12 ± 0.2628.71 ± 0.24
Tanh86.61 ± 0.2065.14 ± 0.3237.24 ± 0.2028.67 ± 0.40

Table 2. Table 2: Transfer performance on ViT-d2 with global self-attention under the common training configuration. Results are mean test accuracy (%) ± sample standard deviation across five seeds. Bold and underlining indicate the highest and second-highest mean accuracies, respectively.

Table 10 summarizes the comparison. On Fashion-MNIST and CIFAR-10, the mean accuracy differences between the two configurations are −0.14-0.14 and 0.000.00 percentage points, respectively. On CIFAR-100 and Food-101, Frozen FAct achieves higher mean accuracies by 0.940.94 and 0.630.63 percentage points, respectively.

DatasetLearnable FActFrozen FActΔ\Delta (pp)
Fashion-MNIST91.08±0.2091.08 \pm0.2090.94±0.1090.94 \pm0.10−0.14-0.14
CIFAR-1075.56±0.2175.56 \pm0.2175.56±0.3575.56 \pm0.350.000.00
CIFAR-10046.73±0.3246.73 \pm0.3247.67±0.4847.67 \pm0.48+0.94+0.94
Food-10135.20±0.4035.20 \pm0.4035.83±0.3635.83 \pm0.36+0.63+0.63

Table 10. Comparison of Frozen FAct transferred from the source ViT-d6 and Learnable FAct optimized within the target ViT-d2. Results report mean test accuracy (%) ±\pm sample standard deviation across five seeds. Δ\Delta denotes the difference in mean accuracy (Frozen − Learnable), in percentage points.

These results indicate that, in the evaluated ViT-d2 settings, transferring the source-learned activation can achieve performance comparable to or higher than learning the activation directly within the target network. This comparison is limited to ViT-d2 and does not establish that the two approaches are equivalent across architectures or datasets.

References

  1. [1]Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning activation functions to improve deep neural networks. In ICLR Workshop, 2015.DOI
  2. [2]Garrett Bingham and Risto Miikkulainen. Discovering parametric activation functions. Neural Networks, 148:48–65, 2022.DOI
  3. [3]Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 – mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), pages 446–461, 2014.DOI
  4. [4]Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In International Conference on Learning Representations (ICLR), 2016.arxiv.org/abs/1511.07289
  5. [5]Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting. Adaptive rational activations to boost deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2024. Earlier version: Recurrent Rational Networks, arXiv:2102.09407.arxiv.org/abs/2102.09407
  6. [6]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009.
  7. [7]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
  8. [8]Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. In Advances in Neural Information Processing Systems (NIPS), pages 472–478, 2001.
  9. [9]Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3–11, 2018.DOI
  10. [10]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In IEEE International Conference on Computer Vision (ICCV), pages 1026–1034, 2015.arxiv.org/abs/1502.01852
  11. [11]Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv:1606.08415, 2016.
  12. [12]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.arxiv.org/abs/1503.02531
  13. [13]Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for MobileNetV3. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 1314–1324, 2019.arxiv.org/abs/1905.02244
  14. [14]Ameya D. Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404:109136, 2020.DOI
  15. [15]Ameya D. Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks. Proceedings of the Royal Society A, 476(2239):20200334, 2020.DOI
  16. [16]Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5156–5165, 2020.arxiv.org/abs/2006.16236
  17. [17]Ismail Khalfaoui-Hassani and Stefan Kesselheim. Polynomial, trigonometric, and tropical activations. In International Conference on Learning Representations (ICLR), 2026. arXiv:2502.01247.
  18. [18]Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017.arxiv.org/abs/1706.02515
  19. [19]Daehwa Ko, JaeHyeon Kim, SeungHyun Ham, and Jay Hoon Jung. TESLA: Taylor expansion of sinusoidal learnable activations. In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 300 of Proceedings of Machine Learning Research, pages 5257–5265, 2026.arxiv.org/abs/2608.11970
  20. [20]Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  21. [21]Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient BackProp. In Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pages 9–50. Springer, 1998.
  22. [22]Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, 2021.arxiv.org/abs/2103.14030
  23. [23]Ningning Ma, Xiangyu Zhang, Ming Liu, and Jian Sun. Activate or not: Learning customized activation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8032–8042, 2021.arxiv.org/abs/2009.04759
  24. [24]Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
  25. [25]Diganta Misra. Mish: A self regularized non-monotonic activation function. In British Machine Vision Conference (BMVC), 2020. arXiv:1908.08681.arxiv.org/abs/1908.08681
  26. [26]Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1907.06732
  27. [27]Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning (ICML), pages 807–814, 2010.
  28. [28]Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359, 2010.DOI
  29. [29]Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. In ICLR Workshop, 2018. arXiv:1710.05941.arxiv.org/abs/1710.05941
  30. [30]Simone Scardapane, Steven Van Vaerenbergh, Simone Totaro, and Aurelio Uncini. Kafnets: Kernel-based non-parametric activation functions for neural networks. Neural Networks, 110:19–32, 2019.DOI
  31. [31]Tamir Shor, Or Litany, and Alex M. Bronstein. Learnable spectral activations. arXiv preprint arXiv:2610.07419, 2026.arxiv.org/abs/2610.07419
  32. [32]Lukas Strack, Mahmoud Safari, and Frank Hutter. Efficient search for customized activation functions with gradient descent. arXiv preprint arXiv:2408.06820, 2024.arxiv.org/abs/2408.06820
  33. [33]Mohammadamin Tavakoli, Forest Agostinelli, and Pierre Baldi. SPLASH: Learnable activation functions for improving accuracy and adversarial robustness. Neural Networks, 140:1–12, 2021.DOI
  34. [34]Damien Teney, Liangze Jiang, Florin Gogli anu, and Ehsan Abbasnejad. Do we always need the simplicity bias? looking for optimal inductive biases in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.arxiv.org/abs/2503.10065
  35. [35]Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-mixer: An all-MLP architecture for vision. In Advances in Neural Information Processing Systems, volume 34, pages 24261–24272, 2021.arxiv.org/abs/2105.01601
  36. [36]Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747, 2017.arxiv.org/abs/1708.07747
  37. [37]Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems (NeurIPS), pages 3320–3328, 2014.arxiv.org/abs/1411.1792

Paper details

vlnc:W_xhpnp4gz
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 3

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents