betaFeedback

WorksComputer Science

Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions

Abstract

Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplained. Activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.We show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in PyTorch for shallow and deep CNNs are given to further strengthen our case.

Introduction

Convolutional Neural Networks (CNNs) are central in image-specific tasks, serving as feature extractors that eliminate the need for explicit feature engineering in image classification. Their applications ranges from diabetic retinopathy screening [1], lesion detection [2], skin lesion classification [3], human action recognition [4], face recognition [5] and document analysis [6].

Despite their widespread use, CNNs grapple with certain limitations, including poorly understood shift-invariance, data overfitting, and reliance on oversimplified nonlinear activation functions like ReLU [7] and leaky ReLU [7]. Nonlinear activation functions like ReLU and leaky ReLU have gained prominence in computer vision [8] and deep neural networks [9]. Though less complex than sigmoids [7] or hyperbolic tangent functions (Tanh) [10], they partially address the vanishing gradient problem [11]. However, optimal results might require various activations for individual filters in an image classification CNN with multiple filters. The ideal number of filters for specific applications remains an open area for exploration.

While these activations enable universal approximation in multilayer perceptrons, endeavors to devise adaptive or fixed PLA functions [12], [13], [14], [15] have emerged. Notably, adaptive activation functions for deep CNNs are introduced in [16], where the author trains the curve’s slope and hinges using gradient descent techniques. The promising results in CIFAR-10, CIFAR-100 image recognition datasets [17], and high-energy physics involving Higgs boson decay modes [18] underscore their performance.

The main contribution of this paper is in understanding the usage of complex piece-wise linear activations, detailed through comprehensive experiments across a spectrum of neural network architectures. The paper offers a deep exploration of these activations in contrast to conventional Rectified Linear Units (ReLUs), showcasing their superior efficacy in CNNs and multilayer perceptrons (MLP). Our proposed adaptive activation algorithm, AdAct, exhibits promising enhancements in performance across various datasets, showcasing its potential as a robust alternative to conventional fixed activation functions. Our research work significantly advances the understanding of activation functions in neural networks, opening avenues for more nuanced design choices for improved model performance across various applications.

The remainder of this paper is organized as follows: Section 2 offers an overview of the CNN structure, notations, and existing literature. In Section 3, we delve into a detailed examination of trainable piecewise linear functions and their mathematical underpinnings. Section 4 focuses on elucidating the computational complexities associated with different algorithms. It presents experimental findings across a range of approximation and classification datasets. Additionally, this section includes results from shallow CNN experiments and insights from transfer learning applied to deep CNNs using the CIFAR-10 dataset. Finally, Section 7 discusses supplementary work and draws conclusions based on the findings presented in this paper.

Prior Work

Structure and Notation

In Figure 1, fpf_p denote the pthp^{th} input image and let ic(p)i_c(p) denote the correct class number of the pthp^{th} pattern, where pp varies from 1 to NvN_v, and NvN_v is the total number of training images or patterns. During forward propagation, a filter of size Nf×NfN_f \times N_f is convolved over the image f1f_1 with NrN_r rows NcN_c columns. The number of channels is denoted by CC, where color input images have CC equal to 3 and grayscale images have CC equal to 1. The net function output is np=tr+∑c=1CWf∗fpn_p = t_r + \sum_{c=1}^{C} W_f * f_p, where, npn_p is of size (K×Mo×No)(K \times M_o \times N_o), where KK is the number of filters, MoM_o is the height of the convolved image output and NoN_o is the width of the convolved image output. WfW_f is the filter of size (K×Nf×Nf×C)(K \times N_f \times N_f \times C). The threshold vector trt_r is added to the net function output. The stride ss is the number of filter shifts over input images. Note that the output npn_p is a threshold plus a sum of CC separate 2-D convolution, rather than a 3-D convolution.

Image not extracted

Figure 1. Shallow CNN with Linear softmax cross-entropy classifier

To achieve non-linearity, the convolved image with element npn_p is passed through a ReLU activation as Op=f′(np)O_p = f'(n_p), where OpO_p is the filter’s hidden unit activation output of size (K×Nrb×Ncb)(K \times N_{rb} \times N_{cb}), where Nrb×NcbN_{rb} \times N_{cb} is the row and column size of the output of the convolved image respectively. The net function npon_{po} for ithi^{th} element of the CNN’s output layer for the pthp^{th} pattern is npo=to+∑m=1Mo∑n=1No∑k=1KWo∗Opn_{po} = t_o + \sum_{m=1}^{M_o} \sum_{n=1}^{N_o} \sum_{k=1}^{K} W_o * O_p where WoW_o is the 4-dimensional matrix of size (M×Mo×No×K)(M \times M_o \times N_o \times K), which connects hidden unit activation outputs to the output layer net vector npon_{po}. OpO_p is a 3-dimensional hidden unit activation output matrix of size (Mo×No×K)(M_o \times N_o \times K) and tot_o is the vector of biases added to net output function. Before computing the final error, the vector npon_{po} undergoes an activation process, commonly through functions like softmax. Subsequently, the cross-entropy loss function [7] is employed to gauge the model’s performance. The optimization objective involves minimizing the loss function EceE_{ce} by adjusting the unknown weights. A common optimizer for CNN weights training is the Adam optimizer [19]. It boasts computational efficiency and ease of implementation compared to the optimal learning factor [20]. Leveraging momentum and adaptive learning rates, it accelerates convergence—a trait inherited from RMSProp [21] and AdaGrad [22].

Scaled conjugate gradient algorithm

The conjugate gradient algorithm [7] conducts line searches in a conjugate direction and exhibits faster convergence compared to the backpropagation algorithm. The scaled conjugate gradient (SCG), is a general unconstrained optimization technique to efficiently train CNN’s [7]. During training, a direction vector is derived from the gradient g\mathbf{g}, where p\mathbf{p} is updated as p←−g+B1⋅p\mathbf{p} \leftarrow-\mathbf{g} + B_1 \cdot\mathbf{p}. Here, p\mathbf{p} represents vec⁡(P,Poh,Poi)\operatorname{vec}(\mathbf{P},\mathbf{P}_{oh},\mathbf{P}_{oi}), and P\mathbf{P}, Poi\mathbf{P}_{oi}, and Poh\mathbf{P}_{oh} denote the direction vectors. The ratio B1B_1 is determined from the gradient energies of two consecutive iterations, and this direction vector updates all weights simultaneously as w←w+z⋅p\mathbf{w} \leftarrow\mathbf{w} + z \cdot\mathbf{p}. The Conjugate Gradient algorithm avoids Hessian matrix inversions so its computational cost remains at O(w)O(\mathbf{w}), where w\mathbf{w} denotes the size of the weight vector. While heuristic scaling adapts learning factor zz for faster convergence, Output Weight Optimization Backpropagation (OWO-BP) presents a non-heuristic Optimal Learning Factor (OLF) derived from error Taylor’s series as in [21].

Levenberg-Marquardt algorithm

The Levenberg-Marquardt (LM) algorithm [23] merges the speed of steepest descent with Newton’s method’s accuracy. It introduces a damping parameter λ\lambda to modify HLM=H+λ⋅I\mathbf{H}_{LM} = \mathbf{H} + \lambda\cdot\mathbf{I}, addressing potential issues with the Hessian matrix and ensuring nonsingularity. λ\lambda balances first and second-order effects: a small value leans toward Newton’s method, while a larger one resembles steepest descent. However, LM is better suited for smaller datasets due to scalability limitations [23]. [21] discusses LM in detail.

Basic MOLF

In the fundamental MOLF based MLP training process, the input weight matrix, denoted as W\mathbf{W}, is initialized randomly using zero-mean Gaussian random numbers. Initializing the output weight matrix, Wo\mathbf{W}_{o}, involves employing output weight optimization (OWO) techniques [24]. OWO minimizes the Mean Squared Error (MSE) function with regard to Wo\mathbf{W}_{o} by solving a system of MM sets of NuN_u equations in NuN_u unknowns, defined by

C=R⋅WoT(1)\mathbf{C} = \mathbf{R} \cdot\mathbf{W}_{o}^{T} \tag*{(1)}

The cross-correlation matrix, C\mathbf{C}, and the auto-correlation matrix, R\mathbf{R}, are respectively represented as C=1Nv∑p=1NvXapT\mathbf{C} = \frac{1}{N_v} \sum_{p=1}^{N_v} \mathbf{X}_{ap}^{T} and R=1Nv∑p=1NvXap⋅XapT\mathbf{R} = \frac{1}{N_v} \sum_{p=1}^{N_v} \mathbf{X}_{ap} \cdot\mathbf{X}_{ap}^{T}. In MOLF, the pivotal strategy involves utilizing an NhN_h-dimensional learning factor vector, z\mathbf{z}, and solving the equation as :

Hmolf⋅z=gmolf(2)\mathbf{H}_{\mathrm{molf}} \cdot\mathbf{z} = \mathbf{g}_{\mathrm{molf}} \tag*{(2)}

where Hmolf\mathbf{H}_{\mathrm{molf}} and gmolf\mathbf{g}_{\mathrm{molf}} represent the Hessian and negative gradient, respectively, concerning the error and z\mathbf{z}. Detailed algorithmic information is available in [21].

Proposed Work

Exsisting Pieciewise Linear Activations

Piecewise linear functions utilize ReLU activations as their primary components [9]. Activations like sigmoid and Tanh can be effectively approximated using ReLU units. Numerous studies have delved into adaptive PLA functions within MLPs and deep learning contexts [13, 16]. One breakthrough includes hybrid piecewise linear units (PLU) that combines Tanh and ReLU activations into a single function [12]. Figure 2 illustrates how PLU combines these activations. [12] concludes that fixed PLUs outperform ReLU functions due to their representation using a greater number of hinges. However, the fixed PLA function comprises only three linear segments (with hinges H=3H = 3) and remains non-adaptive until the α\alpha parameter is trained in each iteration. As HH remains fixed without significant training, it lacks the capacity for universal approximation [25]. An alternative, known as the piecewise linear activation (PLA), has been demonstrated in [16], specifically designed for deep networks to accommodate trainable PLAs. This method significantly outperforms fixed PLAs, enabling the generation of more complex curves. The adaptive nature of these activations allows for more intricate curve representations.In the cited study [16], an adaptive PLA unit is introduced, where the number of hinges HH is a user-chosen hyperparameter. Optimal results were observed for CIFAR-10 data using H=5H = 5 and H=2H = 2, and for CIFAR-100 data using H=2H = 2 and H=1H = 1 (no activation hinge training). However, the initialization method for these adaptive activations remains unspecified. The equation defining the adaptive activation function is given as: op=max⁡(0,np)+∑s=1Has⋅max⁡(0,−np+bs)o_p = \max(0,n_p) + \sum_{s=1}^{H} a^s \cdot\max(0,-n_p+b^s), where asa^s and bsb^s are learned using gradient descent, with asa^s controlling the slopes of linear segments and bsb^s determining sample point locations. Figure 3 demonstrates an adaptive piecewise linear function with slope aa as 0.2 and bb as 0, while Figure 4 shows a similar function with slope aa as -0.2 and bb as -0.5.

Image not extracted

Figure 2. Fixed Piecewise Linear activations

Image not extracted

Figure 3. 2 ReLU curves

Image not extracted

Figure 4. 4 ReLU curves

As discussion in [16], current literature suffers from a major shortcoming in there approach, due to the assumption of initializing networks with ReLU and its variants. In the present work, we lay out a framework to develop adaptive activation algorithm that uses piecewise ramps 7 to create various activation functions. For clarification, we show the usage of our approach on a sigmoid activation subsection 3.2

Mathematical Background

Adaptive PLA dynamically adjusts the positions and gradients of the hinge points. A network featuring only two hinge points surpasses ReLU activation for specific applications as discussed in the experimental section. Notably, a single hinge suffices on the piecewise linear curve to approximate a linear output, while a more substantial number of hinge sets is necessary to approximate a quadratic output. Hence, for complex datasets, an adequate number of hinges becomes imperative, avoiding the need for additional hidden layers and filters, thereby reducing training times. Our study extends to the application of PLA in CNN’s [26]. Initial findings highlight the capability of PLA to emulate various existing activation functions. For instance, the net function n1n_1 of a CNN filter is n1=t+wi⋅xn_1 = t + \mathbf{w}_i \cdot\mathbf{x}, where tt is the threshold, wi\mathbf{w}_i is the filter weight, and x\mathbf{x} is the input to the net function. The filter can be represented as {wi,t}\{\mathbf{w}_i,t\}. The continuous PLA f(n1)f(n_1) is

f(n1)=∑k=1Nsak⋅r(n1−nsk)(3)f(n_1) = \sum_{k=1}^{N_s} a_k \cdot r(n_1-ns_k) \tag*{(3)}

where NsN_s denotes the number of segments in the piecewise linear curve, r()r() denotes a ramp (ReLU) activation, and nskns_k is the net function value at which the kthk^{th} ramp switches on. Figures 5 and 6 show approximate sigmoid curves generated using ReLU activations where figure 5 has Ns=2N_s = 2 ReLU curves and figure 6 has Ns=4N_s = 4 ReLU curves. Comparing the two figures, we see that larger values of NsN_s lead to better approximation. The contribution of f(n1)f(n_1) to the jthj^{th} net function n2(j)n_2(j) in the following layer is +n2(j)=f(n1)⋅wo(j)+n_2(j) = f(n_1) \cdot w_o(j). Decomposing the PLA into its NsN_s components, we can write

+n2(j)=∑k=1Nsak⋅r(n1−nsk)⋅wo(j)=∑k=1Nswo′(j,k)⋅r(n1(k))(4)+n_2(j) = \sum_{k=1}^{N_s} a_k \cdot r(n_1-ns_k) \cdot w_o(j) = \sum_{k=1}^{N_s} w'_o(j,k) \cdot r(n_1(k)) \tag*{(4)}

where wo′(j,k)w'_o(j,k) is ak⋅wo(j)a_k \cdot w_o(j) and n1(k)n_1(k) is n1−nskn_1-ns_k. A single PLA for filter {wi,t}\{w_i,t\} has now become NsN_s ReLU activations f(n1(k))f(n_1(k)) for NsN_s filters, where each ramp r(n1−dk)r(n_1-d_k), is the activation output of a filter. These NsN_s filters are identical except for their thresholds. Although ReLU activations are efficiently computed, they have the disadvantage that back-propagating through the network activates a ReLU unit only when the net values are positive and zero; this leads to problems such as dead neurons[27], which means if a neuron is not activated initially or during training, it is deactivated. This means it will never turn on, causing gradients to be zero, leading to no training of weights. Such ReLU units are called dying ReLU[28].

Approximate sigmoid using 2 ReLU curves

Figure 5. Approximate sigmoid using 2 ReLU curves

Approximate sigmoid using 4 ReLU curves

Figure 6. Approximate sigmoid using 4 ReLU curves

Although the above method is computationally inexpensive, it has its limitation when encountering minimal distances between the heights of two hinges during training. In the context of the multiple ramps function, each term within the sum in equation 4 encompasses a ramp function accompanied by a coefficient. When the difference between the value of n1n_1 and its corresponding index is non-positive, the ramp function yields 0, thereby nullifying the contribution of that term to the overall sum. Consequently, as clear from equation 4, depending on the values of n1n_1 and nskns_k, it’s possible that the function values may equate to zero. In the current study, we define a highly robust PLA capable of initialization through diverse pre-defined activations enabling differentiability.

Piecewise Linear Activations

Figure 7 illustrates the initialization of a PLA for K hidden units. Figure 7a shows a sigmoid output curve for a given net function, where x axis determines net function npn_p and y axis determines its activation output opo_p which is sigmoid in this case. Any initial activation function can be used for reference. We then define the total number of hinges as HH, which as shown in Figure 7. Each of the hinges can be defined by nskns_k for kthk^{th} hidden unit. These hinges are calculated using evenly spaced values between minimum and maximum value of each net function. These hinges are constant throughout training. Let ss represent the maximum value of a net function, and rr denote its minimum value. Using the reference of the sigmoid activation we define aa as the activation output of the subsequent nsns hinges which can be observed in Figure 7a. These hinges can also be called as multiple ramp. These hinges nsns are constant throughout training and aa are the trainable parameters for each of the kthk^{th} hidden unit.Using sigmoid as reference, we calculate the activation output as shown in Figure 7b, where the activations are computed between two points; for instance, net values between the first two hinges utilize ns(1,k)ns(1,k) as m1m_1 and ns(2,k)ns(2,k) as m2m_2. Similarly, activations between subsequent hinges involve denoting ns(2,k)ns(2,k) as m1m_1 and ns(3,k)ns(3,k) as m2m_2. We perform this process for HH hinges. For each hinge, m1m_1 and m2m_2 are calculated using m1=⌈npδns⌉m_1 = \left\lceil\frac{n_p}{\delta ns} \right\rceil and m2=m1+1m_2 = m_1 + 1. Subsequently, with the net function np(k)n_p(k) given, op(k)o_p(k) is computed as follows:

w1p(k)=ns(m2,k)−np(k)ns(m2,k)−ns(m1,k)(5)w_{1p}(k) = \frac{ns(m_2,k)-n_p(k)}{ns(m_2,k)-ns(m_1,k)} \tag*{(5)}
w2p(k)=np(k)−ns(m1,k)ns(m2,k)−ns(m1,k)(6)w_{2p}(k) = \frac{n_p(k)-ns(m_1,k)}{ns(m_2,k)-ns(m_1,k)} \tag*{(6)}
op(k)={a(H,k)for np(k)>sw1p(k)⋅a(m1,k)+w2p(k)⋅a(m2,k)for s>np(k)>ra(1,k)for np(k)<r(7)o_p(k) = \begin{cases} a(H,k) & \text{for } n_p(k)>s \\ w_{1p}(k)\cdot a(m_1,k)+w_{2p}(k)\cdot a(m_2,k) & \text{for } s>n_p(k)>r \\ a(1,k) & \text{for } n_p(k)<r \end{cases} \tag*{(7)}

where w1p(k)w_{1p}(k) and w2p(k)w_{2p}(k) represent the slope equation for each of the two hinges for the kthk^{\text{th}} hidden unit. np(k)n_p(k) denotes the pthp^{\text{th}} pattern and kthk^{\text{th}} hidden unit net function, while op(k)o_p(k) represents its activation function. For all activation values less than ns(1,k)ns(1,k), the slope is zero, thus np(k)=a(1,k)n_p(k)=a(1,k). Similarly, for all values greater than ns(H,k)ns(H,k), the slope is zero, hence np(k)=a(H,k)n_p(k)=a(H,k). It should be noted that the current approach is activation agnostic, in the sense that any conventional optimization algorithm involving activation’s functions can be replaced with the current adaptive activation paradigm.

The proposed piecewise linear activations AA are trained using the steepest descent method. The negative gradient matrix Ga\mathbf{G}_a with respect to EceE_{ce} is calculated as follows:

ga(k,m)=−∂Ece∂a(k,m)(8)g_a(k,m)=-\frac{\partial E_{ce}}{\partial a(k,m)} \tag*{(8)}

where kk is the hidden unit number and mm is the HthH^{\text{th}} hinge.

ga(k,m)=2Nv∑p=1Nv∑i=1M(tp(i)−yp(i))⋅∂yp(i)∂a(u,m)(9)g_a(k,m)=\frac{2}{N_v}\sum_{p=1}^{N_v}\sum_{i=1}^{M}(t_p(i)-y_p(i))\cdot\frac{\partial y_p(i)}{\partial a(u,m)} \tag*{(9)}
∂yp(i)∂a(u,m)=woh(i,u)⋅∂op(i)∂a(u,m)(10)\frac{\partial y_p(i)}{\partial a(u,m)}=w_{oh}(i,u)\cdot\frac{\partial o_p(i)}{\partial a(u,m)} \tag*{(10)}
∂op(i)∂a(u,m)=woh(i,u)⋅((δ(m−m1)⋅w1(p,u))+(δ(m−m2)⋅w2(p,u)))(11)\frac{\partial o_p(i)}{\partial a(u,m)}=w_{oh}(i,u)\cdot\left((\delta(m-m_1)\cdot w_1(p,u))+(\delta(m-m_2)\cdot w_2(p,u))\right) \tag*{(11)}

where, for the pthp^{th} pattern and kthk^{th} hidden unit’s net value, we determine m1m_1 and m2m_2 based on the pthp^{th} pattern of the kthk^{th} hidden unit’s net value falling between the fixed piecewise linear hinge values m1m_1 and m2m_2 of the uthu^{th} hidden unit, as detailed in the search algorithm provided in the appendix. Subsequently, we obtain w1(p,u)w_1(p,u) and w2(p,u)w_2(p,u) from equations 5 and 6. Utilizing a search algorithm described in [26], we locate the correct hinge mm for a specific pattern’s hidden unit. Equation 11 resolves the pthp^{th} pattern’s uthu^{th} hidden unit of the PLAs, accumulating the gradient for all pthp^{th} patterns of their respective uthu^{th} hidden units. Adam optimizer [19] is used to determine the learning factor and update the activation weights. The weight updates are performed as follows:

A=A+z⋅Ga(12)\mathbf{A} = \mathbf{A} + z \cdot\mathbf{G}_a \tag*{(12)}

Using the gradient Ga\mathbf{G}_a, the optimal learning factor for activations training is calculated as, The activation function vector op\mathbf{o}_p can be related to its gradient as,

op(k)=w1(p,k)⋅[a(k,m1)+z⋅go(k,m1)]+w2(p,k)⋅[a(k,m2)+z⋅go(k,m2)](13)\mathbf{o}_p(k) = w_1(p,k) \cdot[a(k,m_1) + z \cdot g_o(k,m_1)] + w_2(p,k) \cdot[a(k,m_2) + z \cdot g_o(k,m_2)] \tag*{(13)}

The first partial derivative of EE with respect to zz is

∂E∂z=2Nv∑p=1Nv∑i=1M(tp(i)−yp(i))⋅∂yp(i)∂z(14)\frac{\partial E}{\partial z} = \frac{2}{N_v} \sum_{p=1}^{N_v} \sum_{i=1}^{M} (t_p(i) - y_p(i)) \cdot\frac{\partial y_p(i)}{\partial z} \tag*{(14)}

where

∂yp(i)∂z=∑k=1Nhwoh(i,k)⋅((w1(p,k)⋅go(k,m1))+(w2(p,k)⋅go(k,m2)))(15)\frac{\partial y_p(i)}{\partial z} = \sum_{k=1}^{N_h} w_{oh}(i,k) \cdot((w_1(p,k) \cdot g_o(k,m_1)) + (w_2(p,k) \cdot g_o(k,m_2))) \tag*{(15)}

where, m1m_1 and m2m_2 for the pthp^{th} pattern and kthk^{th} hidden unit of the net vector np(k)\mathbf{n}_p(k) is again found, and find go(k,m1)g_o(k,m_1) and go(k,m2)g_o(k,m_2) from the gradient calculated from equations (9, 10, 11). Also, the Gauss-Newton [29] approximation of the second partial is

∂2E(z)∂z2=2Nv∑p=1Nv∑i=1M[∂yp(i)∂z]2(16)\frac{\partial^2 E(z)}{\partial z^2} = \frac{2}{N_v} \sum_{p=1}^{N_v} \sum_{i=1}^{M} \left[\frac{\partial y_p(i)}{\partial z}\right]^2 \tag*{(16)}

Thus, the learning factor is calculated as

z=−∂2E(z)∂z2∂E∂z(17)z = \frac{-\frac{\partial^2 E(z)}{\partial z^2}}{\frac{\partial E}{\partial z}} \tag*{(17)}

After finding the optimal learning factor, the PLAs, A\mathbf{A}, are updated in a given iteration as

A=A+z⋅Ga(18)\mathbf{A} = \mathbf{A} + z \cdot\mathbf{G}_a \tag*{(18)}

where zz is a scalar optimal learning factor and Ga\mathbf{G}_a is the gradient matrix calculated in equation 17 and 8. A pseudo-code for the proposed AdAct algorithm is as follows:

Algorithm 1 AdAct algorithm

  1. Initialize W\mathbf{W}, Woi\mathbf{W}_{oi}, Woh\mathbf{W}_{oh}, NitN_{it}

  2. Initialize Fixed hinges ns\mathbf{n}_s and hinge activation a\mathbf{a}, it←0it \leftarrow0

  3. while it<Nitit < N_{it} do

  4. Find gradient Ga\mathbf{G}_a and gmolfg_{molf} from equations 8 and 2 and solve for zz using OLS.

  5. Calculate gradient and learning factor for activation from equation 8 and equation 17 respectively and update the activations as in equation 18.

  6. OWO step : Solve equation (1) to obtain Wo\mathbf{W}_o

  7. it←it+1it \leftarrow it + 1

  8. end while

Experimental Results

We present the experimental results demonstrating the relative performance of our proposed algorithm across diverse approximation and classification datasets. Moreover, we delve into the comparison of network performance among various methodologies, including AdAct [30], MOLF [31], CG-MLP [21], Scaled conjugate gradient (SCG) [21] and LM [23] across approximation and classification datasets. Additionally, we present results for shallow convolutional neural networks utilizing both custom architectures and deep CNNs employing transfer learning strategies. Additional results are available in the appendix which include approximations of a simple sinusoidal function and a more complex Rosenbrock function. These results yield insights into the behaviour of different activation functions. Specifically, the proposed AdAct showcase consistent outputs for the simple sinusoidal function regardless of the initial activation. Conversely, models utilizing ReLU and Leaky ReLU activations can only accurately approximate the sinusoidal function if more hidden units are incorporated. Augmenting the number of piecewise hinges, enhances the accuracy of the sinusoidal approximation but amplifies computational costs. The computational expense associated with adaptive activation encompasses the aggregate of trainable parameters and the product of hidden units and hinge counts.

Computational Burden

The computational burden is used to measure the time complexity for each algorithm. It indicates number of multipliers that a particular algorithm needs to process per iteration using inputs, hidden units and outputs. We calculate computational burden for the proposed AdAct algorithm along with the comparing algorithms. AdAct algorithm has number of hinges as NhingesN_{hinges}. Updating input weights using Newton’s method or LM, requires a Hessian with NwN_w rows and columns, whereas the Hessian used in the proposed AdAct has only NhN_h rows and columns. The total number of weights in the network is denoted as Nw=M⋅Nu+(N+1)⋅NhN_w = M \cdot N_u + (N + 1) \cdot N_h and Nu=N+Nh+1N_u = N + N_h + 1. The number of multiplies required to solve for output weights using the OLS is given by Mols=Nu(Nu+1)[M+16Nu(2Nu+1)+32]M_{ols} = N_u(N_u + 1)\left[M + \frac{1}{6}N_u(2N_u + 1) + \frac{3}{2}\right]. Therefore, the total number of multiplications per training iteration for LM, SCG, CG, MOLF and AdAct algorithm is given as:

Mlm=[MNu+2Nh(N+1)+M(N+6Nh+4)+MNu(Nu+3Nh(N+1))+4Nh4(N+1)2]+Nw3+Nw2(19)M_{lm} = \left[M N_u + 2N_h(N + 1) + M(N + 6N_h + 4) + M N_u(N_u + 3N_h(N + 1)) + 4N_h^4(N + 1)^2\right] + N_w^3 + N_w^2 \tag*{(19)}
Mcg=Mscg=[MNu+M(N+6Nh+4)+MNu(Nu+3Nh(N+1))+4Nh4(N+1)2]+Nw3+Nw2(20)M_{cg} = M_{scg} = \left[M N_u + M(N + 6N_h + 4) + M N_u(N_u + 3N_h(N + 1)) + 4N_h^4(N + 1)^2\right] + N_w^3 + N_w^2 \tag*{(20)}
Mmolf=Mols+NvNh[2M+N+2+M(Nh+1)2](21)M_{molf} = M_{ols} + N_vN_h\left[2M + N + 2 + \frac{M(N_h + 1)}{2}\right] \tag*{(21)}
MAdAct=Mmolf+Nh∗Nhinges(22)M_{AdAct} = M_{molf} + N_h * N_{hinges} \tag*{(22)}

Note that MAdActM_{AdAct} consists of MmolfM_{molf} plus the required multiplies for the number of hinges for each of the hidden units.

Approximation Datasets Results

DatasetSCG/NhCG-MLP/NhLM/NhMOLF/NhAdAct/Nh 3-hingesAdAct/Nh 5-hingesAdAct/Nh 9-hinges
Oh71.971/301.52/1001.41/301.51/151.49/201.46/201.44/15
White Wine0.6/200.56/1000.57/300.55/300.54/1000.56/200.54/100
twod0.5/300.23/1000.17/150.149/150.149/150.18/150.15/15
Super-conductor230.91/15180.21/100170.2/100144.46/100142.53/100139.62/100142.53/100
F241.14/200.31/1000.30/300.281/1000.282/1000.283/1000.279/100
Concrete61.11/534.64/3032.12/2032.29/10030.70/10034.34/1035.56/100
Weather316.68/15283.23/30286.27/30283.20/10284.39/15284.34/15284.73/15

Table 1. Comparison of 10-fold cross-validation mean square error (MSE) testing results for various approximation datasets using different models and configurations. The best-performing MSE results are highlighted in bold. MOLF is using a Leaky – ReLU as activation function.

We using the following datasets for model evaluation Oh7 [32], White wine [33], twod, Superconductivity dataset [34], F24 data [35], concerete [36], weather dataset [37]. From Table 1, we can observe that AdAct is the top performer in 5 out of the 7 datasets in terms of testing MSE. The following best-performing algorithm is MOLF for weather data with a smaller margin and is also tied in comparison to testing MSE for twod datasets. AdAct has slightly more parameters depending on the number of hinges, but the testing MSE is substantially reduced. The next best performer is LM for the Oh7 dataset. However, LM being a second-order method, its performance comes at a significant cost of computation – almost two orders of magnitude greater than the rest of the models proposed. The table displays the 10-fold cross-validation mean square error (MSE) testing results for various datasets across different models, each denoted by a specific configuration. The values indicate the MSE achieved by different algorithms concerning the number of hidden units (Nh) and hinges used in adaptive activation functions. The models with the best testing MSE are highlighted in bold. Observations reveal that the performance of these algorithms varies across datasets, indicating no single superior algorithm across all scenarios. For instance, for the Oh7 dataset, the LM/Nh configuration performs best, while for White Wine and twod datasets, MOLF configurations with different hinge counts deliver the lowest MSE. Notably, more complex datasets, like F24, demonstrate a pattern where a higher number of hinges yields better performance. This suggests that the optimal choice of algorithm and its configuration heavily depends on the dataset complexity and characteristics, emphasizing the need for adaptability in selecting the suitable model for diverse data scenarios.

Classifier Datasets Results

DatasetSCG/NhCG-MLP/NhLM/NhMOLF/NhAdAct/Nh 3-hingesAdAct/Nh 5-hingesAdAct/Nh 9-hinges
GongTrn10.28/10010.46/1008.94/308.62/308.65/308.64/308.72/30
Comf1815.69/10014.50/10012.63/511.83/2011.93/3011.87/3011.79/30
f17c3.22/1003.69/1003.96/1002.45/1002.38/1002.41/1002.34/100
Speechless44.26/10043.07/10039.72/10036.65/10035.96/10038.94/10037.6/100
Cover27.39/10029.87/100NA20.1/3019.43/3019.47/3019.42/30
Scrap25.58/10020.77/100NA19.9/10019.57/10018.8/10019.2/100

Table 2. 10-fold cross validation Percentage of Error (PE) testing results for classification dataset, (best testing MSE is in bold)

For Classification tasks, we utilized various datasets for evaluating our models, including Gongtrn data [38], Comf18 data [39], Speechless data [39], Cover data [40], Scrap data [39], and f17c [39]. Analyzing Table 2, it’s evident that AdAct emerges as the top performer in 5 out of 6 datasets based on testing MSE. MOLF for weather data also exhibits competitive performance, albeit with a smaller margin. Notably, LM, despite its computational cost, isn’t suitable for pixel-based inputs, limiting its applicability in numerous image classification scenarios.

Shallow CNN results

We conducted experiments using shallow CNNs with ReLU, leaky ReLU, and adaptive activations with one, two, and three VGG blocks [41] on the CIFAR-10 dataset for benchmarking.

Figure 8 illustrates the configuration of the One VGG block, comprising 2 convolution layers with 3x3 kernels, 32 filters in the first and third layers, activation in the second and fourth layers, and a final 2x2 maxpool layer. Similarly, the 2 VGG layer network includes two VGG layers, with the first layer’s configuration matching the one depicted in Figure 8 and the second VGG layer consisting of convolution layers with 3x3 kernels and a depth of 64 for each convolution layer, shown in Figure 9. In the 3 VGG layer configuration, we maintain the setup of the 2 VGG layers described earlier, with an additional 3rd VGG layer following the same structure as the 2nd VGG layer but utilizing 64 filters for each convolution layer, detailed in Figure 9.

Image not extracted

Figure 8. 1 VGG layer with classifier,it consists of 2 convolution layers with 3×33 \times3 kernel and 32 filters in the first and 3rd layers. Activations are in the 2nd and 4th layers. The final layer is a 2×22 \times2 maxpool layer

Image not extracted

Figure 9. Two and Three VGG layer with classifier

An important observation is that while the model uses ReLU activations exclusively for ReLU, and similarly for leaky ReLU, in adaptive activations, only the last VGG layer’s activations are trained, leaving the rest as either ReLU or leaky ReLU. For instance, in the 2-VGG layer setup, the 2nd VGG layer’s adaptive activation functions are trained, while the first VGG layer’s adaptive activations are not treated as trainable parameters. The activation range, denoted by ’n’ and set at 3 samples [minimum value, 0, maximum value], represents the range of output values before the adaptive activation. In practical terms, when training a 2-VGG layer model, activations in the VGG1 layer remain untrained. The output from VGG1, the 2x2 maxpool output, serves as the input to the 2nd VGG layer, where the maximum and minimum values of the first convolution layer’s output in the 2nd VGG layer are utilized. Leaky ReLU is the activation function used for non-trainable adaptive activations, and the initialization of these activations also employs leaky ReLU. The choice of activation type for initialization was based on results showcased in Table 3 for the model featuring leaky ReLU activations.Table 3 shows results for 1-VGG layers, 2-VGG layers, 3-VGG layers model on CIFAR-10[17] dataset with Glorot normal initialization From the table we can observe that as the number of VGG layer increase, adaptive activation gives better accuracy. One thing to note is that in 3-VGG layer model, only the 3rd VGG layer activations are trained, and we can observe a significant difference in the accuracy.

ModelsAdaptiveReLULeakyReLU
1 - VGG layer67.866.5666.45
2 - VGG layers74.271.8273.09
3 - VGG layers75.5372.5873.3

Table 3. 10-fold cross validation accuracy testing results on various activation functions for CIFAR-10 dataset using Glorot as weight-initialization(best testing accuracy is in bold)

Transfer Learning using Deep CNN results

In this section, we explore the utilization of two widely-used pretrained deep learning models, VGG11 and ResNet18, initially trained on the ImageNet dataset. Employing transfer learning techniques, we adapt these models for use with the CIFAR-10 dataset. This adaptation involves replacing the final classification layer with a new linear layer containing ten output classes, aligning with CIFAR-10’s class count. The subsequent fine-tuning process involves a minimum of 100 iterations. Transfer learning capitalizes on the knowledge these pretrained models gained from ImageNet, allowing us to achieve commendable results with reduced data and iterations. Additionally, in models featuring adaptive activations, we implement changes to the final layers. For instance, in the ResNet18 architecture, we substitute ReLU activation functions in the last layer’s basic blocks with adaptive activations. In VGG11, modifications target the activations following the 7th and 8th convolution layers, which also serve as the last two activations in the feature layer, introducing adaptive activations. Adopting adaptive activations in the final feature layer holds two primary benefits: parameter reduction and improved modeling of deeper layer’s complex and abstract features [42]. The results showcased in Table 4 indicate that while adaptive activations lead to better results, there is a slight increase in parameters and training time.

ModelsAdaptive ActivationsReLU Activations
VGG1191.7891.44
ResNet1895.3095.1

Table 4. 10-fold cross validation testing accuracy results for classification dataset (best testing accuracy is in bold)

Conclusion and Future Work

The present study focuses on the significance of activation functions within neural networks and the promising role of AdAct algorithm, an adaptive piecewise linear activation algorithm, in comparison to traditional Rectified Linear Units (ReLUs). The exploration of both deep and shallow CNN architectures emphasizes on the efficacy of adaptive activations, demonstrating superior performance across various datasets and model complexities. Based on the experimental results, the AdAct algorithm is robust and has the ability to approximate complex functions, surpassing the limitations of fixed activation functions. While AdAct entail increased computational demands, the attained performance levels substantiate their value, especially in scenarios requiring accurate approximation of curved outputs. Our experimentation underscores the limitations of fixed activations in capturing arbitrary functions and highlights the adaptability and convergence advantages offered by adaptive activations.

The current study adds a valuable perspective to the ongoing discourse on neural network design, urging researchers and practitioners to consider the nuanced details of activation functions for optimal performance across diverse applications. Utilizing fixed activations to approximate complex curves like sinusoids proves challenging, particularly with activation functions like ReLU and leaky ReLUs, which lack a curve in their function. However, employing adaptive activations enables the model to converge to the desired output curve, irrespective of the initial activation, necessitating a greater number of samples for achieving a smoother curve. These findings underscore the potential of AdAct algorithm and their implications in neural network design, encouraging deeper exploration into their usage and optimization for varied applications.

Training weights by orthogonal least squares

OLS is used to solve for the output weights, pruning of hidden units [43], input units [44] and deciding on the number of hidden units in a deep learner [45]. OLS is a transformation of the set of basis vectors into a set of orthogonal basis vectors thereby measuring the individual contribution to the desired output energy from each basis vector.

In an autoencoder, we are mapping from an (N+1)(N+1) dimensional augmented input vector to it’s reconstruction in the output layer. The output weight matrix Woh∈RN×Nh\mathbf{W}_{oh} \in\mathbb{R}^{N \times N_h} and ypy_p in elements wise will be given as

yp(i)=∑n=1N+1woh(i,n)⋅xp(n)(23)y_p(i) = \sum_{n=1}^{N+1} w_{oh}(i,n) \cdot x_p(n) \tag*{(23)}

To solve for the output weights by regression , we minimize the MSE as in (??). In order to achieve a superior numerical computation, we define the elements of auto correlation R∈RNh×Nh\mathbf{R} \in\mathbb{R}^{N_h \times N_h} and cross correlation matrix C∈RNh×M\mathbf{C} \in\mathbb{R}^{N_h \times M} as follows :

r(n,l)=1Nv∑p=1NvOp(n)⋅Op(l)c(n,i)=1Nv∑p=1NvOp(n)⋅tp(i)(24)r(n,l) = \frac{1}{N_v} \sum_{p=1}^{N_v} O_p(n) \cdot O_p(l) \qquad c(n,i) = \frac{1}{N_v} \sum_{p=1}^{N_v} O_p(n) \cdot t_p(i) \tag*{(24)}

Substituting the value of yp(i)y_p(i) in (??) we get,

E=1Nv∑p=1Nv∑m=1M[tp(m)−∑k=1Nhwoh(i,k)⋅Op(k)]2(25)E = \frac{1}{N_v} \sum_{p=1}^{N_v} \sum_{m=1}^{M} \left[t_p(m) - \sum_{k=1}^{N_h} w_{oh}(i,k) \cdot O_p(k)\right]^2 \tag*{(25)}

Differentiating with respect to Woh\mathbf{W}_{oh} and using (24) we get

∂E∂woh(m,l)=−2[c(l,m)−∑k=1Nh+1woh(m,k)r(k,l)](26)\frac{\partial E}{\partial w_{oh}(m,l)}=-2\left[c(l,m)-\sum_{k=1}^{N_h+1}w_{oh}(m,k)r(k,l)\right] \tag*{(26)}

Equating (26) to zero we obtain a MM set of Nh+1N_h+1 linear equations in Nh+1N_h+1 variables. In a compact form it can be written as

R⋅WT=C(27)\mathbf{R}\cdot\mathbf{W}^{T}=\mathbf{C} \tag*{(27)}

By using orthogonal least square, the solution for computation of weights in (27) will speed up. For convineance, let Nu=Nh+1N_u=N_h+1 and the basis functions be the hidden units output O∈R(Nh+1)×1\mathbf{O}\in\mathbb{R}^{(N_h+1)\times1} augmented with a bias of 1. For an unordered basis function O\mathbf{O} of dimension NuN_u, the mthm^{th} orthonormal basis function O′\mathbf{O}' is defines as « add reference »

Om′=∑k=1mamk⋅Ok(28)O'_m=\sum_{k=1}^{m}a_{mk}\cdot O_k \tag*{(28)}

Here amka_{mk} are the elements of triangular matrix A∈RNu×Nu\mathbf{A}\in\mathbb{R}^{N_u\times N_u}

For m=1m=1

O1′=a11⋅O1a11=1∥O∥=1r(1,1)(29)O'_1=a_{11}\cdot O_1\qquad a_{11}=\frac{1}{\lVert O\rVert}=\frac{1}{r(1,1)} \tag*{(29)}

for 2≤m≤Nu2\leq m\leq N_u, we first obtain

ci=∑q=1iaiq⋅r(q,m)(30)c_i=\sum_{q=1}^{i}a_{iq}\cdot r(q,m) \tag*{(30)}

for 1≤i≤m−11\leq i\leq m-1. Second, we set bm=1b_m=1 and get

bjk=−∑i=km−1ci⋅aik(31)b_{jk}=-\sum_{i=k}^{m-1}c_i\cdot a_{ik} \tag*{(31)}

for 1≤k≤m−11\leq k\leq m-1. Lastly we get the coeffeicent AmkA_{mk} for the triangular matrix A\mathbf{A} as

amk=bk[r(m,m)−∑i=1m−1ci2]2(32)a_{mk}=\frac{b_k}{\left[r(m,m)-\sum_{i=1}^{m-1}c_i^2\right]^2} \tag*{(32)}

Once we have the orthonormal basis functions, the linear mapping weights in the orthonormal system can be found as

w′(i,m)=∑k=1mamkc(i,k)(33)w'(i,m)=\sum_{k=1}^{m}a_{mk}c(i,k) \tag*{(33)}

The orthonormal system’s weights W′\mathbf{W}' can be mapped back to the original system’s weights W\mathbf{W} as

w(i,k)=∑m=kNuamk⋅wo′(i,m)(34)w(i,k)=\sum_{m=k}^{N_u}a_{mk}\cdot w'_o(i,m) \tag*{(34)}

In an orthonormal system, the total training error can be written from (??) as

E=∑i=1M∑p=1Nv[⟨tp(i),tp(i)⟩−∑k=1Nu(w′(i,k))2](35)E=\sum_{i=1}^{M}\sum_{p=1}^{N_v}\left[\langle t_p(i),t_p(i)\rangle-\sum_{k=1}^{N_u}(w'(i,k))^2\right] \tag*{(35)}

Orthogonal least square is equivalent of using the QR decomposition [46] and is useful when equation (27) is ill-conditioned meaning that the determinant of R\mathbf{R} is 0.

Example of piecewise linear activation

To initialize PWL activation, we begin by initializing it with sigmoid[7], ReLU[7] and leaky ReLU[7] activation functions. Subsequently, we determine the total number, HH, of units in the network. This approach can be applied individually to each hidden unit. In this study, an equal number of hinges, denoted as nsn_s, are utilized for each hidden unit. The process involves identifying the minimum and maximum hinge values derived from the network function output. To achieve this, data samples are randomly chosen from each class, and convolution is performed. The resulting convolution output yields the minimum and maximum values. The subsequent section outlines the computation for PWL activations considering k=1k = 1 hidden unit. As previously mentioned, the initial activation must be determined. In this illustration, we utilize the sigmoid activation function, as depicted in Figure 10.

Image not extracted

Figure 10. Sigmoid Curve

The figure 10 illustrates the net function, npn_p, along the x-axis and its corresponding sigmoid activation on the y-axis, with the sigmoid’s range spanning from 0 to 1. The second step involves determining the minimum and maximum values from the convolution output, which, in this case, are selected as -4 and 4, respectively. Subsequently, a user-defined step involves selecting nsn_s samples on the sigmoid curve. For this specific example, we opt for H=7H = 7. These selected hinges and their corresponding activations are presented in Table 5. This table displays a total of H=7H = 7 hinges, ranging from -4 to 4, derived from the minimum and maximum values of the net function, with their associated sigmoid activations denoted as ’a’. The final step involves plotting these chosen points onto the sigmoid curve illustrated in Figure 10. The resulting curve, after incorporating these specified points, should resemble the representation in Figure 11.

H1234567
Fixed hinges (ns1n_{s1})-4-2.67-1.01.2.674
Activations for hinges(a1a_1)0.020.070.210.50.790.940.98

Table 5. PWL samples and activations for one hidden unit

Image not extracted

Figure 11. Sigmoid with Fixed Samples

Figure 11 is the plot for a fixed piecewise sigmoid activation for net versus activations values where 7 hinges are plotted onto the sigmoid curve. For the final piecewise linear curve, we remove the sigmoid curve and linearly join 2 points using the linear interpolation technique.

Linear interpolation involves estimating a new value of a function between two known fixed points [7].

Figure 12 demonstrates the application of linear interpolation between two fixed nsn_s points. Suppose we have a new sample net value, n1(1)n_1(1); its corresponding activation value is depicted in the figure. To determine o1(1)o_1(1), the interpolation between a(1,1)a(1,1) and a(2,1)a(2,1) is calculated using the following equation:

o1(1)=ns(2,1)−n1(1,1)ns(2,1)−ns(1,1)⋅a(1,1)+n1(1)−ns(1,1)ns(2,1)−ns(1,1)⋅a(2,1)(36)o_1(1) = \frac{ns(2,1)-n_1(1,1)}{ns(2,1)-ns(1,1)} \cdot a(1,1) + \frac{n_1(1)-ns(1,1)}{ns(2,1)-ns(1,1)} \cdot a(2,1) \tag*{(36)}

Subsequently, equation (7) is employed to compute all activation outputs. The resulting plot of net versus activation is presented in Figure 7.

Image not extracted

Figure 7. Piecewise Linear Activation Initialization using Sigmoid

Image not extracted

Figure 12. Linear interpolation between 2 points

Adavantages of piecewise linear activation

Now we demonstrate the advantage of adaptive activations using a simple sine data function and a more complicated Rosenbrock function[47]. For each of the experiments, we will compare the approximation results for MOLF algorithm explained in § 2.4 with constant activation functions such as sigmoid, Tanh, ReLU and leaky ReLU and with AdAct algorithm described in [1] with initial activations as sigmoid, Tanh, ReLU and leaky ReLU respectively.

Sinusoidal Approximation

The sine data used for training is generated using a single feature and consists of 5000 uniformly distributed random samples within the range of 0 to 4π4\pi. The resulting output is calculated as the sine function applied to these uniformly distributed random samples. For testing, we randomly select 100 uniformly distributed random samples within the range of 0 to 2π2\pi. Training for both the MOLF and AdAct algorithms involves 100 iterations. Each training algorithm employs 1 hidden unit, except for the AdAct algorithm, which utilizes 20 samples. Additionally, results are presented for the MOLF algorithm using 10 hidden units. In this case, 20 samples are used, as piecewise linear activations (PLAs) would involve 10 smaller ReLU-like functions. Figure 13 displays the prediction of the MOLF model with one hidden unit after 100 iterations, employing the aforementioned fixed activations. The figure illustrates that none of the fixed activations closely approximate the sine function, as labeled the ’original target’ in Figure 13a. Subsequently, when increasing the hidden units to 10, Figure 13b showcases the model’s prediction with ten hidden units after 100 iterations, using the fixed activations mentioned earlier. Notably, curve-based functions like Tanh and sigmoid approximate the sine function, whereas activation functions like ReLU and leaky ReLU, which employ piecewise linear approximations, do not approximate even with more hidden units.

Image not extracted

(a) MOLF with fixed activations with one hidden unit

Image not extracted

(b) MOLF with fixed activations with ten hidden units

Figure 13. MOLF with fixed activations

Figure 14 shows AdAct model prediction with ten hidden units after 100 iterations with initial activations as each of the mentioned fixed activations. From the figure 14, we can observe that the model trained using the adaptive activations approximates the function with similar output no matter what initial activations are used.

Figure 15 shows the hidden units after the model is trained for each fixed initial hidden unit. We can observe that figure 15c and figure 15d are trained with initial activation as ReLU and Leaky ReLU mimics the input versus output, which is sinusoidal which very high activation output but the activation outputs for sigmoid and Tanh are not as high as shown in figure 15a and figure 15b. We also observed that as the activation output is so high, the trained output weights were equally small, with the input weight range being close to similar to the model trained with Tanh’s activation.

Brief Explanation The experimentation above demonstrates

  1. During AdAct algorithm training, using any initial activation can help model converge to a single point. which is evident from Figure 14

Image not extracted

Figure 14. Adaptive activations algorithm with 20 samples

Figure 15. (a) Sigmoid (b) Tanh (c) ReLU (d) Leaky ReLU. Figure 15: Adaptive Activation hidden units

  1. As the model uses piecewise linear samples, to achieve a perfect output curve would need more number of hinges which can be seen in Figure 16

Figure 16. (a) Connected Scatter plot of Adaptive activations algorithm with 20 samples (b) Connected Scatter plot of Adaptive activations algorithm with 30 samples. Figure 16: Connected Scatter plot with Adaptive activations

References

  1. [1]Samer Ellahham. Artificial intelligence: the future for diabetes care. The American journal of medicine, 133(8):895–900, 2020.
  2. [2]Paras Lakhani and Baskaran Sundaram. Deep learning at chest radiography: Automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology, 284(2):574–582, 2017. PMID: 28436741.
  3. [3]Hardik Nahata and Satya P. Singh. Deep Learning Solutions for Skin Cancer Detection and Diagnosis, pages 159–182. Springer International Publishing, Cham, 2020.
  4. [4]German Ignacio Parisi. Human action recognition and assessment via deep neural network self-organization. CoRR, abs/2001.05837, 2020.arxiv.org/abs/2001.05837
  5. [5]Patrik Kamencay, Miroslav Benco, Tomas Mizdos, and Roman Radil. A new method for face recognition using convolutional neural network. Advances in Electrical and Electronic Engineering, 15, 11 2017.
  6. [6]S. Marinai, M. Gori, and G. Soda. Artificial neural networks for document analysis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27:23–35, 2005.
  7. [7]Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006.
  8. [8]Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics, 2011.
  9. [9]Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www. deeplearningbook.org.
  10. [10]Kamel Abdellouahab, Maxime Pelcat, and Francois Berry. Why tanh is a hardware friendly activation function for cnns. In Proceedings of the 11th International Conference on Distributed Smart Cameras, ICDSC 2017, page 199–201, New York, NY, USA, 2017. Association for Computing Machinery.DOI
  11. [11]Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowl.-Based Syst., 6(2):107–116, April 1998.DOI
  12. [12]Andrei Nicolae. PLU: the piecewise linear unit activation function. CoRR, abs/1809.09534, 2018.arxiv.org/abs/1809.09534
  13. [13]Samantha Guarnieri, Francesco Piazza, and Aurelio Uncini. Multilayer feedforward networks with adaptive spline activation function. IEEE transactions on neural networks / a publication of the IEEE Neural Networks Council, 10:672–83, 02 1999.DOI
  14. [14]P. Campolucci, F. Capperelli, S. Guarnieri, F. Piazza, and A. Uncini. Neural networks with adaptive spline activation function. In Proceedings of 8th Mediterranean Electrotechnical Conference on Industrial Applications in Power Systems, Computer Science and Telecommunications (MELECON 96), volume 3, pages 1442–1445 vol.3, 1996.
  15. [15]Ameya Dilip Jagtap, K. Kawaguchi, and G. Karniadakis. Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks. Proceedings of the Royal Society A, 476, 2020.
  16. [16]Forest Agostinelli, M. Hoffman, Peter Sadowski, and P. Baldi. Learning activation functions to improve deep neural networks. CoRR, abs/1412.6830, 2015.DOI
  17. [17]Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
  18. [18]P. Baldi, P. Sadowski, and D. Whiteson. Enhanced higgs boson particle search with deep learning. Physical Review Letters, 114(11), Mar 2015.
  19. [19]Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.arxiv.org/abs/1412.6980
  20. [20]Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient BackProp, pages 9–48. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.
  21. [21]Kanishka Tyagi, Chinmay Rane, and Michael Manry. Supervised learning. In Artificial Intelligence and Machine Learning for EDGE Computing, pages 3–22. Elsevier, 2022.
  22. [22]John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011.
  23. [23]Roberto Battiti. First-and second-order methods for learning: between steepest descent and newton’s method. Neural computation, 4(2):141–166, 1992.DOI
  24. [24]Sanjeev S. Malalur and Michael T. Manry. Multiple optimal learning factors for feed-forward networks. In Defense + Commercial Sensing, 2010.
  25. [25]George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
  26. [26]Chinmay Appa Rane. Multilayer perceptron with adaptive activation function. Masters Thesis, 2016.
  27. [27]Andrew L. Maas. Rectifier nonlinearities improve neural network acoustic models. 2013.
  28. [28]Lu Lu. Dying ReLU and initialization: Theory and numerical examples. Communications in Computational Physics, 28(5):1671–1706, jun 2020.DOI
  29. [29]Adrian J. Shepherd. Second-order methods for neural networks - fast and reliable training methods for multi-layer perceptrons. In Perspectives in neural computing, 1997.
  30. [30]Chinmay Rane, Kanishka Tyagi, Sanjeev Malalur, Yash Shinge, and Michael Manry. Optimal input gain: All you need to supercharge a feed-forward neural network, 2023.
  31. [31]Kanishka Tyagi, Nojun Kwak, and Michael T Manry. Optimal conjugate gradient algorithm for generalization of linear discriminant analysis based on l1 norm. In ICPRAM, pages 207–212, 2014.
  32. [32]Y. Oh, K. Sarabandi, and F.T. Ulaby. An empirical model and an inversion technique for radar scattering from bare soil surfaces. IEEE Transactions on Geoscience and Remote Sensing, 30(2):370–381, 1992.
  33. [33]Almeida F Cortez Paulo, Cerdeira A. Wine quality. UCI Machine Learning Repository, 2009. DOI: https://doi.org/10.24432/C56S3T.
  34. [34]Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science, 154:346–354, 2018.
  35. [35]Michael T Manry, Cheng-Hsiung Hsieh, and Hema Chandrasekaran. Near-optimal flight load synthesis using neural nets. In Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No. 98TH8468), pages 535–544. IEEE, 1999.
  36. [36]I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Cement and Concrete research, 28(12):1797–1808, 1998.
  37. [37]Gautam R. Eapi. Comprehensive neural network forecasting system for ground level ozone in multiple regions, 2015.
  38. [38]Hung-chun Yau and Michael Manry. Iterative improvement of a nearest neighbor classifier. Neural Networks, 4, 09 2000.
  39. [39]Classification data files. IPNN Lab, The University of Texas Arlington. https://ipnnl.uta.edu/training-data-files/classification/, 2002.
  40. [40]Dr. Denis J. Dean Jock A. Blackard and Dr. Charles W. Anderson. UCI machine learning repository, 2000.
  41. [41]Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.DOI
  42. [42]Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. CoRR, abs/1311.2901, 2013.arxiv.org/abs/1311.2901
  43. [43]Kanishka Tyagi, Son Nguyen, Rohit Rawat, and Michael Manry. Second order training and sizing for the multilayer perceptron. Neural Processing Letters, 51(1):963–991, 2020.DOI
  44. [44]Kanishka Tyagi and Michael Manry. Multi-step training of a generalized linear classifier. Neural Processing Letters, 50:1341–1360, 2019.DOI
  45. [45]Kanishka Tyagi. Automated multistep classifier sizing and training for deep learners. PhD thesis, Department of Electrical Engineering, The University of Texas at Arlington, Arlington, TX, 2018.
  46. [46]Gene H. Golub and Charles F. Van Loan. Matrix Computations. The Johns Hopkins University Press, third edition, 1996.
  47. [47]H.H Rosenbrock. An automatic method for finding the greatest or least value of a function. The Computer Journal, pages 175–184, 1960.

Paper details

vlnc:W_8njgrq5r
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents