SQNL

---

In the context of Artificial Neural Networks, the Square Nonlinearity (SQNL) is an activation function defined mathematically as:

$f(x)={\begin{cases}1&:x>2.0\\x-{\frac {x^{2}}{4}}&:0\leq x\leq 2.0\\x+{\frac {x^{2}}{4}}&:-2.0\leq x<0\\-1&:x<-2.0\end{cases}}$

$f'(x)=1\mp {\frac {x}{2}}$

Plot of the Tanh (blue), Softsign (red) and SQNL (yellow) functions

Where x is the input to a neuron/ is the weighted sum. This activation function is similar in shape to the popular Tanh and the computationally efficient softsign/elliotsig.. The mathematical proofs and experimental analysis to justify the usability and generalizability of the SQNL function has been shown in the original paper

Properties

• SQNL exhibits a quadratic nonlinearity.
• The function is continuous (−∞ − +∞), bounded outputs (range = ±2.0) and non-constant in nature
• The SQNL activation function is easily differentiable (fB (x)) and symmetrical in nature which has an advantage in learning as described by the universal approximation theorem
• A smooth transition between the input and output is demonstrated which thereby improves the neural response

Other Commonly used activation functions

• Tanh
• Hard Tanh
• Softsign
• ISRU

Advantages

• The quadratic nonlinearity exhibit by SQNL makes it desirable because it uses a square operation instead of an exponent.
• The SQNL derivative requires one difference and one multiplication by 0.5. Both of these are single cycle operations and hence have a computational advantage over the derivative of the TanSig function given by (1 − a)(1 + a) where a = TanSig(x)
• From a hardware perspective, the SQNL function can be implemented using Look-Up Table (LUT) or direct implementation using multipliers depending on the speed or available resources.

Computational Speed on ARM M3 processor

Comparing SQNL function with morphologically similar functions the following are the speedup exhibited

• SQNL is on an average 12x faster than popular Tanh function
• SQNL is 9x faster than the ISRU function
• SQNL is 8x faster than SoftSig function.

Applications and General Usage

SQNL finds application in various machine learning architectures which include but not limited to Multilayer perceptron (Single Layer Feed Forward Networks), LSTM and so on. The computational simplicity of this function is making it gain acceptance in the deep learning community. It was applied to the Goldilocks Neural Networks architectures Support Vector machine Smooth Activation functions SQ-RBF Mish

Example

Computation of SQNL using Python code:

>>> import numpy as np
>>> from scipy import arange
>>> def sqnl(x):
      
            if x > 2.0:
               y = 1.0
            elif x < -2.0:
               y = -1.0
            elif x <= 2.0 and x >= 0:
               y = x - (x*x)/4.0
            else:
               y = x + (x*x)/4.0
            
            return y

>>> sqnl = np.vectorize(sqnl)
>>> x = np.arange(-3.0,3.0,0.1)
>>> sqnl(x)

Here is an example of MATLAB code: The full implementation of SQNL as a toolbox in MATLAB is available on this project's repository "SQNL_MATLAB". Retrieved 2019-09-28.

function a = sqnl(n)
a = zeros(size(n));

idx = find(n > 2);
n(idx) = 2;

idx = find(n < -2);
n(idx) = -2;

idx = find(n >= 0);
a(idx) = n(idx) - n(idx).^2/4;

idx = find(n < 0);
a(idx) = n(idx) + n(idx).^2/4;
end

References

This article "SQNL" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:SQNL. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.

📰 Article(s) of the same category(ies)

• Upsilon function
• Modified hyperbolic tangent
• Causal Neural Paradox (Thought Curvature)
• Norse (neuron simulator)
• Seq2SeqSharp
• Swirl function
• 2 × 2 real matrices

---