Activation Function 激勵函數

在類神經網路中，每個節點都需要激活涵數，有很多種，但目的就是為了讓非性線的性質引入網路之中。這個非線性的是怎樣的一個影響，我想我們可以用以下的簡單例子來說明

這是一個簡單的三個輸入的神經網路。三個輸入為x，y，跟一個常數40。權重分別為0.5，3，-1。這樣的網路在不經過Sigmoid的處理為0.5x+3y+40=0，圖上的斜線，這樣的一條直線。經過Sigmoid函數就是圖上的綠線。

這是另一個簡單的三個輸入的神經網路。三個輸入為x，y，跟一個常數4。這樣的網路在不經過Sigmoid的處理為0.5x+3y-4=0，這樣的一條直線。即圖上左上到右下的斜線。經過sigmoid函數即為紅線。

我們可以先看如果不經過sigmoid函數，就是兩條交叉的直線，如上圖。這兩條直線不管權重如何改變，永遠是兩條直線。

現在我們把兩條線的神經網路再連接到一個節點上。就會如上圖，我們給A的神經網路連結權重為5，給B的神經網路連結權重為10。最後會得到5A+10B。如果最後這個節點為輸出節點。我們就可以選擇是否要再經過激勵函數。

眼尖的朋友一定會注意到，其實整個圖還可以簡化為上圖，兩組輸入合為一組即可。這張圖就是一般我們常看到的神經網路圖了。

我們可以看上圖，5A+10B。我們可以看出，兩個激勵函數相加後得到的曲線(最上面像浴缸曲線那條)。這裡我們可以理解到，一個激勵函數可以讓曲線折兩次，所以越多的A網路與B網路，理論上就可以組合出任意曲線才是。沒錯，這就是universal approximation theorem。那麼，如果我們在做曲線的擬合或是線性回歸，輸出就可以不用再一次的激勵函數，又或者找一個等比例型的激勵函數。如果是需要輸出羅輯的判斷，就可以再算一次的sigmoid或是tanh。

既然我們理解到激勵函數可以折彎曲線，那麼可以想像，如果我們的節點多過我們問題本身的需求，就會產生過擬合的問題。為了避免這個問題，就出現了bagging的方法。

另外就是計算cost function的關係，所以激勵函數都需要可微分，我對逆傳遞的方法實在無法理解，大自然裡總有其它的方法可以解決相同的問題，所以出現了NEAT，利用基因演算法來讓類神經網路演化。這樣就不用再算逆傳遞了。

最後從維基百科偷來的別見怪

Name	Plot	Equation	Derivative (with respect to x)	Range	Order of continuity	Monotonic	Derivative Monotonic	Approximates identity near the origin
Identity		${\displaystyle f(x)=x}$	${\displaystyle f'(x)=1}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{\infty }}$	Yes	Yes	Yes
Binary step		${\displaystyle f(x)=\left\{{\begin{array}{rcl}0&{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle f'(x)=\left\{{\begin{array}{rcl}0&{\mbox{for}}&x\neq 0\\?&{\mbox{for}}&x=0\end{array}}\right.}$	${\displaystyle \{0,1\}}$	${\displaystyle C^{-1}}$	Yes	No	No
Logistic (a.k.a. Soft step)		${\displaystyle f(x)={\frac {1}{1+e^{-x}}}}$	${\displaystyle f'(x)=f(x)(1-f(x))}$	${\displaystyle (0,1)}$	${\displaystyle C^{\infty }}$	Yes	No	No
TanH		${\displaystyle f(x)=\tanh(x)={\frac {2}{1+e^{-2x}}}-1}$	${\displaystyle f'(x)=1-f(x)^{2}}$	${\displaystyle (-1,1)}$	${\displaystyle C^{\infty }}$	Yes	No	Yes
ArcTan		${\displaystyle f(x)=\tan ^{-1}(x)}$	${\displaystyle f'(x)={\frac {1}{x^{2}+1}}}$	${\displaystyle \left(-{\frac {\pi }{2}},{\frac {\pi }{2}}\right)}$	${\displaystyle C^{\infty }}$	Yes	No	Yes
Softsign [7][8]		${\displaystyle f(x)={\frac {x}{1+|x|}}}$	${\displaystyle f'(x)={\frac {1}{(1+|x|)^{2}}}}$	${\displaystyle (-1,1)}$	${\displaystyle C^{1}}$	Yes	No	Yes
Rectified linear unit (ReLU)[9]		${\displaystyle f(x)=\left\{{\begin{array}{rcl}0&{\mbox{for}}&x<0\\x&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle f'(x)=\left\{{\begin{array}{rcl}0&{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle [0,\infty )}$	${\displaystyle C^{0}}$	Yes	Yes	No
Leaky rectified linear unit (Leaky ReLU)[10]		${\displaystyle f(x)=\left\{{\begin{array}{rcl}0.01x&{\mbox{for}}&x<0\\x&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle f'(x)=\left\{{\begin{array}{rcl}0.01&{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	Yes	Yes	No
Parameteric rectified linear unit (PReLU)[11]		${\displaystyle f(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha x&{\mbox{for}}&x<0\\x&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle f'(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha &{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	Yes iff ${\displaystyle \alpha \geq 0}$	Yes	Yes iff ${\displaystyle \alpha =1}$
Randomized leaky rectified linear unit (RReLU)[12]		${\displaystyle f(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha x&{\mbox{for}}&x<0\\x&{\mbox{for}}&x\geq 0\end{array}}\right.}$[1]	${\displaystyle f'(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha &{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	Yes	Yes	No
Exponential linear unit (ELU)[13]		${\displaystyle f(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha (e^{x}-1)&{\mbox{for}}&x<0\\x&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle f'(\alpha ,x)=\left\{{\begin{array}{rcl}f(\alpha ,x)+\alpha &{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle (-\alpha ,\infty )}$	${\displaystyle C^{1}}$ when ${\displaystyle \alpha =1}$, otherwise ${\displaystyle C^{0}}$	Yes iff ${\displaystyle \alpha \geq 0}$	Yes iff ${\displaystyle 0\leq \alpha \leq 1}$	Yes iff ${\displaystyle \alpha =1}$
Scaled exponential linear unit (SELU)[14]		${\displaystyle f(\alpha ,x)=\lambda \left\{{\begin{array}{rcl}\alpha (e^{x}-1)&{\mbox{for}}&x<0\\x&{\mbox{for}}&x\geq 0\end{array}}\right.}$ with ${\displaystyle \lambda =1.0507}$ and ${\displaystyle \alpha =1.67326}$	${\displaystyle f'(\alpha ,x)=\lambda \left\{{\begin{array}{rcl}f(\alpha ,x)+\alpha &{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.}$	${\displaystyle (-\lambda \alpha ,\infty )}$	${\displaystyle C^{0}}$	Yes	No	No
S-shaped rectified linear activation unit (SReLU)[15]		${\displaystyle f_{t_{l},a_{l},t_{r},a_{r}}(x)=\left\{{\begin{array}{rcl}t_{l}+a_{l}(x-t_{l})&{\mbox{for}}&x\leq t_{l}\\x&{\mbox{for}}&t_{l}<x<t_{r}\\t_{r}+a_{r}(x-t_{r})&{\mbox{for}}&x\geq t_{r}\end{array}}\right.}$ ${\displaystyle t_{l},a_{l},t_{r},a_{r}}$ are parameters.	${\displaystyle f'_{t_{l},a_{l},t_{r},a_{r}}(x)=\left\{{\begin{array}{rcl}a_{l}&{\mbox{for}}&x\leq t_{l}\\1&{\mbox{for}}&t_{l}<x<t_{r}\\a_{r}&{\mbox{for}}&x\geq t_{r}\end{array}}\right.}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	No	No	No
Adaptive piecewise linear (APL) [16]		${\displaystyle f(x)=\max(0,x)+\sum _{s=1}^{S}a_{i}^{s}\max(0,-x+b_{i}^{s})}$	${\displaystyle f'(x)=H(x)-\sum _{s=1}^{S}a_{i}^{s}H(-x+b_{i}^{s})}$[2]	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{0}}$	No	No	No
SoftPlus[17]		${\displaystyle f(x)=\ln(1+e^{x})}$	${\displaystyle f'(x)={\frac {1}{1+e^{-x}}}}$	${\displaystyle (0,\infty )}$	${\displaystyle C^{\infty }}$	Yes	Yes	No
Bent identity		${\displaystyle f(x)={\frac {{\sqrt {x^{2}+1}}-1}{2}}+x}$	${\displaystyle f'(x)={\frac {x}{2{\sqrt {x^{2}+1}}}}+1}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{\infty }}$	Yes	Yes	Yes
SoftExponential [18]		${\displaystyle f(\alpha ,x)=\left\{{\begin{array}{rcl}-{\frac {\ln(1-\alpha (x+\alpha ))}{\alpha }}&{\mbox{for}}&\alpha <0\\x&{\mbox{for}}&\alpha =0\\{\frac {e^{\alpha x}-1}{\alpha }}+\alpha &{\mbox{for}}&\alpha >0\end{array}}\right.}$	${\displaystyle f'(\alpha ,x)=\left\{{\begin{array}{rcl}{\frac {1}{1-\alpha (\alpha +x)}}&{\mbox{for}}&\alpha <0\\e^{\alpha x}&{\mbox{for}}&\alpha \geq 0\end{array}}\right.}$	${\displaystyle (-\infty ,\infty )}$	${\displaystyle C^{\infty }}$	Yes	Yes	Yes iff ${\displaystyle \alpha =0}$
Sinusoid		${\displaystyle f(x)=\sin(x)}$	${\displaystyle f'(x)=\cos(x)}$	${\displaystyle [-1,1]}$	${\displaystyle C^{\infty }}$	No	No	Yes
Sinc		${\displaystyle f(x)=\left\{{\begin{array}{rcl}1&{\mbox{for}}&x=0\\{\frac {\sin(x)}{x}}&{\mbox{for}}&x\neq 0\end{array}}\right.}$	${\displaystyle f'(x)=\left\{{\begin{array}{rcl}0&{\mbox{for}}&x=0\\{\frac {\cos(x)}{x}}-{\frac {\sin(x)}{x^{2}}}&{\mbox{for}}&x\neq 0\end{array}}\right.}$	${\displaystyle [\approx -.217234,1]}$	${\displaystyle C^{\infty }}$	No	No	No
Gaussian		${\displaystyle f(x)=e^{-x^{2}}}$	${\displaystyle f'(x)=-2xe^{-x^{2}}}$	${\displaystyle (0,1]}$	${\displaystyle C^{\infty }}$	No	No	No