rheddlZddlmZddlZddlmZmZddlmZeje Z GddejZ Gdd ejZ Gd d ejZGd d ejZGddejZGddejZGddejZGddejZGddejZGddejZGddejZGddeZideded d!d"fd#ed$e d%ed&d'ifd(e d)ed*ed+ej0d,ed-ed.ed/ej2d0ed1ej4d2ej6d3ej8ej8ej:ej<d4ZeeZ d5Z!e!d%Z"e!d$Z#e!dZ$e!d#Z%e!d.Z&e!d3Z'e!d-Z(e!d,Z)y)6N) OrderedDict)Tensornn)loggingc eZdZdZdedefdZy)PytorchGELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. inputreturncDtjj|dS)Ntanh) approximate)r functionalgeluselfr s k/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/activations.pyforwardzPytorchGELUTanh.forward$s}}!!%V!<<N__name__ __module__ __qualname____doc__rrrrr r s=V==rr c eZdZdZdedefdZy)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 r r c d|zdtjtjdtjz |dtj |dzzzzzS)N??@Hm?g@)torchr mathsqrtpipowrs rrzNewGELUActivation.forward.sPU{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrNrrrrrr(s wVwwrrcJeZdZdZddeffd ZdedefdZdedefdZxZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 use_gelu_pythonct||r|j|_ytj j |_yN)super__init__ _gelu_pythonactrrr)rr* __class__s rr.zGELUActivation.__init__:s/  ((DH}}))DHrr r cj|dzdtj|tjdz zzS)Nrr r!)r#erfr$r%rs rr/zGELUActivation._gelu_pythonAs,s{cEIIediin.D$EEFFrc$|j|Sr,r0rs rrzGELUActivation.forwardDxxr)F) rrrrboolr.rr/r __classcell__r1s@rr)r)2s=**G&GVGVrr)c eZdZdZdedefdZy)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs r r c\d|zdtj|dzdd|z|zzzzzS)Nrr g3E?r")r#r rs rrzFastGELUActivation.forwardMs:U{cEJJu|/CsXX]M]`eMeGe/f$gghhrNrrrrr;r;HsiViirr;c eZdZdZdedefdZy)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs r r c8|tjd|zzS)NgZd;?)r#sigmoidrs rrzQuickGELUActivation.forwardVsu}}UU]333rNrrrrr>r>Qs4V44rr>c<eZdZdZdedeffd ZdedefdZxZS)ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxcl||kDrtd|d|dt| ||_||_y)Nzmin should be < max (got min: z, max: )) ValueErrorr-r.rCrD)rrCrDr1s rr.zClippedGELUActivation.__init__gs> 9=cU'#aPQ Q rxr cjtjt||j|jSr,)r#cliprrCrD)rrHs rrzClippedGELUActivation.forwardos!zz$q'488TXX66r) rrrrfloatr.rrr8r9s@rrBrBZs. E77F7rrBc2eZdZdZfdZdedefdZxZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cxt|tjdtjz |_y)N)r-r.r$r%r&precomputed_constantrr1s rr.zAccurateGELUActivation.__init__{s' $(IIa$''k$:!rr r c d|zdtj|j|dtj|dzzzzzS)Nrrr")r#r rPr'rs rrzAccurateGELUActivation.forwardsEU{a%**T-F-F%RZ]b]f]fglno]pRpJp-q"rrssr)rrrrr.rrr8r9s@rrMrMss#;tVttrrMcBeZdZdZfdZdedefdZdedefdZxZS)MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish c`t|tjj|_yr,)r-r.rrmishr0rQs rr.zMishActivation.__init__s ==%%rr r cl|tjtjj |zSr,)r#r rrsoftplusrs r _mish_pythonzMishActivation._mish_pythons%uzz"--"8"8"?@@@rc$|j|Sr,r5rs rrzMishActivation.forwardr6r) rrrrr.rrZrr8r9s@rrUrUs6 &A&AVAVrrUc eZdZdZdedefdZy)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. r r c|Sr,rrs rrzLinearActivation.forwards rNrrrrr]r]sVrr]ceZdZdZddZy)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability c||z j|tjdz}ddtj|zzS)Nr!rr )divr$r%r#r3)rr musigmas rrzLaplaceActivation.forwards<  3!78cEIIe,,--rN)g۞?g ^/ ?rrrrrrrrr`r`s .rr`ceZdZdZdZy)ReLUSquaredActivationz` Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668v2 cntjj|}tj|}|Sr,)rrrelur#square)rr relu_appliedsquareds rrzReLUSquaredActivation.forwards)}}))%0 ,,|,rNrerrrrgrgs rrgceZdZfdZxZS)ClassInstantiercdt||}t|tr|n|if\}}|di|S)Nr)r- __getitem__ isinstancetuple)rkeycontentclskwargsr1s rrpzClassInstantier.__getitem__s6'%c*!+GU!;g'2 V}V}r)rrrrpr8r9s@rrnrns rrnrgelu_10i )rCrD gelu_fastgelu_new gelu_pythonr*Tgelu_pytorch_tanh gelu_accuratelaplace leaky_relulinearrW quick_gelurirelu2relu6r@silu)swishr preluc ||tvr t|Std|dttj)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationrsBF"'((#4"55RSWX^XcXcXeSfRghiir)*r$ collectionsrr#rrutilsr get_loggerrloggerModuler rr)r;r>rBrMrUr]r`rgrn LeakyReLUReLUReLU6SigmoidSiLUTanhPReLUACT2CLSrrr{rzrryrrrW linear_actrrrrsl #    H % =bii =w wRYY,ii4"))47BII72 tRYY t RYY"ryy . .BIIk  N  %s2'>? # !   N%6$=>     +   ",,    N %  BGG  "  RXX  rzz! " BGG# $WW GG XX) ,  !j]+ * %f ; ' L ) ff H % r