rh*dZddlmZddlmZmZddlZddlmcm Z ddl ZddlmZddl m Z ddlmZdd lmZdd lmZmZmZmZd d lmZerdd lmZddlmZmZddlmZdZn4ddl m!Z!ddlm"Z"m#Z#d dlm$Z$e%ejLejLejLfZ'e(e)e'fZdZd`dejLdeee*ejLfdejLfdZ dadejLdejLdejLdejLdejLdejLd ejLd!ejLd"ejLd#ejLd$ee*d%e)d&e)de%ejLejLejLffd'Z+ dbd)ejLdejLdejLdejLdejLd ejLdejLdejLd$e*d%e)d&e)d*e*de%ejLejLejLffd+Z, dcd,ejLd-ejLd.ejLd/ejLd0ejLd1ejLd2ejLd3ejLd$ee*d4e-d5e-d%e)d*e*de%ejLejLejLee%ejLejLejLfee%ejLejLejLfffd6Z. ddd,ejLd-ejLd.ejLd/ejLd0ejLd7ejLd8ejLd9ejLd4e-d*e*d%e)deejLe%ejLe%ejLejLejLffffd:Z/d(ej`fd,ejLd-ejLd.ejLd/ejLd0ejLd1ejLd2ejLd3ejLd*e*d;ejbde%ejLe%ejLejLejLfffd<Z2ddddd(ej`fd,ejLd-ejLd.ejLd/ejLd0ejLd7ejLd8ejLd9ejLd4e-d*e*d;ejbde%ejLejLejLee%ejLejLejLfee%ejLejLejLfffd=Z3ddddd(ejhdfd>e"d,ejLd-ejLd.ejLd0ejLd/ejLd7ejLd8ejLd9ejLd4e-d*e*d?ejbd%e)deejLe%ejLe%ejLejLejLffffd@Z5ddddd(ejhddfd>e"dAe"dBe"d,ejLd-ejLd.ejLd0ejLd/ejLd7ejLd8ejLd9ejLd4e-d*e*d?ejbd%e)dCe-deejLe%ejLe%ejLejLejLffff"dDZ6GdEdFejnZ8GdGdHejnZGdIdJejnZ9GdKdLejnZ:GdMdNejnZ;GdOdPejnZdQZGdUdVZ?eeGdWdXeZ@eGdYdZe>ZAeeGd[d\eZBeGd]d^e>eZCgd_ZDy)ezPyTorch xLSTM Model.) dataclass)OptionalUnionN)nn)CrossEntropyLoss)GenerationMixin)PreTrainedModel) ModelOutputauto_docstringcan_return_tupleis_xlstm_available) xLSTMConfig) mLSTMBlock)mLSTMStateTypesoft_cap) xLSTMRMSNormT)partial)CallableLiteral)round_up_to_next_multiple_ofFvalues cap_valuereturnc@||S|tj||z zS)a Soft caps a tensor to a value. Performs a tanh operation on the logits and scales the result to the cap value. Common technique in attention and output language heads to prevent large logits from dominating the softmax. See for example Gemma2: https://arxiv.org/abs/2408.00118 Args: values: The tensor to cap. cap_value: The value to cap the values to. If None, no cap is applied. Returns: The capped values. )torchtanh)rrs {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/xlstm/modeling_xlstm.pyrr0s(  M5::fy&8999@matKmatVvecBvecI matC_states vecN_statesscaMinter_states matC_initial vecN_initialscaMinter_initialqk_scale chunk_size num_chunksc g|j|jd\} }}}}| }|j|j}}| |dz} |"tj| ||dz|z|f||}|!tj| ||dz|zf||}|tj| ||dzf||}|tj| |||f||n|}|tj| ||f||n|}| tj| |df||n| }|d|z |z}|d}|j dj }|jd}td| D]Q}||dddd||z|dz|zddf<||dddd||z|dz|zf<||dddd|f<|dddd|f}|dddd|f}tj ||z|}|dddd|| z|dz| zddf}|dddd|| z|dz| zddf} |dddd|ddf}!tj|!|dz dddddddf}"||"z}#tj||z|z dddddf}$|$d|z|#jd d| zz}%|$|z|#jd djdz}&|}|%}|&}T||dddd| dddf<||dddd| df<||dddddf<|||fS) Nrdtypedevice).r0N).r0r).N) shaper3r4rzerosmaxrsqueezerangeexp transposesum)'r"r#r$r%r&r'r(r)r*r+r,r-r. batch_sizenh_dhqkdhhvnc_dtype_devicematC_kvecN_k scaM_inter_kvecAscaGscaA_maxkey scaA_max_kscaG_kscaM_inter_k_next matK_chunk matV_chunkvecA_k vecAbar_kmatK_chunk_gated scaGbar_k matC_k_next vecN_k_nexts' rmlstm_chunkwise_recurrent_fw_CrXCs)D(CTZZ^(C% B4 **dkk  TzH  ++z2Q$&MU[dklK  ++z2Q$&Gv^efK  #${{JR!V+EV\cd  # KKRt4F7 S  R^QeEKKR.fW Mkw  !( KKR+6' J"  M"T)D0G}88B<&&#++B/ J' !CCIK1cDjC!Gt+;;Q> ??EK1cDjC!Gt+;;; <*6 Q3Y '"!Q),J!Q)_F % &<*? L aC*$4a:7M$MqPQJaC*$4a:7M$MqPQJ!QQ,'F &+matQepsc H|j} | | } } |j\}}}}|j||| || z|}|j||| || z}|}|j||| | |}|j||| | |}|j||| | |}tjtj | | ftj | }|dddddddddf|dddddddddfz }tj||td }||dddddddddfz}tj|ddj}||dddddddfz}tj||}|dddddddddf}|dddddddddf}||z }tj|}||jddz|z}||z}tj||z } || z|z}!|!|z||zz}"|!|jdz|jddz}#tjtj |#tj| }$|"|$| zz }%|%j||| | z|}&|$j#||| | z}'|||| | z}(|&|'|(fS)Nr2infr0Fdimkeepdimr5T)r4r6viewrtrilonesboolwherefloatr8rmaximumr;r< unsqueezer=absreshape))rZr"r#r&r'r(r%r$r,r-r.r[rErCr>r?dqkdhv matC_k_states vecN_k_statesscaMinter_k_statesltrmatF_logsig_chunkmatF_logsig_mask_chunk matLogD_chunk vecMintra_k vecM_b_intervecM_k_combinematLogD_stabilized_chunk matD_chunk matS_chunk matM_chunkvecBbarmatQ_chunk_gatedmatNumerator_commonvecDenom_l_commonvecDenom_max_common matH_k_chunkmatH_outvecN_outvecM_outs) rmlstm_chunkwise_parallel_fw_Hrs++#ZJ#.#4#4 BS#((RSBYL #((RSBYG -yyRZ=yyRZ=yyRZ=jj JJZ(jj  !Aq!T!12T!Q4:J5KK!&S2CeEl]!S.aAtQ6F1GG ii 2uELL 0Aq$?? |[A'1aD(89#Aq!Q$45 #0>#A YY78 T^^B33x? *, ))L>9:'>H4.>dARR,}/F/Fr/JJZ^^`blp^Mqq#mmEII6G,H%))UcTcJde*.AC.GH $$ZR*_cJ'..z2rJO!*b"z/B8++r queryrLvalueigatefgatecstatenstatemstatereturn_last_statesreturn_all_statesc d|j\} }}}|| zdk7rtd|d| d|| z}|j| ||| }|j| ||| }|j|}|j d}||dz}t ||||||||| | \}}}t ||||ddddd| ddf|ddddd| f|ddddddf|||| ||  \}}}|||f}| r3||dddd| dddf|dddd| df|ddddddfffz }n|d z }| r ||||ffz }|S|d z }|S) NrSequence length is not divisible by chunk size .r0r1 r"r#r$r%r)r*r+r,r-r. rZr"r#r&r'r(r%r$r,r-r.r[N)r6 ValueErrorra logsigmoidcumsumrXr)rrLrrrrrrr,rrr-r[r>r?sequence_lengthrArCr%vecF vecF_logsigr$rmrnrorrr ret_tuples rmlstm_chunkwise_fwrs*16 - B Z '1 ,//@@`ak`llmno o  *zz*b"j9zz*b"j9&&t, !!"%  TzHr?rrArCr%rrr$r,rmrnrorrr last_statess rmlstm_chunkwise_native_autogradr;s16 - B Z '1 ,//@@`ak`llmno o  *zz*b"j9zz*b"j9ll4( !!"%:r?rAr@rBscaF_logscaM_state_newscaF_actscaI_act vecQ_scaledmatC_state_newvecN_state_newh_num qn_dotproductmax_valh_denomhs rmlstm_recurrent_step_nativer|sGKK 99;9/99;9/99;9/${{ B[[ 1d ;;#)) #EF F >>j"dD9 9=hnn=MNO O >>j"d3 3=hnn=MNO O >>j"a0 0=hnn=MNO O ;;:r1- -9%++GH H ;;:r1- -9%++GH H88&&11%88h#6>99X0>AB99U^34t~. !!Q4-08;hq!QPT}>U 1a q!T1}!5 5?  "H,x#~=Aq$M*^->->Y->-OO a ##+#6#Aq$M2^Aq!TM5R5U5U\e5U5ff %--a0 ))^O,==!2!2!4g>DHH{H[ GO DDyD !'***='***='***=>>>BBBr c  |j\} } }}|jd}|j}|U|| td|| td|j| |j| |j| }}}nRt j | | ||f| |}t j | | |f| |}t j | | df| |}g}t |D]w}|dddd|df|dddd|df}}|dddd|ddf|dddd|ddf|dddd|ddf}}}td ||||||||| | d | \}\}}}|j|yt j|d}|r||||ffS|S) Nr0z)Initial states must be provided together.rr2r) rrrrrLrrrr[rr5r_) r6r4rrrr7r:rappendstack)rrLrrrrrrrr[rrr>r?rrArlr4 matC_state vecN_state vecM_state vecH_listtvecF_tvecI_tvecQ_tvecK_tvecV_tvecHmatHs rmlstm_recurrent_sequence_nativers(16 - Bkk"o   I$5 !LMM I$5 !LMM ; / ; / ; /%/ Jj"dC%@ \bcJj"d%;;W]^Jj"a%8 TZ[J ' #A"1aD=15Aq$3GFF&+1aA:%6Aq!QJqRSUVXYzIZFFF:U :!!!' : : 6D6:z:   T "+ #.{{9"- *j*== =Kr mlstm_chunkwise_kernelautocast_kernel_dtypec  | r tdd|j\}}}}|}|| zdk7r|| zdz | z| z}|j||||jd}|j||||jd}|j||||jd}|j|||}|j|||}||ddddd|ddf<||ddddd|ddf<||ddddd|ddf<||ddddd|f<||ddddd|f<n |}|}|}|}|}|d||||||||| | | | d | }|ddddd|ddf}|S)Nz6We are padding zeros, so we cannot return last states,z*as they would be not the true last states.rrr) rrLrrrrrrrr[rr-r)rr6 new_zeros)rrrLrrrrrrrr[rr-rr>r?rrA S_unpaddedS_paddedq_padk_padv_padi_padf_padrs rwrap_chunkwise_pad_zerosrs H<  16 - B$ Z '1 ,(:59jHJVHOOJHekk!nMEMM*b(CIIaLIEOOJHekk!nMEOOJH=EOOJH=E*/E!Q  Q& '*-E!Q  Q& '*/E!Q  Q& '',E!Q  # $',E!Q  # $EEEEE% 1"7!  Aq+:+q() r mlstm_sequence_kernelmlstm_step_kernelenable_loggingcv|j\}}}}|jd}||n2tj|||||jtj}| | n1tj||||jtj}| | n1tj||d|jtj}|dkDr~g}d}||z }||z}|dkDr||z}||z}||d||ddfj |d||ddfj |d||ddfj |d||fj |d||fj ||||d| |  \}\}}}||z }|j |||z }|dkDr||d||ddfj |d||ddfj |d||ddfj |d||fj |d||fj |||d| \}\}}}|j |tj|d }nh|dk7rtd |d ||jd |jd |jd ||||||  \}\}}}|dddddddf}| r||||ffS|S)af This function computes the last hidden state and matH outputs of the mLSTM, independently of the sequence length. For this it uses three kernels: - mlstm_chunkwise_kernel: mlstm chunkwise kernels that processes chunks of a given chunk size in parallel. - mlstm_sequence_kernel: mlstm kernel that processes the remaining sequence length in a single step recurrence. - mlstm_step_kernel: mlstm kernel that processes a sequence length of 1 in a single step. It tries to maximize the chunksizes to improve performance. It will start with the given chunk size and then divides the chunksize by 2 until the chunk size is smaller than 16. At every chunksize it will process the maximal number of chunks that fit into the remaining sequence length. E.g. for chunk_size = 64, this function will try the chunksizes [64, 32, 16] if necessary. For the remaining sequence length, which is smaller than 16, we use a different kernel that computes the mLSTM in a single step and loop over this in pytorch. Args: mlstm_chunkwise_kernel: The mLSTM chunkwise kernel that processes chunks of a given chunk size in parallel mlstm_sequence_kernel: The mLSTM kernel that processes the remaining sequence length in a single step recurrence query: The query tensor (batch_size, nh, sequence_length, dhqk) key: The key tensor (batch_size, nh, sequence_length, dhqk) value: The value tensor (batch_size, nh, sequence_length, dhhv) fgate: The forget gate tensor (batch_size, nh, sequence_length) igate: The input gate tensor (batch_size, nh, sequence_length) c_initial: The initial cell state tensor (batch_size, nh, dhqk, dhhv) n_initial: The initial hidden state tensor (batch_size, nh, dhqk) m_initial: The initial memory state tensor (batch_size, nh, 1) return_last_states: If True, the function will return the last states of the mLSTM eps: The epsilon value used for numerical stability autocast_kernel_dtype: The dtype used for the kernel computation chunk_size: The chunk size used for the chunkwise kernel enable_logging: If True, the function will log debug information. Default is False. Returns: The last hidden state tensor (batch_size, nh, sequence_length, dhhv) or a tuple containing the last hidden state tensor and the last states of the mLSTM Last states are (cstate (batch_size, nh, dhqk, dhhv), nstate (batch_size, nh, dhqk), mstate (batch_size, nh, 1)). r0Nr4r3rr.T) rrLrrrrrrr-rrr[) rrLrrrrrrrr[rrz)Received empty sequence (sequence_length=z3), require at least single element in the sequence.) rrLrrrrrrr[) r6rr7r4float32 contiguousr concatenaterr9)rrrrrLrrrrrrrr[rr-rr>r?rrArBc_staten_statem_stateh_outsseq_len_start_idxremaining_seq_lenr. iter_seq_len seq_len_idxh_outs r(wrap_chunkwise_arbitrary_sequence_lengthrBsbp14 - B{{2$ ZT4 RWR_R_` $ ZT#**EMMZ $ ZQszzW  Q F !  /2C C *j8JA~)J6 /,> 5K%6{%BA EFQQSC!2;!>ABMMO%6{%BA EFQQS%6{%B BCNNP%6{%B BCNNP%%%)'+*? 622'"\1! e$ /2C C  1$5J%6%F IJUUWC!2?!BAEFQQS%6%F IJUUW%6%F FGRRT%6%F FGRRT%%%'+ 622' e$%%f!4E!# ??PQDE2CmmA&KKNmmA& 2 .E.GWg!Qa-(E 7GW55 5Lr ceZdZdZeZdeffd Z ddejdejdejdejdejd ejd ejd ejd e d e e dde eje eje ejejejffffdZdefdZxZS) xLSTMBackendzxLSTM Backend Module for PyTorch. This module wraps the xLSTM kernels and provides a high-level interface for training and inference. configc t|||_t|_t |_t|_tt|jt|j tt|jt|jtt|j|j|jtt|j d|_t|jtt|j |j|j}d|j$vrtt&|}||_y)N)rT)rrrr-r[rr)rr[r- with_padding)r)super__init__rrchunkwise_kernel_fnrsequence_kernel_fnrstep_kernel_fnrrgetattrrinference_state_dtyper-r[r _inference_fnmoder _train_fn)selfrtrain_kernel_fn __class__s rrzxLSTMBackend.__init__s G   DK'FD $&ED #"=D !(8'+'?'?&-++ 'v/K/K L'#*'' 'v/K/K L#",,JJ&-eV5Q5Q&R#'"D "&((&-eV5Q5Q&RJJ!,, O ,")*B[j"k,DNr rrLrrrrrrrr)train inferencerc h| |jj} d| vrX| |jj} |jjdk(r | r td|j |||||||||  Sd| vr|j ||||||||Std|jj)aForward pass of the mLSTM backend. Depending on the configured mode, this method will call the appropriate kernel function. Args: query: The query tensor of shape (batch_size, nh, sequence_length, dhqk). key: The key tensor of shape (batch_size, nh, sequence_length, dhqk). value: The value tensor of shape (batch_size, nh, sequence_length, dhhv). igate: The input gate preactivation tensor of shape (batch_size, nh, sequence_length). fgate: The forget gate preactivation tensor of shape (batch_size, nh, sequence_length). c_initial: The initial cell state tensor of shape (batch_size, nh, dhqk, dhhv). Defaults to None. n_initial: The initial hidden state tensor of shape (batch_size, nh, dhqk). Defaults to None. m_initial: The initial memory tensor of shape (batch_size, nh, 1). Defaults to None. return_last_states: Whether to return the last states of the sequence. Defaults to None. If None, the value from the config is used. Returns: hidden states of shape (batch_size, nh, sequence_length, dhhv) hidden states and last states the last states are the cell state cstate (batch_size, nh, dhqk, dhhv), the normalizer state nstate (batch_size, nh, dhqk), and the max state mstate (batch_size, nh, 1) rtrain_with_paddingzFreturn_last_states=True is not supported with train_with_padding mode.) rrLrrrrrrrrrrLrrrrrrzUnknown mode: )rrrrrr) rrrLrrrrrrrrs rforwardzxLSTMBackend.forwardsF|{{''$%-)-)G)G&;;##';;)()qrr~~''''9&  $))'''*  !>$++2B2B1C!DEEr c|jSrrrs r extra_reprzxLSTMBackend.extra_reprJskk] #r )NNNFN)__name__ __module__ __qualname____doc__r config_classrrTensorrdrrrtuplerstrr __classcell__rs@rrrs% #  -; -R'+&*&*',<@G F<<G FG F<< G F << G F << G F||G F||G F||G F!%G F7#789G F5<<u||U5<<W\WcWc;c5d'd!ee fG FR $ $r rc eZdZdZ d dededededef fd Zdejd ejfd Z dejd ejfd Z dejd ejfd Z xZ S)ra3Root mean square normalization layer implementation similar to https://pytorch.org/docs/stable/generated/torch.nn.RMSNorm.html. It normalizes the input tensor by the root mean square of the last dimension. Args: num_features: The number of features in the input tensor. eps: A small value to avoid division by zero. use_weight: Whether to use a learnable weight. use_bias: Whether to use a learnable bias. force_float32_reductions: Whether to force float32 reductions. num_featuresr[ use_weightuse_biasforce_float32_reductionsc(t|||_||_||_|r.t j tj||_ nd|_ |r.t j tj||_ yd|_ yr) rrrr[rr Parameterrrcweightr7bias)rrr[rrrrs rrzxLSTMRMSNorm.__init__[sm G   ,D DH,DD ) ll5::l+CD " LL\)BC   r xrcr|j||jz}|j||jz}|Srrrrrs r_apply_weight_biaszxLSTMRMSNorm._apply_weight_biasr5{{& Oyy$ MHr c|j}|jr|j}|tj|j dj dd|jzz}|j|S)Nrr0Tr^) r3rrfrrsqrtpowmeanr[r)rrin_dtypes r_rms_normalizezxLSTMRMSNorm._rms_normalizeys^wwH,,GGIEKKa "d Cdhh NOOA44> !r cJ|j|}|j|}|Sr)r&rrs rrzxLSTMRMSNorm.forwards'##A&A''*AHr rYTFT)rr r r intrfrdrrr rr&rrrs@rrrMs #"-1  ! ! !  !   ! '+  !.     "ELL "U\\ " U\\ ell r rc eZdZdZ ddedededededef fd Zd ejd ejfd Z d ejd ejfd Z d ejd ejfd Z xZ S)xLSTMMultiHeadLayerNormamMulti-head version of the LayerNorm layer. It normalizes the last dimension of the input tensor. The input is assumed to have the shape (batch_size, sequence_length, nh, DH), where: batch_size: batch size sequence_length: sequence length nh: number of heads DH: head dimension The normalization is applied over the last dimension (DH) of the input tensor. Args: num_heads: The number of heads. head_dim: The head dimension. eps: A small value to avoid division by zero. use_weight: Whether to use a learnable weight. use_bias: Whether to use a learnable bias. force_float32_reductions: Whether to force float32 reductions Returns: The normalized tensor with the shape (batch_size, sequence_length, nh * DH). num_headshead_dimr[rrrcrt|||z|_||_||_|r8t j tj|j|_ nd|_ |r8t j tj|j|_ nd|_ ||_ ||_ yr)rrrr[rrrrrcrr7rr,r-)rr,r-r[rrrrs rrz xLSTMMultiHeadLayerNorm.__init__s G   )H 4D DH,DD ) ll5::d6G6G+HI " LLT5F5F)GH   &DN$DMr rrcr|j||jz}|j||jz}|Srrrs rrz*xLSTMMultiHeadLayerNorm._apply_weight_biasr r c|j}|jr|j}||jddz }|t j |j ddd|jzz}|j|S)Nr0Tr^F)r_r`unbiased) r3rrfr$rr"varr[r)rrr% x_centeredys r_layer_normalizez(xLSTMMultiHeadLayerNorm._layer_normalizesrwwH,,GGIQVVDV99JU[[2te)TW[W_W_)_``A44> !r cn|j\}}}}||jk7r(td|jd|d|j||jk7r(td|jd|d|j|j |}|j ||d}|j |}|S)Nz Expected z heads, got z, input shape: z head dimension, got r0)r6r,rr-r5rjr)rrr>rr?DHs rrzxLSTMMultiHeadLayerNorm.forwards34'' /JRT^^# 9T^^,!A--$ &(YY $ 7 7!"T[[%:%:!:&" !# D !-T[[!AD "9++//t{{'<'<<KK((--)-)Q)Q #D II JJ![[44[[))DMr rstaterc |jdk7rtd|j|j\}}}|jjdk(r|j |}|j |}|j|}|j|} t|j||jj} t|j||jj} n|jjdk(r|j|} tj| |j d|j zd|j z|j"zfd\}}}} t|j%||jj} tj| |jj&fd\} } j)|||jj&dj+d d}j)|||jj&dj+d d}j)|||jj&dj+d d} j+d d}  j+d d} |d \}}}n|\}}}|j-|||| | ||| \}}||jj&||j"|jj&zf}|j|k7rtd |jd ||j+d d}|j/|}|j)||d}|j1 |z}|j3|}||fS)Nrz=Input must have shape [batch_size, sequence_length, HD], got r;)rr?rr0rr)NNNrzGot z , expected )ndimrr6rrDrXrYrZr[rr\ gate_soft_capr]r^rrNrWrUr_r,rjr<rbrerarf)rrrgr>rr@rrLro_preacti_preactf_preactr^ if_preactrrrrexpected_h_shapeh_normrr4s rrzxLSTMLayer.forwardLs7vv{ #`abahah`i!jkk-.WW *J{{&&(2q ffQiq ,,Q/#D$5$5a$8DKKD]D]^#D$5$5a$8DKKD]D]^((G3"..q1 .3.@.@ DKKDKK$**4 /+sE8%T%7%7%:dkkF_F_` %*%7%7 DKKDYDYC[ac%d"(MM*ot{{?T?TVXYccdeghiE++j/4;;;P;PRTU__`acdeCMM*ot{{?T?TVXYccdeghiE))!Q/H))!Q/H}2B/ 9i27/ 9i))###* HAu %% dkk333    ww** 4y > ?A r rRc teZdZdeffd Z ddej deede ej effdZ xZ S) xLSTMBlockrcdt|||_t|j|j d|j |j|_t||_ t|j|j d|j |j|_ t||_ y)NT)rr[rrr)rrrrr@rcrrd norm_mlstmrR mlstm_layernorm_ffnr9ffnrLs rrzxLSTMBlock.__init__s G   DK*#//OO)/)L)L DO *&1D (#//OO)/)L)L DM(/DHr rrgrc|j|}|j||\}}||z}|j|}|j|}||z}||fSr)rurvrwrx)rrrgx_mlstmx_ffns rrzxLSTMBlock.forwards^ooa(G!--gu=NGUG AMM!$EHHUOEE Ae8Or r) rr r rrrr rrrrrrs@rrsrssJ 0; 0*FJ \\ *2>*B  5<</ 0 r rsc(dd|zz dzfd}|S)a> Adapted from: https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/init_functions.py Fills the input Tensor with values according to the method described in Transformers without Tears: Improving the Normalization of Self-Attention - Nguyen, T. & Salazar, J. (2019), using a normal distribution.r?c\tjjj|dSNg)r$stdrrinitnormal_tensorrs rinit_z small_init_method..init_#xx}}$$V#3$??r r)r_rrs @rsmall_init_methodrs$ C=e $C@ Lr c(d|z |dzz fd}|S)zh Adapted from https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/init_functions.py rr~c\tjjj|dSrrrs rrzwang_init_method..init_rr r)n_layersr_rrs @rwang_init_methodrs$ h, 'C@ Lr c2eZdZdZeZdZdgZdZdZ dZ dZ y)xLSTMPreTrainedModelzR An abstract class for an interface to loading a pre-trained xLSTM model. backbonersTcD|jD] \}}||us |cSy)N) named_modules)rmodulenamemods r_module_name_mapz%xLSTMPreTrainedModel._module_name_maps/++- ID#f}  r c  t|tjr:t|jj |j jyt|tjr8|j3tjjj|j|jjdk(rGd|j|vr3tjjj|jtj5d|j|vr<|jj!dtj"|jznd|j|vr|jj!tj$dd|jj&dj)|jj*|jj, dddy|jjd k(rd|j|vrxtjjj|jtj5|jd|jj.xxx|jd|jj. d tj"|jzz z ccc|jd|jj.xxx|j|jj.d tj$dd|jj&dj)|jj*|jj, zz cccdddyd |j|vrIt1|jj&d |jj2|jyd|j|vrFt1|jj |jj2|jy|j0t|jj |jyyt|t4s t7|drtjjj9|jt7|drA|j4tjjj|jyyyy#1swYyxYw#1swYyxYw)Nr;rOrg$rg@g@r0rr?g$@rIr)r_rrfr5r) isinstancer Embeddingrrr@ embeddingsrrErrrzeros_rDrno_gradcopy_ ones_likelinspacer6rr4r3r,rnum_hidden_layersrhasattrones_)rrs r _init_weightsz"xLSTMPreTrainedModel._init_weightss fbll + 6 dkk55 6t7M7M N  *{{& $$V[[1{{&&(2vAVAVW]A^7^ $$V]]3]]_ $"7"7"?? ))%%//&++2N*NO D$9$9&$AA ))!NN # # & 1 1" 5!b'-{{'9'9&,kk&7&7!   ((G3$BWBWX^B_8_ $$V]]3]]_ KK 7$++"7"78V[[/$++//>=uv{{;;=<<8KK 7$++"7"78V[[I^I^I`=a>> from transformers import AutoTokenizer, xLSTMForCausalLM, xLSTMCache >>> model = xLSTMForCausalLM.from_pretrained("NX-AI/xLSTM-7b") >>> tokenizer = xLSTMTokenizer.from_pretrained("NX-AI/xLSTM-7b") >>> inputs = tokenizer(text="I am an xLSTM", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> cache_params = xLSTMCache(config=model.config, max_batch_size=1, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, cache_params=cache_params, use_cache=True) >>> outputs.cache_params xLSTMCache() Nrmax_batch_sizer3r4c d|_||_||_t|jDcic]}|t j ||j|j|jg||t j ||j|jg||t j ||jdg||fc}|_ ycc}w)Nrr2r) seqlen_offsetr3rr:rrr7r, qk_head_dim v_head_dim rnn_state)rrrr3r4rlayers rrzxLSTMCache.__init__/s  v778    #V%5%5v7I7I6K\K\]!  ^V-=-=v?Q?QRZ_hno ^V-=-=qAW]^     sBC c &|jDcic]q}|tj|j|dtj|j|dtj|j|dfsc}|_ycc}w)Nrrr)rr zeros_like)rrs rresetzxLSTMCache.resetGs     !6q!9:  !6q!9:  !6q!9:   sA6B) rr r r rbfloat16rr)r3rrrrrr rrr sL L#^^ $    {{    0 r rcxeZdZUdZeej ed<dZee ed<dZ ee ej ed<y) xLSTMOutputz cache_params (`xLSTMCache`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. last_hidden_stateN cache_params hidden_states) rr r r rr FloatTensor__annotations__rrrrrr rrrRsC   1 122)-L(:&-8||jr2|j1| j2|||j&| nd\}} n| |||j&| nd\}} |rSt)t+|j&| D](}| |}|j&| |j-|*d|_|s||fz}|r"|xj4|jdz c_|j7|}|r|fz}t9|| S#1swYTxYw) r cache_params (`xLSTMCache`, *optional*): The xLSTMCache that carries the RNN states. NFz:You must specify exactly one of input_ids or inputs_embedsrrr)rr>)rgr)rrr)rrtrainingrrrrrsizer4r3max_inference_chunksizer6rrrmin enumeraterrr:lenrrnn_state_initial_gradient_checkpointing_func__call__rrr)rrrrrrrroffset final_statehidden_states_chunk layer_idx xlstm_blockr state_idxlocal_rnn_stateall_hidden_statess rrzxLSTMModel.forwardrs %9$D $++JjJj "+!6IZ^ZgZgT[[=R=Rmr  & &4==YI  -t"; <YZ Z   OOI6M -% ]//2=;O;OWdWjWjL&   33m6I6I!6LL(F ,'#-T[[]M`M`abMc#dL#..}= }22155*76C1T1T(TVcViVijkVl$mmm+'3$..y9)DJJ?[\6;L2'(9] ` 'tn `  uk! "` ` r rceZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejed<y)xLSTMCausalLMOutputaP loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). cache_params (`xLSTMCache`, *optional*, carrying the RNN states): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Nlosslogitsrr)rr r r rrrrrrrrrrrr rrrs\)-D(5$$ %,*.FHU&& '.)-L(:&-8 6 'tn6  u)) *6 6 r r)rrrr) NNNNNNNr!r)r!rrY)NNNNFFr!rY)NNNFrYr!)Er  dataclassesrtypingrrrtorch.nn.functionalrrrtorch.utils.checkpointtorch.nnr generationr modeling_utilsr utilsr r r rconfiguration_xlstmrxlstm.xlstm_large.modelrrsrrrexternal_xlstm functoolsrrrrrr rqdictr)rfrXrrdrrrr3rrrrrModulerr+r9rRrrrrrrrr__all__rr rrs !" %)-VV,@@DN!(A ellELL HI#223NN::(5 AT;U2V:bgbnbn:0%)$()-%)%)*.$(X:llX:llX:llX:ll X: \\ X: \\ X: ,,X:llX:llX:!<<X:5/X:X:X: u||U\\5<<7 8X:JK,llK,llK,llK, \\ K, \\ K, ,,K,llK,llK,K,K,K,K, u||U\\5<<7 8K,f $##$(#("'O||O \\O||O|| O || O  O O O5/O!O OOO    u||U\\5<<?@Au||U\\5<<?@A  C On#'"&"&#(?||? \\?||?|| ? || ? << ?<<?<<?!??? u||U5<<u||U\\SXS_S_7_1`#`aa b?T#(===C||=C \\=C||=C|| =C || =C  =C =C =C=C[[=C u||U5<<u||#KLL M=CJ#'"&"&#(#(==G||G \\G||G|| G || G << G<<G<<G!GG[[G    u||U\\5<<?@Au||U\\5<<?@A  C G`#'"&"&#(-2^^< (<||<\\<|| < || < || <<<<<<<<<<!<< %{{<< u||U5<<u||U\\SXS_S_7_1`#`aa b<N#'"&"&#'-2^^$!S (S'S$S|| S \\ S || S||S||S<<S<<S<<S!SS %{{SS !S" u||U5<<u||U\\SXS_S_7_1`#`aa b#Sjt$ryyt$l7ryy7rP"))Pd-299-^FRYYFP RYY F  >2?>2BC C L  =+ =  =s %s s l =+= ="f +_f f R r