rhQ(ddlZddlZddlZddlmZmZddlZddlmcm Z ddl m Z ddl mZmZmZmZmZmZej(eZdej.dej.dej.fd Zd0d Zd Zd ej.deej.ej.effd Zdej.dej.dej.d ej.def dZdZdZ d0deejBfdZ"dee#fdZ$dZ%dZ&dZ'GddedZ( d1dej.d ej.d!ej.d eej.ded"e)d#e*d$eej.d%ee*d&eed'e)d(ee*d)ee)d*eejVd+eejVd,eed-eedeejBd.ee#f&d/Z,y)2N)Optional TypedDict)is_kernels_available)is_flash_attn_2_availableis_flash_attn_3_availableis_flash_attn_greater_or_equal#is_flash_attn_greater_or_equal_2_10is_torch_npu_availableloggingtensorindicesreturnch|jjdg|jdd}||S)N) contiguousreshapeshape)r rreshapeds ~/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/modeling_flash_attention_utils.py_index_first_axisr%s8*v  "**2A QR0@AH G c|||zn|}|jdtj}|jdtj}tj|j dj }|j j }tjtj|dtjd}t||||||fS)a FA3-compatible unpad_input function. Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. rdimdtypeFas_tuplerrr) sumtorchint32nonzeroflattenmaxitemFpadcumsumr) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchrmax_seqlen_in_batch cu_seqlenss r_fa3_unpad_inputr3*s3>2I+-~I }}5;;}?*..2U[[.ImmI--/%@HHJG*..0557u||$4!5;;OQWXJ -1  rc|jdd}tj||zg||j|jd}|||<|j ||g|S)a FA3-compatible pad_input function. Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) rNdevicer)rr"zerosr6rview)r+rbatchseqlenroutputs r_fa3_pad_inputr<Hsb   ab !C [[%&. hC h 8L8LTaTgTg hF#F7O 6;;uf +s ++rr,cd|jdtj}tj|j dj }|j j }tjtj|dtjd}|||fS)ao Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. rrFrrr ) r!r"r#r$r%r&r'r(r)r*)r,r/rr1r2s r_get_unpad_datar>Ys&))b )DmmN224uEMMOG+..0557u||$4!5;;OQWXJ r query_layer key_layer value_layer query_lengthct|\}}}|jd|jdx} kDr"|ddd| ddddf|ddd| ddddf}}|j\} } } } t||}t||}|| k(rt||}|}|}|}nk|dk(rLd}tj| dztj |j }|dd}|jd}n|dd| df}|||^}}}}}||||||f||ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. unpad_input_func: The function to use for unpadding the input tensors. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rrN)rr6)r>rrr"aranger#r6squeeze)r?r@rAr,rBunpad_input_func indices_k cu_seqlens_kmax_seqlen_in_batch_kseq_len batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_q_s r _upad_inputrSts`N6E^5T2I|2q(<((9:JZ[fhvJwG Y .Ca  |$  56  rc<|jjd|jd|jd}|jjd|jd|jd}|jjd|jd|jd}|j}t j |jd|j tj}t j||dk(t j|j|j tjf}|jjj}||||||f||ffS)aF This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cumulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cumulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rrr5) rr8sizer%r"rDr6r#catr diffr&r')querykeyvalue position_idsrQ cu_seq_lens max_lengths r_prepare_from_posidsr_sG:     # #B 2 2 GE ..   CHHRL#((2, ?C     # #B 2 2 GE'')L \..q1,:M:MUZU`U`aI)) la' ( LL**,\5H5HPUP[P[ \ K!!#'')..0J 3y; *DzS]F^ __rcRtjdtt||||S)NzEprepare_fa2_from_position_ids is deprecated, use _prepare_from_posids)warningswarn FutureWarningr_)rYrZr[r\s r*_prepare_flash_attention_from_position_idsrds' MMO sE< @@r target_dtypec|ri|jtjk(rLtj d|d|j ||j ||j |}}}|||fS)NzCasting fp32 inputs back to z for flash-attn compatibility.)rr"float32logger warning_onceto)qkvres rfa_peft_integration_checkrns`5==0:<.Hfgh$$|$add<&8!$$|:La1 a7Nrimplctxs t}t}|dk(s||r|s ddlm}m}ddlm}m}||||dfS|dk(s||rddlm}m}t$t&}}||||dfSt$t&}}t)|d dt)|d ||dfS#t$r}tjdds6tdjj}|dvtd<tdrPts td dd lm} | d }t$t&}}t)|d dt)|d ||dfcYd}~Std|d}~wwxYw)Nflash_attention_2r)flash_attn_funcflash_attn_varlen_func) pad_input unpad_inputFuse_remote_fa2zUnable to import the official flash attention, do you want to try to use `kernels-community/flash-attn` (trust remote code) Yes or No? >1yyesz2You need to install kernels: `pip install kernels`) get_kernelzkernels-community/flash-attnrrrsTzTFailed to import flash attention 2, please install it or use another implementation.flash_attention_3)rr r flash_attnrrrsflash_attn.bert_paddingrtru ImportErrorglobalsgetinputstriplowerrkernelsrzr<r3getattrflash_attn_interface) rois_fa2is_fa3rrrsrtruervrzs r _lazy_importsrs & ( D,B,DF & (F ""t|v"  J F"$:I{TYY Y> ""t|P!/1A;  6 ;PTTT!/1A; D+T 2 D2 3      I 9==!148bUWUW /=@Q.Q *+y)*+-%&Z[[.!"@A)79I; D"3T:D":; "j5 s$B EB'EE EEcFtxstxs tSN)rrr rris_flash_attn_availablerAs $ & a*C*E aI_Iaarc\trytr t Sddlm}|S)NFr'is_npu_fa2_top_left_aligned_causal_mask)rrr integrations.npu_flash_attentionrrs r!flash_attn_supports_top_left_maskrEs) " "6888Y 2 44rcVeZdZUeej ed<eej ed<y)FlashAttentionKwargscumulative_seqlens_qcumulative_seqlens_kN)__name__ __module__ __qualname__rr" LongTensor__annotations__rrrrrPs$"5#3#344"5#3#344rrF)total query_states key_states value_states is_causaldropoutr\ softmax_scalesliding_windowuse_top_left_masksoftcap deterministic cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kimplementationc ,tddDst|\}}}}}|td<|td<|td<|td<|td<dtj|j v}|td <nNtd}td}td}td}td}td }|xr | xr|d k( }t xs|xr| xr|jd | kD}|rd| | fini}|s||d <td r"| | ntjd ddk(}||d<| | |d<d|vr|jd|d<t||||\}}}|duxstd| |||fD}|t||||||\}} }!}"\}#}$\}%}&dt|jvr|$j!}$||| |!f|#j#t$j&|$j#t$j&|%|&||d|}'t)|'t*r|'d}'||'|"|jd|}(n|r| |)| t-dt/||||\}} }!}"\}#}$\}%}&n|j1d|j3d|j3d}|j1d|j3d|j3d} |j1d|j3d|j3d}!||}&}%| |}$}#dt|jvr|$j!}$||| |!f|#j#t$j&|$j#t$j&|%|&||d|}(t)|(t*r|(d}(|(j5|jdd|(j3d|(j3d}(n||||f||d|}(t)|(t*r|(dS|(S)Nc34K|]}|tvywr)r.0rls r z+_flash_attention_forward..ksl!qGI~ls) _flash_fn_flash_varlen_fn_pad_fn _unpad_fn_is_fa3rrrrr window_size_flash_supports_windowr dropout_pz2.4.1FLASH_ATTENTION_DETERMINISTIC0rwrrs_auxc3$K|]}|du ywrrrs rrz+_flash_attention_forward..s/ /smps)rOrH max_seqlen_q max_seqlen_krcausalrzePosition ids should be passed if the attention mask is not passed and the cu_seq-lens are not passed.rrU)rr)allrrinspect signature parametersrrr osgetenvrrnrSstrr6clonerjr"r# isinstancetuple ValueErrorr_rrVr8))rrrr,rBrrr\rrrrrrrrrrerkwargsflash_fnflash_varlen_fnpad_fnunpad_fnrflash_supports_windowruse_sw flash_kwargsdetuse_maskrkrlrmidxcu_qcu_kmqmk out_unpadouts) r_flash_attention_forwardrUs, l'kl l>KN>[;/68V!) +(7 $%% )!) +% ) -1B1B?1S1^1^ ^.C *+9[)!)$679%9[)9% ' *B C  H 1 Gla6GHF  8#8unuQ[QaQabcQdguQu IOMNN#CDTVL $+ [!%g.,8mbiiHgil>mqt>t(+ _%") Y& & 7 3 W-Fj, .*L*l4'3/ -}lLY/,H!/: *lNLRZ0 ,1altTHR CM !:: @Q@QRT@UVA""2zr':JOOB @Q@QRT@UVA!<B& $D CM !::rSr_rdrrnrrrrrrboolfloatrrrrrrs & @   H %ellU\\ell <,"ELLU5<<WZ;Z5[6JJ||JJLL J  JZ3`lAXekk5J5  5 pb559E5+/%)$(##$(0404"&"&*.$('v5,,v5 v5,,v5U\\* v5  v5  v5v55<<(v5E?v5SMv5v5e_v5D>v5E,,-v5E,,-v5 3-!v5"3-#v5$5;;'%v5&SM'v5r