rh ddlZddlmZddlmZ erddlmZ ddejjdejdejd ejd ejd ed ejfd Z y#e$rYiwxYw)N)PagedAttentionCache)is_flash_attn_2_available)flash_attn_varlen_funcmoduleqkvattention_maskcachereturnc |j|||jfd|i| \}}t|ddsdn |jdf} | | j}d| j di}|j dd jdj|j dd jdj|j dd jdj|jtj|jtjj|| f|jd | d |}t|tr|d}|dfS) aPerform the forward pass of attention with paged key-value cache. This function handles the cache updates and performs the attention computation using the flash_attn_varlen_func for efficient processing. Args: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. but if there is a block table it can be the full k v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. but if there is a block table it can be the full v cumulative_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cumulative_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. cumulative_seqlens_ksliding_windowF)rrNs_auxrT) softmax_scalecausal window_size)update layer_idxgetattrrrget transposesqueeze contiguoustotorchint32clonescaling isinstancetuple)rrr r r r cumulative_seqlens_qr max_seqlen_q max_seqlen_k block_tablesimplementationkwargsrr custom_kwargs attn_outputs x/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/integrations/flash_paged.pypaged_attention_forwardr.sUJ 5<<1f.. dEY d]c dDAq%,V5Eu%MXTZTiTiklSmN!!/!F!Ffjj12M( Aq!!!$//1 Aq!!!$//1 Aq!!!$//1 , ,224 nn"   K+u%!!n  )NNNNNNNN) rgeneration.continuous_batchingrutilsr flash_attnr ExceptionnnModuleTensorr.r/r-r8s @-  "5$(!%; HHOO; ||; ||; || ; LL ;  ; \\;   s BB B