rhJddlmZmZmZmZmZddlZddlmcm Z ddlmZddl m Z ddl mZmZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,e*rddl-m.Z.ddl/m0Z0m1Z1ndZ.e)r ddl2m3Z3m4Z4nd\Z4Z3e&r ddl5m6Z6ddl7m8Z8e'jre:Z;dZd?ejZRGd@dAejZSGdBdCeZTe$GdDdEe ZUGdFdGejZVe$GdHdIeUZW dRdJeej|eXej|dfdKee?d"eej|deej|e?ffdLZYGdMdNeUeZZgdOZ[y)S)AnyCallableOptional TypedDictUnionN)nn)ACT2FN)Cache DynamicCache DynamicLayer)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)is_causal_conv1d_availableis_mamba_2_ssm_available)GraniteMoeHybridConfig)selective_state_update)mamba_chunk_scan_combined mamba_split_conv1d_scan_combined)causal_conv1d_fncausal_conv1d_updateNN) BlockMask)make_flex_block_causal_maskc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py rotate_halfr5@sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer5)qkcossin position_ids unsqueeze_dimq_embedk_embeds r4apply_rotary_pos_embrAGsY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r6 hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r.expandreshape)rBrCbatchnum_key_value_headsslenhead_dims r4 repeat_kvrLbso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr6modulequerykeyvalueattention_maskscalingdropoutc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr+r r*)r-dtype)ptrainingr)rLnum_key_value_groupsr/matmul transposer.r functionalsoftmaxfloat32torVrSrX contiguous) rMrNrOrPrQrRrSkwargs key_states value_states attn_weights causal_mask attn_outputs r4eager_attention_forwardrgns3 ; ;Q5RR S0)r6ric eZdZdZdZdZdZejdfde ffd Z ddejdejde d e eeefd eejejff d Zd ej&fd Zdde e d e fdZd eeejeejffdZedde eeej0d dfdZxZS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. NFrjc Vt |t|j|_d|_|j }|j }g|_g|_g|_ t|jD]*}|j|dk(r|xjtj||j|jzd|j z|zz|||gz c_|xjtj||j"|j$|||gz c_|xjtj&gg|z|gz c_|xjtj&gg|z|gz c_|jj)|-t|jDcgc]}tj&gg|z|c}|_t|jDcgc]}tj&gg|z|c}|_ycc}wcc}w)N) layer_classesFmambar+devicerVr)rorpr layers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statestransformer_layersrangenum_hidden_layersr/zeros mamba_expandrvmamba_n_groups mamba_n_heads mamba_d_headtensorappend key_cache value_cache) rrj batch_sizerVrconv_kernel_sizessm_state_sizeirrss r4rpz)HybridMambaAttentionDynamicCache.__init__s |4!'!9!9"'!..--"$v//0 2A%%a(G3  KK",,v/A/AAAH]H]D]`nDnn(%# % KK",,++&%#  $   U\\2$2CF%S$TT ELL" 1B6$R#SS''..q11 24SXX^XpXpRqrQ%,,tj'8HrTYZ`ZrZrTstqELL" ):6Jtsts 5"H!6"H&rbrcrkrrDc|j|jddk(r||j|<||j|<nftj|j||gd|j|<tj|j||gd|j|<|j||j|fS)Nr*rr+r,)rr.rr/r0)rrbrcrkrs r4rz'HybridMambaAttentionDynamicCache.updates >>) $ * *2 .! 3(2DNN9 %*6D  Y '(- 4>>)3Lj2Y_`(aDNN9 %*/))T5E5Ei5PR^4_ef*gD  Y '~~i($*:*:9*EEEr6beam_idxctt|jD]S}|j|j}|j|j d|j ||j|<|j |j}|j |j d|j ||j |<|j|j}|j|j d|j ||j|<|j|j}|j|j d|j ||j|<Vy)zDReorders the cache for beam search, given the selected beam indices.rN) rlenrr index_selectr_rrr)rrrkrs r4 reorder_cachez.HybridMambaAttentionDynamicCache.reorder_cache+sDs4>>23 iI^^I.55F(,y(A(N(NqRZR]R]^dRe(fDNN9 %%%i077F*.*:*:9*E*R*RSTV^VaVabhVi*jD  Y '%%i077F*.*:*:9*E*R*RSTV^VaVabhVi*jD  Y '__Y/66F)-)C)P)PQRT\T_T_`fTg)hDOOI & ir6c||jvr|jdn|}t|j|kry|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rrU)rrrr.)rrks r4get_seq_lengthz/HybridMambaAttentionDynamicCache.get_seq_length8sR3<4CZCZ2ZD++A.`i t~~ ) +~~i(..r22r6ctdNzIHybridMambaAttentionDynamicCache does not have a legacy cache equivalent.NotImplementedErrorrs r4to_legacy_cachez0HybridMambaAttentionDynamicCache.to_legacy_cache@s!"mnnr6past_key_valuesr ctdrr)clsrs r4from_legacy_cachez2HybridMambaAttentionDynamicCache.from_legacy_cacheCs!"mnnr6N)r)rtrrrrris_compileabler/float16r rprrrdictstrrrrrrrr classmethod FloatTensorrrrs@r4rrs+ IKNIN_c%u5%uX26 FLLFllF F tCH~. F u||U\\) * F" ie&6&6 i3 3c3ouU\\':E%,,