rh`,ddlmZmZmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZddlmZddlmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&e"jNe(Z)dZ*d4dZ+dejXde-dejXfdZ. d5dej^dejXdejXdejXdeejXde0de0d eefd!Z1Gd"d#ej^Z2ed$Gd%d&ej^Z3Gd'd(ej^Z4Gd)d*eZ5e Gd+d,eZ6Gd-d.ej^Z7e Gd/d0e6Z8e Gd1d2e6e Z9gd3Z:y)6)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)check_model_inputs) GraniteConfigc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/granite/modeling_granite.py rotate_halfr)-sZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer))qkcossin position_ids unsqueeze_dimq_embedk_embeds r(apply_rotary_pos_embr54sY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r* hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r"expandreshape)r6r7batchnum_key_value_headsslenhead_dims r( repeat_kvr@Oso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr*modulequerykeyvalueattention_maskscalingdropoutkwargsc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nrrr)r!dtype)ptrainingr)r@num_key_value_groupsr#matmul transposer"r functionalsoftmaxfloat32torKrGrM contiguous) rArBrCrDrErFrGrH key_states value_states attn_weights causal_mask attn_outputs r(eager_attention_forwardr[[s3 ; ;$>&B\B\$\!22 !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r*r6position_embeddingsrEpast_key_valuecache_positionrHr8c 4|jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7rt|jj}||| | | |f|jsdn |j|j d|\}}|j"g|dj%}|j'|}||fS)Nrrr)r0r/rveager)rGrF)r"r?rmviewrPrnror5updater_r[r^_attn_implementationrrMrirFr;rUrp)rrr6rtrErurvrH input_shape hidden_shape query_statesrVrWr/r0 cache_kwargsattention_interfacerZrXs r(forwardzGraniteAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'))!. )) ! )) !!1!12 ))+,)) u||U\\) *))r*r]RMSNormc,eZdZdfd ZdZdZxZS)GraniteRMSNormct|tjt j ||_||_y)z= GraniteRMSNorm is equivalent to T5LayerNorm N)rcrdr Parameterr#onesweightvariance_epsilon)rrrfepsrss r(rdzGraniteRMSNorm.__init__s1 ll5::k#:; #r*c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrrT)keepdim) rKrTr#rSpowmeanrsqrtrr)rrr6 input_dtypevariances r(rzGraniteRMSNorm.forwardsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r*c^t|jjd|jS)Nz, eps=)rrr"rrrs r( extra_reprzGraniteRMSNorm.extra_reprs* ))*+6$2G2G1HIIr*)gư>)rrrrdrrrrs@r(rrs$;Jr*rc$eZdZfdZdZxZS) GraniteMLPct|||_|j|_|j|_t j |j|j|j|_t j |j|j|j|_ t j |j|j|j|_ t|j|_ y)Nra)rcrdr^rfintermediate_sizerrkmlp_bias gate_projup_proj down_projr hidden_actact_fnrrr^rss r(rdzGraniteMLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r*c|j|j|j||j|z}|Sr)rrrr)rrr%rs r(rzGraniteMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r*)rrrrdrrrs@r(rrs 0r*rcfeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ejee ejejfffd ZxZS)GraniteDecoderLayerr^r_cBt||j|_t|||_t ||_t|j|j|_ t|j|j|_ |j|_ y)N)r^r_r) rcrdrfr] self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormresidual_multiplierrqs r(rdzGraniteDecoderLayer.__init__sz !--)9Mf%-f.@.@fFYFYZ(6v7I7IvObOb(c%#)#=#= r*r6rEr1ruoutput_attentions use_cachervrtr8c |} |j|}|jd||||||||d| \}} | ||jzz}|} |j|}|j |}| ||jzz}|f} |r| | fz } | S)a. Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r6rEr1rurrrvrt)rrrrr) rrr6rEr1rurrrvrtrHresidualself_attn_weightsoutputss r(rzGraniteDecoderLayer.forwardsD!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=43K3K#KK !55mD /  =43K3K#KK  "  )+ +Gr*)NNNFFNN)rrrrrrdr#rrrr boolr FloatTensorrrrs@r(rrs>}>>2637*.,1$)59KO?||?!.?u//0 ? ! ? $D> ?D>?!!1!12?&eELL%,,,F&GH? u  (51B1BEDUDU1U+V"WW X?r*rcJeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZy)GranitePreTrainedModelr^modelTrpast_key_values)r6 attentionsN)rrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr]_can_record_outputsrr*r(rr-sQ &*#./#4"5N!"&,&r*rc^eZdZddeffd Zej edZxZ S)GraniteRotaryEmbeddingr^ct|t|drUt|jt r;|jj d|jj d|_nd|_|j|_ |j|_ ||_ t|j|_ |j|j|\}|_|jd|d|j |_y)N rope_scaling rope_typetypedefaultinv_freqF) persistent)rcrdhasattr isinstancerdictgetrmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr^r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)rrr^devicerrss r(rdzGraniteRotaryEmbedding.__init__As  6> *z&:M:Mt/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.r) rrrrrdr#no_gradrrrrs@r(rr@s3/}/"U]]_ < _ $D>_ 'tn_ !!1!12_ +,_  !_ _ r*rceZdZdgZddiZddgdgfiZfdZdZdZe e dd e e jd e e jd e e jd e eeee j$fd e e j$de e jde ede ede ede e jdeee jfdeedefdZxZS)GraniteForCausalLMzlm_head.weightlm_head colwise_repr6logitsct||t||_|j|_t j |j|jd|_|jy)NFra) rcrdrrrrrkrfrrrs r(rdzGraniteForCausalLM.__init__sU  !&)  ++yy!3!3V5F5FUS  r*c||_yrr)rrdecoders r( set_decoderzGraniteForCausalLM.set_decoders  r*c|jSrrrs r( get_decoderzGraniteForCausalLM.get_decoders zzr*rrEr1rrlabelsrrrrvlogits_to_keeprHr8c ||n|jj}| | n|jj} |jd|||||||| | d | } | j}t | t r t| dn| }|j|dd|ddf}||jjz }d}|*|jd|||jjd| }t||| j| j| jS)a Example: ```python >>> from transformers import AutoTokenizer, GraniteForCausalLM >>> model = GraniteForCausalLM.from_pretrained("meta-granite/Granite-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-granite/Granite-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrEr1rrrrrrv)rrr)lossrrr6rr)r^rrrrrrslicerlogits_scaling loss_functionrrrr6r)rrrrEr1rrrrrrrvr rHrr6 slice_indicesrr"s r(rzGraniteForCausalLM.forwards/D2C1N-TXT_T_TqTq$8$D $++JjJj  ,64:: , )%+'/!5) ,  ,  11 8B>SV8W~ot4]k mA}a,?@A$++444  %4%%pVFt{{OeOepiopD%#33!//))   r*) NNNNNNNNNNr)rrr_tied_weights_keys_tp_plan_pp_planrdrrrrrr#rrrr listrrrrrrrrrs@r(rrs*+=)H_-z:;H151537KO59-1$(,0/35934C E,,-C !.C u//0 C "%tE4E4E/F(F"GH C   1 12 C ))*C D>C $D>C 'tnC !!1!12C c5<</0C +,C  C C r*r)rrr)Nr)ry);typingrrrr#r activationsr cache_utilsr r generationr integrationsr masking_utilsr modeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrconfiguration_graniter get_loggerrr r)r5rrr@Modulerr[r]rrrrrrr__all__rr*r(r<s,-, !.)7/9OKF&RR/0   H %(6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%'(%4C)ryyC)LY'JRYYJ(J(   J4JZ_$