rh:ddlmZddlmZmZmZddlmZddlm Z m Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-e(r ddl.Z.ddl.m/Z/edGdde/j`Z1Gdde/j`Z2Gdde/j`Z3Gdd e/j`Z4Gd!d"e/j`Z5d#Z6Gd$d%e/j`Z7Gd&d'e/j`Z8Gd(d)e/j`Z9d*Z:dUd+Z;d,e.jxd-e=d.e.jxfd/Z> dVd0e/j`d1e.jxd2e.jxd3e.jxd4ee.jxd5e?d6e?d7e e"fd8Z@Gd9d:e/j`ZAGd;deZCe#Gd?d@eZDGdAdBe/j`ZEe#GdCdDeCZFe#GdEdFeCe ZGee#dGHGdIdJeZHee#dKHGdLdMeZIe#dNHGdOdPeDZJe#dQHGdRdSeDe ZKgdTZLy)W) dataclass)CallableOptionalUnion)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)check_model_inputs)is_torch_available) AutoModel) AriaConfigAriaTextConfigN)nnRMSNormc,eZdZdfd ZdZdZxZS)AriaTextRMSNormct|tjt j ||_||_y)z> AriaTextRMSNorm is equivalent to T5LayerNorm N)super__init__r" Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/aria/modeling_aria.pyr(zAriaTextRMSNorm.__init__1s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor*float32powmeanrsqrtr-r,)r. hidden_states input_dtypevariances r2forwardzAriaTextRMSNorm.forward9sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r3c^t|jjd|jS)Nz, eps=)tupler,shaper-r.s r2 extra_reprzAriaTextRMSNorm.extra_repr@s* ))*+6$2G2G1HIIr3)gư>)__name__ __module__ __qualname__r(r@rE __classcell__r1s@r2r%r%/s$;Jr3r%c(eZdZdZfdZdZxZS)AriaProjectorMLPa! Feed-Forward Network module for the Aria Projector. Args: in_features (`int`): Input embedding dimension. hidden_features (`int`): Hidden dimension of the feed-forward network. output_dim (`int`): Output dimension. ct|tj||d|_tj||d|_t d|_y)NFbiasgelu_new)r'r(r"Linear linear_in linear_outract)r. in_featureshidden_features output_dimr1s r2r(zAriaProjectorMLP.__init__QsB ;eL))OZeL*%r3ch|j|j|}|j|}|SN)rTrRrS)r.r=s r2r@zAriaProjectorMLP.forwardWs- !>?  6 r3rFrGrH__doc__r(r@rIrJs@r2rLrLDs & r3rLc6eZdZdZddedeffd ZddZxZS)AriaCrossAttentionzv Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. config dropout_ratecBt||jj}|jj}||_t j||d|_t j||d|_ t j||d|_ t j||d|_ t j|||_ t j||_t j ||_t j ||_y)NFrNT) batch_first)r'r( vision_configr/num_attention_heads num_headsr"rQq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)r.r^r_r/rdr1s r2r(zAriaCrossAttention.__init__fs **66 ((<< "ii [uE ii [uE ii [uE !33KX\]ii [9 zz,/ ,,{3\\+6r3c|j|j|}|j|}|j|}|j |}|j ||||\}}|j |j|}|S)a Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)rernrorfrgrirlrj) r.key_value_statesr=rrquerykeyvalue attn_output_s r2r@zAriaCrossAttention.forwardws  DOOM:;--.>?kk*+ ,-,,UC),T Qll4;;{#;< r3)rrY) rFrGrHr[r floatr(r@rIrJs@r2r]r]]s 7z77"r3r]cheZdZdZdeffd ZddejdeejfdZ xZ S) AriaProjectora  Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. r^ct||j|_|jj |_|jj|_|jj |_ |jj |_ |jj |_ tjtj |j"|j |_t'||_tj*|j |_t/|j |j|j|_yrY)r'r(projector_patch_to_query_dictpatch_to_query_dictrbr/rUrcrdkv_dim text_configrVrWr"r)r*zeros'max_value_projector_patch_to_query_dictrtr] cross_attnrmrnrL feed_forwardr.r^r1s r2r(zAriaProjector.__init__s #)#G#G !//;;--AA**66 %11== ,,88\\%++f.\.\^b^n^n"op ,V4,,t'7'78,T-=-=t?S?SUYUdUder3rsrrc4|jd|jd}}||jvr*td|d|jjd|j|}|jd|j dj |dd}|M|j|jd}|j djd|jdd}|j|||}|j|j|}|S) a Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). rrzNumber of patches z: not found in patch_to_query_dict amongst possible values .Nr5rq)rCr~KeyErrorkeysrt unsqueezerepeatrepeat_interleaverdexpandsizerrrn) r.rsrr batch_size num_patches query_numqueries attention_outouts r2r@zAriaProjector.forwards6#3"8"8";=M=S=STU=VK d66 6$[M1klpmEmEmJmJmLlMMNO ,,[9 **Zi(2215<? r3rY) rFrGrHr[r r(r*Tensorrr@rIrJs@r2r{r{s7ff( %,,AWr3r{c.eZdZdZdeffd ZdZxZS)AriaSharedExpertsMLPa/ Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. r^c t|||_|j|_|j|j z|_t j|j|j|j|_ t j|j|j|j|_ t j|j|j|j|_ t|j|_y)NrN)r'r(r^r/intermediate_sizemoe_num_shared_expertsr"rQmlp_bias gate_projup_proj down_projr hidden_actact_fnrs r2r(zAriaSharedExpertsMLP.__init__s  !--!'!9!9F++auzz:K:R:RSK ;0A"BCN0034 !*- Q/eC(ll6>*#=>uS   Mr3c(eZdZdZfdZdZxZS)AriaGroupedExpertsGemmaP Grouped GEMM (General Matrix Multiplication) module for efficient expert computation. This module utilizes the grouped_gemm library (https://github.com/fanshiqing/grouped_gemm) for optimized performance. If the grouped_gemm library is not installed, it gracefully falls back to a sequential GEMM implementation, which may be slower but ensures functionality. Args: in_features (`int`): Number of input features. out_features (`int`): Number of output features. groups (`int`): Number of expert groups. ct|||_||_||_t j tj||||_ yrY) r'r(rUrgroupsr"r)r*emptyr,)r.rUrrr1s r2r(zAriaGroupedExpertsGemm.__init__sB &( ll5;;v{L#QR r3cLt||j|jS)au Perform grouped matrix multiplication. Args: input (`torch.Tensor`): Input tensor of shape (num_tokens, in_features). tokens_per_expert (`torch.Tensor`): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). )rr,cpu)r.inputrs r2r@zAriaGroupedExpertsGemm.forward&s''  KK  ! ! #  r3rZrJs@r2rrs S r3rc2eZdZdZdeddffd ZdZxZS)AriaGroupedExpertsMLPz Grouped MLP module for Mixture of Experts. Args: config (`AriaTextConfig`): Configuration object for the model. r^returnNct|||_t|j|j dz|j |_t|j |j|j |_y)Nr) r'r(r^rr/rmoe_num_expertsfc1fc2rs r2r(zAriaGroupedExpertsMLP.__init__Csa  )&*<*V>VYZ>Z\b\r\rs)&*B*BFDVDVX^XnXnor3c|j||}tj|dd\}}tjj ||z}|j ||}|S)a5 Forward pass of the Grouped MLP. Args: permuted_tokens (torch.Tensor): Permuted input tokens. tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor after passing through the MLP. rr5r)rr*chunkr" functionalsilur)r.permuted_tokensr fc1_output projectiongate fc2_outputs r2r@zAriaGroupedExpertsMLP.forwardIs\XXo/@A  ;;z1"= D]]'' 3d: XXj*;< r3rrJs@r2rr:s#p~p$p r3rc`eZdZdZdeffd ZdejdejfdZxZ S)AriaTextMoELayerz Aria Text Mixture of Experts (MoE) Layer. This layer applies a gating mechanism to route input tokens to different experts. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. r^ct|tj|j|j d|_t||_t||_ ||_ yNFrN) r'r(r"rQr/rrouterrexpertsrshared_expertsr^rs r2r(zAriaTextMoELayer.__init__gsO ii 2 2F4J4JQVW ,V4 26: r3r=rc|j}|jd|jd}|j|}t j ||j jd\}}tjj|d}|j}t j|jjtj|j j d|j j dz j|}|} | jd} t j"| } |j%d| |j jz} |j'| |} t j(|jd|j jz| jdf| j| j*}|j-d| | |jd|j j| jd}||j/dzj1dj|}|j3|j|}||zS)a. Forward pass of the MoE Layer. Args: hidden_states (`torch.Tensor`): Input tensor of shape (batch_size, sequence_length, hidden_size). Returns: torch.Tensor: Output tensor after passing through the MoE layer. Process: 1. Route tokens to experts using the router. 2. Permute tokens based on routing decisions. 3. Process tokens through experts. 4. Unpermute and combine expert outputs. 5. Add shared expert output to the final result. r5r)krrr)binsminmaxr)rCviewrrr*topkr^moe_topkr"rsoftmaxr7histcflattenr8r9rargsort index_selectrrr index_copy_rsumr)r.r=original_shapelogits top_logits top_indicesscoresoriginal_dtyperindicesflatten_indicessorted_indicesr expert_outputunpermuted_tokensrshared_expert_outputs r2r@zAriaTextMoELayer.forwardos$',,%**2}/A/A"/EF ]+"'**Vt{{7K7KQR"S K&&zr&:$**!KK    ! $ $U]] 3,, ++a/  "^   ",,r*7'44Q$++J^J^8^_ _6GH "KK \\!_t{{33 3]5G5G5J K%% ''  %%aG-222t{{7K7K]M_M_`aMbc#f&6&6r&::??A?FKKN[ $22=3E3En3UV,,,r3) rFrGrHr[r!r(r*rr@rIrJs@r2rr\s/~9-U\\9-ell9-r3rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr5rr)rCr*r)rx1x2s r2 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r3c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )rr)qrcossin position_ids unsqueeze_dimq_embedk_embeds r2apply_rotary_pos_embrsY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r3r=n_reprc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rCrreshape)r=rbatchnum_key_value_headsslenhead_dims r2 repeat_kvr so 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr3modulertrurvattention_maskscalingrlkwargsc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nrrr5)rr7)ptrainingr)r num_key_value_groupsr*r transposerCr"rrr9r8r7rlr contiguous) r rtrurvr r rlr  key_states value_states attn_weights causal_maskrws r2eager_attention_forwardrs3 ; ;$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r3r=position_embeddingsr past_key_valuecache_positionr rc 4|jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7rt|jj}||| | | |f|jsdn |j|j d|\}}|j"g|dj%}|j'|}||fS)Nr5rr)rrr&eager)rlr )rCrrerrrfrgrupdaterrr^_attn_implementationrrrr rrr")r.r=r$r r%r&r  input_shape hidden_shape query_statesrrrr cache_kwargsattention_interfacerwrs r2r@zAriaTextAttention.forward s$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'))!. )) ! )) !!1!12 ))+,)) u||U\\) *))r3rc,eZdZdZdedeffd Z ddejde ejde ejde e d e e d e ejd e e ejejfd eed e ejfdZxZS)AriaTextDecoderLayerag Aria Text Decoder Layer. This class defines a single decoder layer in the language model, incorporating self-attention and Mixture of Experts (MoE) feed-forward network. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. layer_idx (`int`): Index of the layer. r^rc t||j|_t|||_t ||_t|j|j|_ t|j|j|_ y)N)r^rr0) r'r(r/r self_attnrmlpr% rms_norm_epsinput_layernormpost_attention_layernormr#s r2r(zAriaTextDecoderLayer.__init__Fsm !--*&IN#F+.v/A/AvGZGZ[(78J8JPVPcPc(d%r3r=r rr% use_cacher&r$r rc |} |j|}|jd|||||||d|\}} | |z}|} |j|}|j|}| |z}|S)N)r=r rr%r<r&r$)r:r7r;r8) r.r=r rr%r<r&r$r residualrxs r2r@zAriaTextDecoderLayer.forwardOs!,,]; )4>>  ')%)) 3     q!=0 !55mD /  =0 r3)NNNFNN)rFrGrHr[r!r1r(r*rrr2r boolrBrrr@rIrJs@r2r4r49s e~e#e2637*.$)59KO||!.u//0  !  D> !!1!12&eELL%,,,F&GH+, u|| r3r4cTeZdZUeed<dZddgZdZdZdZ dZ dZ e e dZfd ZxZS) AriaTextPreTrainedModelr^modelr4rTpast_key_valuesFr= attentionsct||t|tr<|jj j d|jjyy)Nr))r;std) r' _init_weights isinstancerr,datanormal_r^initializer_ranger.r r1s r2rIz%AriaTextPreTrainedModel._init_weightssG f% f4 5 MM   & &CT[[5R5R & S 6r3)rFrGrHr!__annotations__base_model_prefix_no_split_modulessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_attention_backendr4r_can_record_outputsrIrIrJs@r2rBrBqsR /1IJ&*#"3 N"&-' TTr3rBc\eZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZfdZxZS) AriaPreTrainedModelr^TAriaDecoderLayerrDFrEct||t|tr@tj j |j|jjyy)N)rH) r'rIrJr{r"init trunc_normal_rtr^rMrNs r2rIz!AriaPreTrainedModel._init_weightssD f% fm , GG ! !&,,DKK4Q4Q ! R -r3)rFrGrHr rOrPrRrQrSrTrU_supports_flex_attn_can_compile_fullgraphrVr4rrWrIrIrJs@r2rYrYs^ &*#+,#4"5N""&-' SSr3rYc^eZdZddeffd Zej edZxZ S)AriaTextRotaryEmbeddingr^ct|t|drUt|jt r;|jj d|jj d|_nd|_|j|_ |j|_ ||_ t|j|_ |j|j|\}|_|jd|d|j |_y)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r'r(hasattrrJrddictgetremax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr^r rope_init_fnattention_scalingregister_bufferrhoriginal_inv_freq)r.r^rrhr1s r2r(z AriaTextRotaryEmbedding.__init__s  6> *z&:M:Mt/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.rY) rFrGrHr!r(r*no_gradrr@rIrJs@r2rbrbs3/~/"U]]_ < 8 +,8  !8 8 r3rcleZdZdgZddiZddgdgfiZdeffd ZdZd Z e dd e e jd e e jd e e jd e ede e j de e jde ede e jdeee jfdeedefdZxZS)AriaTextForCausalLMlm_head.weightlm_head colwise_repr=rr^ct||t||_|j|_t j |j|jd|_|jyr) r'r(rrCrr"rQr/rrrs r2r(zAriaTextForCausalLM.__init__sU  "6*  ++yy!3!3V5F5FUS  r3c||_yrYrCr.decoders r2 set_decoderzAriaTextForCausalLM.set_decoders  r3c|jSrYrrDs r2 get_decoderzAriaTextForCausalLM.get_decoder!s zzr3rr rrDrlabelsr<r&logits_to_keepr rc z|jd|||||||d| } | j} t| tr t | dn| } |j | dd| ddf}d}|*|j d|||jjd| }t||| j| j| jS)a Example: ```python >>> from transformers import AutoTokenizer, AriaTextForCausalLM >>> model = AriaTextForCausalLM.from_pretrained("meta-aria_text/AriaText-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-aria_text/AriaText-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rr rrDrr<r&NrrrlossrrDr=rFr>) rCrrJr1slicer loss_functionr^rrrDr=rF)r.rr rrDrrr<r&rr outputsr= slice_indicesrrs r2r@zAriaTextForCausalLM.forward$s>,64:: , )%+') ,  ,  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD%#33!//))   r3) NNNNNNNNr)rFrGrH_tied_weights_keys_tp_plan_pp_planr!r(rrrrr*r2rr rr@rr1rrrr@rIrJs@r2rrs9*+=)H_-z:;H~151537+/59-1$(59348 E,,-8 !.8 u//0 8 "% 8   1 12 8 ))*8 D>8 !!1!128 c5<</08 +,8  8 8 r3rzP Base class for Aria causal language model (or autoregressive) outputs. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<y) AriaCausalLMOutputWithPasta] loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. NrrrDr=rFimage_hidden_states)rFrGrHr[rrr*rrOrrDlistr=rBrFrr>r3r2rr`s )-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8