rh \NddlmZddlmZmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZddlmZddlmZedGddej@Z!Gddej@Z"Gddej@Z#eedGddeZ$eedGdd eZ%eGd!d"eZ&ed#Gd$d%e&Z'ed&Gd'd(e&e Z(gd)Z)y)*) dataclass)OptionalUnionN)nn)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)FlashAttentionKwargs)BaseModelOutputWithPast ModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple) AutoModel)Mistral3ConfigRMSNormc,eZdZdfd ZdZdZxZS)Mistral3RMSNormct|tjt j ||_||_y)z> Mistral3RMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/mistral3/modeling_mistral3.pyrzMistral3RMSNorm.__init__+s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetorfloat32powmeanrsqrtr"r!)r# hidden_states input_dtypevariances r'forwardzMistral3RMSNorm.forward3sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r(c^t|jjd|jS)Nz, eps=)tupler!shaper"r#s r' extra_reprzMistral3RMSNorm.extra_repr:s* ))*+6$2G2G1HIIr()gư>)__name__ __module__ __qualname__rr5r: __classcell__r&s@r'rr)s$;Jr(rcxeZdZdZdeffd ZdejdejdejfdZxZ S)Mistral3PatchMergerz< Learned merging of spatial_merge_size ** 2 patches configc"t|||_|jj}|j |_|jjj |_tj||j dzz|d|_ y)NrFbias) rrrB vision_configr$spatial_merge_size patch_sizerLinear merging_layer)r#rBr$r&s r'rzMistral3PatchMerger.__init__Csr  **66 "(";";++33>>YY{T5L5La5O'OQ\chir(image_features image_sizesreturnc|Dcgc]&}|d|jz|d|jzf(}}|Dcgc] \}}||z }}}|jd}g}t|j|D]\} } || \}}| j |||j dddj d} tjjj| |j|j} | j ||jdzzdj} |j| tj|d}|j|}|Scc}wcc}}w)Nrrr*r) kernel_sizestridedim)rHr8 enumeratesplitviewpermute unsqueezerr functionalunfoldrGtappendcatrJ) r#rKrL image_sizehwtokens_per_imagedpermuted_tensor image_index image_tokens image_gridgrids r'r5zMistral3PatchMerger.forwardLslcn U_Z]doo -z!}/O P  /::daAE::   $)2>3G3GHX3Y)Z ) %K{+DAq%**1a3;;Aq!DNNqQJ88&&--(?(?H_H_.D99Q!8!8!!;;R@BBDD  " "4 ( )?:++N;) ;s +E"E') r;r<r=__doc__rrrTensorr5r>r?s@r'rArA>s?j~jellRWR^R^r(rAc\eZdZdeffd Zdej dej fdZxZS)Mistral3MultiModalProjectorrBct|t|jj|j j |_t||_ t|jtrdnt|j}tj|jj|z|j j|j |_t$|j&|_tj|j j|j j|j |_y)N)r%rrD)rrrrFr$ text_config rms_norm_epsnormrA patch_merger isinstancevision_feature_layerintlenrrImultimodal_projector_biaslinear_1rprojector_hidden_actactlinear_2)r#rBnum_feature_layersr&s r'rz$Mistral3MultiModalProjector.__init__es #F$8$8$D$D&J\J\JiJij /7",V-H-H#"NQTWX^XsXsTt  , ,/A A    * *11  &556    * *F,>,>,J,JQWQqQq  r(rKrLc|j|}|j||}|j|}|j|}|j |}|SN)rnrorurwrx)r#rKrLr2s r'r5z#Mistral3MultiModalProjector.forwardusR>2**>;G n5 /  m4 r() r;r<r=rrrrhr5r>r?s@r'rjrjds* ~ ellr(rjzT Base class for Mistral3 causal language model (or autoregressive) outputs. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<y) Mistral3CausalLMOutputWithPasta] loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_valuesr2 attentionsimage_hidden_states)r;r<r=rgrrr FloatTensor__annotations__rrlistr2r7rrr(r'r~r~~s )-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8