rh<ddlZddlmZddlmZmZmZddlZddlm cm Z ddlm Z ddl m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZmZmZd dlmZm Z m!Z!eeGddeZ"edGdde jFZ$Gdde jFZ%Gdde jFZ&Gdde jFZ' d9de jFdejPdejPdejPdeejPde)d e)fd!Z*Gd"d#e jFZ+Gd$d%eZ,Gd&d'e jFZ-Gd(d)e jFZ.eGd*d+eZ/ed,-Gd.d/e/Z0ed0-Gd1d2e/Z1d3ejPd4ejPfd5Z2eGd6d7e/Z3gd8Z4y):N) dataclass)AnyCallableOptional)nn)ACT2FN)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputauto_docstringcan_return_tuple) Aimv2ConfigAimv2TextConfigAimv2VisionConfigceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd Zy) Aimv2Outputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Aimv2TextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Aimv2VisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`Aimv2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Aimv2VisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputreturncHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))rr N)getattrto_tuple).0kselfs {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/aimv2/modeling_aimv2.py z'Aimv2Output.to_tuple..Hs=  LLDGRYZ^`aRbRkRkRm m s-0)tuplekeysr(s`r)r%zAimv2Output.to_tupleGs# YY[   )__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrrr r+rr%r.r)rr)s&)-D(5$$ %,48hu001837OXe//07/3K%++,304L(5,,-448186:3: %* r.rRMSNormc,eZdZdfd ZdZdZxZS) Aimv2RMSNormct|tjt j ||_||_y)z; Aimv2RMSNorm is equivalent to T5LayerNorm N)super__init__r Parameterr3onesweightvariance_epsilon)r( hidden_sizeeps __class__s r)r<zAimv2RMSNorm.__init__Ps1 ll5::k#:; #r.c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor3float32powmeanrsqrtr@r?)r( hidden_states input_dtypevariances r)forwardzAimv2RMSNorm.forwardXsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r.c^t|jjd|jS)Nz, eps=)r+r?shaper@r-s r) extra_reprzAimv2RMSNorm.extra_repr_s* ))*+6$2G2G1HIIr.)gư>)r/r0r1r<rQrT __classcell__rCs@r)r9r9Ns$;Jr.r9c$eZdZfdZdZxZS)Aimv2MLPct|||_|j|_|j|_t j |j|j|j|_t j |j|j|j|_ t j |j|j|j|_ t|j|_ y)Nbias)r;r<configrAintermediate_sizerLinearmlp_bias gate_projup_proj down_projr hidden_actact_fnr(r\rCs r)r<zAimv2MLP.__init__ds  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r.c|j|j|j||j|z}|SN)rbrdr`ra)r(xrbs r)rQzAimv2MLP.forwardns6NN4;;t~~a/@#ADLLQRO#ST r.)r/r0r1r<rQrUrVs@r)rXrXcs 0r.rXceZdZdeffd ZedddejfdejfdZ dejdejfd Z xZ S) Aimv2VisionEmbeddingsr\cBt|||_|j|_t j |j |j|j|j|_t|j|j|_ |j|jzdz}|jjs%t j||j|_|j!dt#j$|j'ddy)N) kernel_sizestriderE position_idsrrFF persistent)r;r<r\ patch_sizerConv2d num_channelsrA patch_embedr9 rms_norm_epsrms_norm image_size is_native Embeddingposition_embeddingregister_bufferr3arangeexpand)r(r\ num_patchesrCs r)r<zAimv2VisionEmbeddings.__init__ts   ++99   !3!3ARAR[a[l[l %V%7%79L9LM ((F,=,==!C {{$$&(ll;@R@R&SD # ^U\\+-F-M-Mg-Vchir.g@cpur!c:tjt|||}tjt|||}tj||d\}}|dz}tj||||z } d|| zz } |j d| dddfz} |j d| dddfz} tj | j | j| j | jgddddddfS) NrHdevicexy)indexingg?).Nrdim)r3r}intmeshgridflattenconcatsincos) heightwidth embed_dim temperaturerrHgrid_wgrid_hpos_dimomegaout_hout_ws r)"build_2d_sincos_position_embeddingz8Aimv2VisionEmbeddings.build_2d_sincos_position_embeddingsc%jfEc&kvFFq. WE&AGK{E)* +eD!Gn< +eD!Gn<||UYY[%))+uyy{EIIKPVWXY]_`bcYcddr. pixel_valuesc|j\}}}}|j|jdjdd}|j |}|j j rY|j||jz||jz|j j|j|j}n|j|j}||z}|S)NrEr)rrrH)sizerur transposerwr\ryrrrrArrHr{rn)r(r_rrrN pos_embeds r)rQzAimv2VisionEmbeddings.forwards*//11fe((6>>qAKKAqQ  m4 ;; ??$//)(++11$++#)) @I//0A0ABI% 1 r.) r/r0r1rr< staticmethodr3rJTensorrrQrUrVs@r)rjrjss] j0 j!$'%u}}e ee ELLU\\r.rjc eZdZdeffd Z ddeejdeejdeejdejfdZ xZ S) Aimv2TextEmbeddingsr\cNt||j}tj|j ||_tj|j||_|jdtj|jjddy)NrnroFrp) r;r<rArrz vocab_sizetoken_embeddingmax_position_embeddingsr{r|r3r}r~)r(r\rrCs r)r<zAimv2TextEmbeddings.__init__s && !||F,=,=yI"$,,v/M/My"Y  ELL)G)GHOOPWXej  r. input_idsrn inputs_embedsr!c8||jdn|jd}|jjjd}||kDrtd|d|||jddd|f}||j |}|j|}||z}|S)NrFrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rSr{r? ValueErrorrnr)r(rrnr seq_lengthmax_position_embeddingposition_embeddings embeddingss r)rQzAimv2TextEmbeddings.forwards -6,AY__R(}GZGZ[]G^ !%!8!8!?!?!E!Ea!H . .d,<=S> )T^^ ;MdnnM]^NN#2' ]]D( // iiV__U iiV__U iiV__U  $..$..vW r.rNrr!c :|j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rt|jj} | |||| ||j|j|jsdn |j\} } | j!|||j#} |j%| } | | fS)z#Input shape: Batch x Time x ChannelrrEeager)rrr)rSrrrviewrrrrr\_attn_implementationrrrrrreshaperr) r(rNrr batch_sizerrqueriesr,valuesattention_interfacerrs r)rQzAimv2Attention.forwardsa-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0 L((r.rg) r/r0r1r2r<r3rrr+rQrUrVs@r)rrsVGX,26$)||$)!.$) u||Xell33 4 $)r.rc eZdZdeffd Z ddej deej deede ej ej ffdZ xZ S) Aimv2EncoderLayerr\ct|t||_t ||_t |j|j|_ t |j|j|_ yrg) r;r<r attentionrXffnr9rArv rms_norm1 rms_norm2res r)r<zAimv2EncoderLayer.__init__$sZ '/F#%f&8&8&:M:MN%f&8&8&:M:MNr.rNroutput_attentionsr!c|j|}|j||\}}||z}|j|}|j|}||z}|r||fS|dfS)N)rNr)rrrr)r(rNrrnorm_hidden_statesrr mlp_outputs r)rQzAimv2EncoderLayer.forward+sv "^^M:$(NNASdrN$s! \% 3 !^^M:XX01 % 2 0A |,\ W[G\\r.NF) r/r0r1rr<r3rrboolr+rQrUrVs@r)rr#smO0O26,1 ]||]!.]$D> ] u||U\\) * ]r.rc xeZdZdZdeffd Ze d deejdee dee de fdZ xZ S) Aimv2Encoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Aimv2EncoderLayer`]. Args: config: Aimv2Config r\ct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) r;r<r\r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r(r\rrCs r)r<zAimv2Encoder.__init__EsO  mmfNfNfHg$h1%6v%>$hi &+#%isA#rroutput_hidden_statesr!c||n|jj}||n|jj}|rdnd}|rdnd}|}|jD]&}|r||fz}||||} | d}|s|| dfz}(|r||fz}t |||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr6)rrr)last_hidden_staterN attentions)r\rrrr ) r(rrrrencoder_statesall_attentionsrN encoder_layer layer_outputss r)rQzAimv2Encoder.forwardLs<2C1N-TXT_T_TqTq$8$D $++JjJj  40d% ![[ FM#!/=2B!B)"3M *!,M !/=3C2E!E F +}.>>N+(%  r.r)r/r0r1r2rr<rrr3rrr rQrUrVs@r)rr<sl,{,26,0/3 < !.< $D> < 'tn <  < < r.rc\eZdZdeffd Zdej dej fdZxZS)Aimv2AttentionPoolingHeadr\c&t||j|_|j|_t j |j|j|j|_t j |j|j|j|_ t jtjdd|j|_ t j |j|jd|_y)NrZrT)r;r<rArrrr^rrrr=r3zeros cls_token output_projres r)r<z"Aimv2AttentionPoolingHead.__init__s !--33ii 0 0$2B2BY ii 0 0$2B2BY ekk!Q8H8H&IJ99T%5%5t7G7GdSr.rNr!c|j\}}}|jj|dd}|j|j |||j ||j z}|j |j |||j ||j z}|j |d|j ||j z}|jdddd}|jdddd}|jdddd}tj|||} | jddj |d|} | jd} |j| } | S)NrFrrrErr) rSrr~rrrrpermuteFscaled_dot_product_attentionrrLr) r(rNrseq_len hidden_dimrrrrroutputs r)rQz!Aimv2AttentionPoolingHead.forwardsH*7*=*=' GZNN))*b"= kk-(00WdnnV`dhdrdrVrs M*22:wXbfjftftXtu!!*at~~A]^kk!Q1% aAq) aAq)44UCG !++Aq199*aT !&&1&- !!+. r.) r/r0r1rr<r3rrQrUrVs@r)rrs- T0 TU\\ellr.rcJeZdZUdZeed<dZdZgdZdZ dZ dZ fdZ xZ S)Aimv2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. The model is only intended for inference and doesn't support finetuning. r\aimv2T)rrrjrct||t|dr^t|jt j r9|jjjtjdyyt|tr<|jjjd|jjyy)N logit_scaleg$I$I,@r)rLstd)r; _init_weightshasattr isinstancer rr=datafill_mathlogrrnormal_r\initializer_range)r(rrCs r)rz"Aimv2PreTrainedModel._init_weightss f% 6= )&,,bll;""''--dhhx.@A<  9 :    ! ! ) )s 8U8U ) V;r.)r/r0r1r2rr5base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attnrrUrVs@r)r r sC &*# NWWr.r zL The Vision model from AIMv2 without any head or projection on top. ) custom_introc eZdZUeed<dZdeffd ZdejfdZ e e d de e jde ede edefd ZxZS) Aimv2VisionModelr\rc6t||||_t||_t ||_t|j|j|_ |j|_ |jrt||_ |jyrg)r;r<r\rjrrencoderr9rArvrwuse_headrhead post_initres r)r<zAimv2VisionModel.__init__sq   /7#F+ $V%7%79L9LM  ==1&9DI r.r!c.|jjSrg)rrur-s r)get_input_embeddingsz%Aimv2VisionModel.get_input_embeddingss***r.rrrcd||n|jj}||n|jj}|j|}|j |||}|d}|j |}|j r|j|nd}t|||j|jS)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Siglip2VisionModel >>> model = Aimv2VisionModel.from_pretrained("apple/aimv2-large-patch14-native") >>> processor = AutoProcessor.from_pretrained("apple/aimv2-large-patch14-native") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled features ```N)rrrrr pooler_outputrNr) r\rrrr!rwr"r#rrNr) r(rrrrrNencoder_outputsrr)s r)rQzAimv2VisionModel.forwards:2C1N-TXT_T_TqTq$8$D $++JjJj  5 ,,'/!5'  ,A. MM*;<8<  "344 )/')77&11   r.r)r/r0r1rr5main_input_namer<rModuler&rrrr3rrrrQrUrVs@r)rrs $O 0 +bii+26,0/3 2 !.2 $D> 2 'tn 2 $ 2 2 r.rzJ The text model from AIMv2 without any head or projection on top. c eZdZdZdeffd ZdejfdZdZ e e d de e jde ed e edefd ZxZS) Aimv2TextModelrr\ct||||_t||_t ||_t|j|j|_ |j|_ |jyrg) r;r<r\rrrr!r9rArvrw eos_token_idr$res r)r<zAimv2TextModel.__init__$sa   -f5#F+ $V%7%79L9LM "// r.r!c.|jjSrgrrr-s r)r&z#Aimv2TextModel.get_input_embeddings/s...r.c&||j_yrgr2)r(rs r)set_input_embeddingsz#Aimv2TextModel.set_input_embeddings2s*/'r.rrrc2||n|jj}||n|jj}|j|}|j\}}}t j |t j|j} | jdj|d} |t|j|| || d}|j||||} | d} |j| } | t j | jd| j|jt j| j|j k(jj#df} t%| | | j&| j(S) NrrrF)r\ input_embedsrnrcache_positionpast_key_values)rrrr)rrr()r\rrrrSr3r}longr unsqueezer~r r!rwrIrr0argmaxrrNr)r(rrrrrNrrrr7rnr*r pooled_outputs r)rQzAimv2TextModel.forward5s2C1N-TXT_T_TqTq$8$D $++JjJj  2 !.!4!4 GQgUZZ H\H\]%//299*bI  %/{{*)-- $ N,,')/!5 ' ,A. MM*;<* LL*003 0 'tn 0 $ 0 0 r.r.tensorr!ctj|d}tj|dd}tj|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 rErFT)rrGg?)r3rKsum)r= square_tensor sum_tensor normed_tensors r)_get_vector_normrCjs< IIfa(M=b$?JIIj#.M r.ceZdZUeed<gdZdeffd Ze ddee jdee jdee jdee dee d e jf d Z e dd ee jdee dee d e d e jf d Zee ddee j d ee jdee jdee dee d ef dZxZS) Aimv2Modelr\)rrrjct|||j|_|jj|_|j j|_tj|j|_ tj|j |_ tj|j |jd|_tj|j|jd|_tj"t%j&|j(j*|_t/j0|j2|_|j7y)NFrZ)r;r<projection_dim vision_configrAvision_embed_dim text_configtext_embed_dimr _from_config vision_modelr. text_modelrr^visual_projectiontext_projectionr=r3r=r\logit_scale_init_valuer rrmax_logit_scalemax_log_logit_scaler$res r)r<zAimv2Model.__init__zs  $33 & 4 4 @ @$00<<,99&:N:NO(55f6H6HI!#4+@+@$BUBU\a!b!yy)<)Q>QX]^<< T[[5W5W(XY#'88F,B,B#C  r.rrrnrrr!c||n|jj}||n|jj}|j|||||}|j}|j |}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Aimv2TextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, Aimv2Model >>> model = Aimv2Model.from_pretrained("openai/aimv2-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/aimv2-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```)rrrnrr)r\rrrNr)rP) r(rrrnrr text_outputsr< text_featuress r)get_text_featureszAimv2Model.get_text_featuress42C1N-TXT_T_TqTq$8$D $++JjJj 48??)%/!5 4C4 %22 ,,]; r.rinterpolate_pos_encodingc||n|jj}||n|jj}|j||||}|j}|j |}|S)aI Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Aimv2VisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Aimv2Model >>> model = Aimv2Model.from_pretrained("openai/aimv2-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/aimv2-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```)rrrrX)r\rrrMr)rO)r(rrrrXvision_outputsr<image_featuress r)get_image_featureszAimv2Model.get_image_featuress>2C1N-TXT_T_TqTq$8$D $++JjJj 6:5F5F%/!5%= 6G6 '44 // >r.cp||n|jj}||n|jj}|j|||}|j ||||}|j }|j |}|j } |j| } |t|z }| t| z } |jjd|jjj| j} | | z|jz} | j} t!| | | |||S)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Aimv2Model >>> model = Aimv2Model.from_pretrained("apple/aimv2-large-patch14-224-lit") >>> processor = AutoProcessor.from_pretrained("apple/aimv2-large-patch14-224-lit") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```)rrr)rrrrr)rrrrrr )r\rrrMrNr)rOrPrCr clamprSexprIrtr) r(rrrrrrZrUrrr rrs r)rQzAimv2Model.forwardsUB2C1N-TXT_T_TqTq$8$D $++JjJj 6:5F5F%/!56G6  48??)/!5 4C4 &33 --l; "00 **;7 $&6|&DD !$4[$AA &&,,S$2J2JKOOQTTU`UgUgh &4 8HH*,,.-+#%* .   r.)NNNNN)NNNF)r/r0r1rr5rr<rrr3rrr4rWr\rrrrQrUrVs@r)rErEus ]{$-115/3,0/3 )ELL))!.)u||, ) $D> ) 'tn )   ))V59,0/3). -u001-$D>-'tn - #' -    --^154815,0/3 F E,,-F u001F !. F $D> F 'tn F  F F r.rE)rrEr r.)r)5r dataclassesrtypingrrrr3torch.nn.functionalrrr activationsr integrationsr masking_utilsr modeling_layersr modeling_outputsr rmodeling_utilsrrutilsrrrconfiguration_aimv2rrrrr,r9rXrjrrfloatrrrrrr rr.rCrE__all__r6r.r)rns. !** !7/9KFBBPP   +     FY'J299J(J( ryy  1BII1h%"))%^% II% <<% % << % U\\* %  %%.:)RYY:)z]2]2M 299M ` DW?WW8 I +I  I X F )F  F RU\\ellz %z z z Wr.