rhk`ddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddl m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*edGdde jVZ, dDde jVdejZdejZdejZdeejZde.de.fdZ/Gdde jVZ0e#Gd d!eZ1ee#d"#Gd$d%eZ2Gd&d'e jVZ3Gd(d)e jVZ4Gd*d+e jVZ5e jle,d,Z7Gd-d.eZ8Gd/d0e jVZ9e#Gd1d2e1Z:e#Gd3d4eZ;Gd5d6e jVZee#d=#Gd>d?e!Z?e#d@#GdAdBe;eZ@gdCZAy)EN) dataclass)CallableOptionalUnion)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple torch_int) AutoModel)InternVLConfigInternVLVisionConfigRMSNormc,eZdZdfd ZdZdZxZS)InternVLVisionRMSNormct|tjt j ||_||_y)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/internvl/modeling_internvl.pyr#zInternVLVisionRMSNorm.__init__.s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor&float32powmeanrsqrtr)r()r* hidden_states input_dtypevariances r.forwardzInternVLVisionRMSNorm.forward6sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r/c^t|jjd|jS)Nz, eps=)tupler(shaper)r*s r. extra_reprz InternVLVisionRMSNorm.extra_repr=s* ))*+6$2G2G1HIIr/)gư>)__name__ __module__ __qualname__r#r<rA __classcell__r-s@r.r r ,s$;Jr/r modulequerykeyvalueattention_maskscalingdropoutc |}|} tj||jdd|z} |#|ddddddd|jdf} | | z} tj j | d} tj j| ||j} tj| | } | jddj} | | fS)Nrrr1dim)ptrainingr) r&matmul transposer?r$ functionalsoftmaxrMrS contiguous) rGrHrIrJrKrLrMkwargs key_states value_states attn_weights causal_mask attn_outputs r.eager_attention_forwardr_AsJL<<z';';Aq'ABWLL!$Q1.D 0@0@0D.D%DE #k1 ==((2(>L==((6??([L,,|\:K''1-88:K  $$r/c eZdZdZdeffd Z d dejdeejdeejde e fdZ xZ S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ |j}|j}d|_ tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j|_|dkDrtj*|ntj,|_ |rt/|jntj,|_|rt/|j|_ytj,|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r"r#rbr+ embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr$Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityr q_normk_norm)r*rb proj_dropoutqk_normr-s r.r#z InternVLVisionAttention.__init___s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!900 $$ii0NU[UjUjk ii0NU[UjUjk ii0NU[UjUjk " $..$.. I>JQ>N"**\":TVT_T_Ta?F+DNN;BKKM ?F+DNN; BKKM r/r9rKoutput_attentionsrYc |j\}}}|j|}|j|} |j|} |j |}|j | } |j |||j|jjdd}| j |||j|jjdd} | j|||j|jjdd} t} |jjdk7rt|jj} | ||| | |f|jsdn |j |j"dd|\} } | j |||j$} |j'| }|j)|}|r|| f}|S|df}|S)NrreagerF)rMrLro)sizerrrsrtrxryreshaperhrirUviewr_rb_attn_implementationrrSrlrkrfrurm)r*r9rKr|rY batch_sizeseq_len_ query_statesrZr[attention_interfacer^r\outputoutputss r.r<zInternVLVisionAttention.forward{s"/!3!3!5 GQ{{=1 [[/ {{=1 {{<0 [[, #++JQUQ^Q^_iijkmno '' GT^^T]][eefgijk #((Wdnndmm\ffghjkl (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$2H2HJJ %  % ! \"))*gt~~N &&{3((0,=6<(EKD>r/NN) rBrCrD__doc__rr#r&Tensorrrr r<rErFs@r.rara\sd5Z3Z>2648 )||)!.)$ELL1 ) -. )r/racLeZdZUeed<dZdZdZdgZdZ dZ dZ dZ fdZ xZS)InternVLVisionPreTrainedModelrbinternvl_vision pixel_valuesTInternVLVisionLayercVt||t|tr|jj j |j$|jj j |j%|jj j yyt|trs|jj j|jj|jj j|jjyy)zInitialize the weightsN)r" _init_weights isinstanceInternVLVisionEmbeddings cls_tokendatazero_ mask_tokenposition_embeddingsrlambda_1fill_rblayer_scale_init_valuelambda_2)r*rGr-s r.rz+InternVLVisionPreTrainedModel._init_weightss f% f6 7    ! ! ' ' )  ,!!&&,,.))5**//5576  3 4 OO & &t{{'I'I J OO & &t{{'I'I J5r/)rBrCrDr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrrErFs@r.rrsF  )$O&*#./N"& K Kr/rz7 Class for outputs of [`InternVLVisionModel`]. ) custom_introceZdZdZy)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)rBrCrDrr/r.rrsr/rcZeZdZdZfdZdej dej fdZxZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c^t||j|j}}|j|j }}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_||_tj|||||_ y)Nrr) kernel_sizestride) r"r# image_size patch_size num_channelsr+ num_patches patch_shaper$Conv2d projection) r*rbrrrr+rrr-s r.r#z&InternVLVisionPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L+:^hir/rreturnc|j\}}}}||jk7r td|j|}|jd|jd}}|j dj dd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r?rrjrflattenrU) r*rrrheightwidth embeddings patch_height patch_widths r.r<z%InternVLVisionPatchEmbeddings.forwards2>2D2D/ L&% 4,, ,w __\2 $.$4$4Q$79I9I!9Lk ''*44Q: L+666r/) rBrCrDrr#r&rr<rErFs@r.rrs) j 7ELL 7U\\ 7r/rceZdZdZdeddffd Zdejdededejfd Z d d ejd e ejdejfd Z xZ S)rzc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rbrNc2t|tjt j dd|j |_|jr:tjt j dd|j |_ nd|_ t||_ |j|_ t|jtj j"r |jn|j|jf|_|jj$}|j&r=tjt j d|dz|j |_nd|_tj*|j,|_y)Nr)r"r#r$r%r&zerosr+ruse_mask_tokenrrpatch_embeddingsrrr collectionsabcIterabler use_absolute_position_embeddingsrrvhidden_dropout_probrM)r*rbrr-s r.r#z!InternVLVisionEmbeddings.__init__s$ ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO =f E ++&++[__-E-EF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r/rrrc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j dz} ||j dz} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr1r?rrbicubicF)rmode align_cornersrP)r?rr&jit is_tracingrrrpermuter$rV interpolatercat) r*rrrr num_positionsclass_pos_embedpatch_pos_embedrQ new_height new_widthsqrt_num_positionss r.interpolate_pos_encodingz1InternVLVisionEmbeddings.interpolate_pos_encodingsj!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"tq11 T__Q// &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr/rbool_masked_posc|j\}}}}|j|\}\}}|j\} } }|K|jj | | d} |j dj | } |d| z z| | zz}|jj | dd} tj| |fd}|j||j|||z}|j|}|||ffS)Nr1rrP) r?rrrexpand unsqueezetype_asrr&rrrrM)r*rrrrrrrrrr mask_tokensw cls_tokenss r.r<z InternVLVisionEmbeddings.forward:s +001fe262G2G 2U/ /\;!+!2 GQ  &//00WbIK))"-55kBA#q1u- a?J^^**:r2> YY J7Q?  # # /#d&C&CJPVX]&^^J\\*- L+666r/N)rBrCrDrrr#r&rintrr BoolTensorr<rErFs@r.rrs >3>>,&D5<<&D&DUX&D]b]i]i&DV7;7ll7"%"2"237  7r/rcVeZdZfdZdej dej fdZxZS)InternVLVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r"r#rbr hidden_act activation_fnr$rpr+intermediate_sizefc1fc2r*rbr-s r.r#zInternVLVisionMLP.__init__Usd  #F$5$5699V//1I1IJ99V55v7I7IJr/r9rcl|j|}|j|}|j|}|Sr)rrr)r*r9s r.r<zInternVLVisionMLP.forward\s4/ **=9 / r/)rBrCrDr#r&rr<rErFs@r.rrTs$KU\\ellr/r) layer_normrms_normc eZdZdZdeddffd Z d dejdede e eje ejejfffdZ xZ S) rz?This corresponds to the Block class in the timm implementation.rbrNct||j|_d|_t ||_t ||_t|j|j|j|_ t|j|j|j|_ |j}tj |t#j$|jzd|_tj |t#j$|jzd|_tj*|j,|_y)Nrr,T) requires_grad)r"r#chunk_size_feed_forward seq_len_dimra attentionrmlpNORM2FN norm_typer+layer_norm_epslayernorm_beforelayernorm_afterrr$r%r&r'rrrvrrM)r*rb init_valuesr-s r.r#zInternVLVisionLayer.__init__is '-'E'E$08$V, '(8(8 9&:L:LRXRgRg h&v'7'789K9KQWQfQfg33  [5::f>P>P3Q%Qaef  [5::f>P>P3Q%Qaef zz&"<"<= r/r9r|c"|j|j||\}}|j|z}||z}|j|}|j |}|j |}|j |j |z}||z}||fS)N)r|)rrrrrrMr)r*r9r|attention_outputattention_weights layer_outputs r.r<zInternVLVisionLayer.forwardxs /3nn  ! !- 0//=/ ++  ==+;;)=8 ++M: xx - ||L1 == $==<7L$m3 ...r/)F) rBrCrDrrr#r&rboolrr>r<rErFs@r.rrfsiI >3 > >$#(/||/ / uU\\"E%,, *D$EE F /r/rc neZdZdeddffd Ze d dejdedede e e ffdZ xZ S) InternVLVisionEncoderrbrNct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r"r#rbr$ ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r*rbir-s r.r#zInternVLVisionEncoder.__init__sO  ]]vOgOgIh#iA$7$?#ij &+#$jsA#r9r|output_hidden_statesc|rdnd}|rdnd}t|jD]'\}}|r||fz}|||}|d}|s||dfz})|r||fz}t|||S)Nrrrlast_hidden_stater9 attentions) enumerater r) r*r9r|rall_hidden_statesall_self_attentionsr layer_module layer_outputss r.r<zInternVLVisionEncoder.forwards#7BD$5b4(4 POA|#$58H$H!(8IJM)!,M &9]1=Mrr<rErFs@r.rrsg,3,, #(%*  ||   #  uo% &   r/rceZdZdeddffd ZdZee d dejde ejde e d e e de eeff d ZxZS) InternVLVisionModelrbrNc2t||||_t||_t ||_|jrtjn*tj|j|j|_ |jy)Nr)r"r#rbrrrencoderuse_mean_poolingr$rw LayerNormr+r layernorm post_initrs r.r#zInternVLVisionModel.__init__so   26:,V4 $44BKKM",,vGYGY_e_t_t:u  r/c.|jjSr)rrr@s r.get_input_embeddingsz(InternVLVisionModel.get_input_embeddingss///r/rrr|rc.||n|jj}||n|jj}|j||\}}|j |||}|d}|j |}t ||j|jS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )r)r|rrr) rbr|rrrrrr9r) r*rrr|rembedding_outputrencoder_outputssequence_outputs r.r<zInternVLVisionModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj #oolOo\!,, /!5'  *!,..93-)77&11  r/)NNN)rBrCrDrr#r"rrr&rrrrrr>rr<rErFs@r.rrs 3  07;,0/3  ll "%"2"23 $D>  'tn  u:: ;   r/rc8eZdZUeed<dZdZdZdZdZ dZ dZ dZ y)InternVLPreTrainedModelrbTpast_key_valuesN) rBrCrDrrrr_skip_keys_device_placementrr_can_compile_fullgraphrrrr/r.r(r(s7 &*#"3N!"&r/r(c*eZdZdeffd ZdZxZS)InternVLMultiModalProjectorrbc*t|tj|jj t d|jz dzz|_tj|jj t d|jz dzz|jj |_ t|j|_tj|jj |jj |_y)Nrr)r"r#r$r vision_configr+rdownsample_ratiorrp text_configlinear_1rprojector_hidden_actactlinear_2rs r.r#z$InternVLMultiModalProjector.__init__s ,,v';';'G'G#aRXRiRiNiJjnoJo'op  , ,s1v7N7N3N/OST/T TV\VhVhVtVt  &556 &"4"4"@"@&BTBTB`B`a r/c|j|}|j|}|j|}|j|}|Sr)rr3r5r6)r*image_featuresr9s r.r<z#InternVLMultiModalProjector.forward s@7  m4 /  m4 r/)rBrCrDrr#r<rErFs@r.r.r.sb~br/r.zM Base class for InternVL outputs, with hidden states and attentions. c:eZdZUdZdZeejed<y)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) rBrCrDrr;rr& FloatTensorrrr/r.r:r:s 8<%"3"34;r/r:zx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. c#ReZdZddiZdeffd ZdZdZdZdZ dd e jd e e eeefd e efd Zd e j$de jde jfdZee d d e j$d e jde e j,de e j$de ede e jd e e eeefd e ede ede ede ede ede e j$deede eeffdZd!de j,defdZxZ S)" InternVLModelzlanguage_model.modellanguage_modelrbct||tj|j|_t ||_tj|j|_ |jyr) r"r#r from_configr0 vision_towerr.multi_modal_projectorr2r?r rs r.r#zInternVLModel.__init__1sY  %11&2F2FG%@%H"'33F4F4FG r/c6|jjSr)r?r"r@s r.r"z"InternVLModel.get_input_embeddings9s""7799r/c:|jj|yr)r?set_input_embeddingsr*rJs r.rFz"InternVLModel.set_input_embeddings<s 007r/c||_yrr?r*decoders r. set_decoderzInternVLModel.set_decoder?s %r/c|jSrrIr@s r. get_decoderzInternVLModel.get_decoderBs"""r/rvision_feature_layervision_feature_select_strategyc J||n|jj}||n|jj}|jj}|dk(r|j |j }n|j |j|}|dk(r|ddddddf}|jd}t|dz}|jd} |j| ||d}|j||}|j| d|jd}|j|}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. Nr1)rdefaultrrr) scale_factor) rbrOrPr1rBr vision_modelr9r?rr pixel_shufflerC) r*rrOrPrYr1vision_featureschannels feature_sizers r.get_image_featuresz InternVLModel.get_image_featuresEsD&%9$D $++JjJj  .9 +;; '  ;;77 2 %"//\/J\\O"//\/JXXYmnO )Y 6-aQh7O#((+8S=) $**1- *11*lLZ\],,_K[,\*11*b/BWBWXZB[\44_Er/ input_ids inputs_embedsr8cP|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jdj|j|j }|jd|jdz}||j|jk7rtd|d||S)z Obtains multimodal placeholdr mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )r3devicer1rrz6Image features and image tokens do not match: tokens: z , features )r"r&tensorrbimage_token_idlongr]allsumr expand_asr4r?numelrj)r*rZr[r8special_image_maskn_image_tokensn_image_featuress r.get_placeholder_maskz"InternVLModel.get_placeholder_maskzs  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1/99"=GG VYYZgZnZno)//2^5I5I!5LL + , 2 2 48L8L8N NHHXXcdtcuv "!r/rK position_idsr* use_cacher|r return_dictcache_positionrYrc | | n|jj} | | n|jj} | | n|jj} ||n|jj}||n|jj }|du|duz r t d||j|}|`|j|||}|j|j|j}|j|||}|j||}|jd||||| | | d| d |}t|j |j"|j$|j&|SdS)Nz:You must specify exactly one of input_ids or inputs_embedsrrOrP)r[r8T) rKrir*r[rjr|rrkrl)rr*r9rr;r)rbr|ruse_return_dictrOrPrjr"rYr4r]r3rhmasked_scatterr?r:rr*r9r)r*rZrrKrir*r[rOrPrjr|rrkrlrYr8rers r.r<zInternVLModel.forwards&2C1N-TXT_T_TqTq$8$D $++JjJj &1%2J   QU   r/rVrSc |j\}}}}||zdk7s||zdk7r td|j||t||zt||z }|j ddddj }|j|t||zt||zt||dzz }|j ddddj }|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rrjrrrrX)r*rVrSrrrrWs r.rUzInternVLModel.pixel_shuffles/>.B.B.D+ E68 L A %)=)Bjk k*.. s6L#893x,?V;W *11!Q1=HHJ*.. F\12C 8L4MsS[_kmn_nSoOp  *11!Q1=HHJr/r) NNNNNNNNNNNNN)r)!rBrCrD_checkpoint_conversion_mappingrr#r"rFrLrNr&r<rrrliststrrY LongTensorrhrrrr rrr r>r:r<floatrUrErFs@r.r>r>)s '=>N%O"~:8&# AE8< 3''3'uS$s)^'<=3)1 3j"))":?:K:K"]b]n]n"0'+*.1537+/59@D8<$(,0/3&*59D ##D ''D !. D u//0 D "% D   1 12D 'uS$s)^'<=D )1 D D>D $D>D 'tnD d^D !!1!12D -.D u11 2!D D L!U\\!!r/r>zT Base class for InternVL causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeejed<y) InternVLCausalLMOutputWithPasta] loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsr*r9rr;)rBrCrDrryrr&r<rrzr*rsr9r>rr;rr/r.rxrxs )-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8modelr$rpr2r+ vocab_sizer}r rs r.r#z)InternVLForConditionalGeneration.__init__+sS  "6* yy!3!3!?!?ASASA^A^ejk  r/c6|jjSr)rr"r@s r.r"z5InternVLForConditionalGeneration.get_input_embeddings1szz..00r/c:|jj|yr)rrFrGs r.rFz5InternVLForConditionalGeneration.set_input_embeddings4s ''.r/rc|jSr)r}r@s r.get_output_embeddingsz6InternVLForConditionalGeneration.get_output_embeddings7s ||r/c:|jj|yr)rrLrJs r.rLz,InternVLForConditionalGeneration.set_decoder:s w'r/c6|jjSr)rrNr@s r.rNz,InternVLForConditionalGeneration.get_decoder=szz%%''r/rrOrPc B|jjd|||d|S)Nrnr)rrY)r*rrOrPrYs r.rYz3InternVLForConditionalGeneration.get_image_features@s5-tzz,, %!5+I    r/c.|jjSr)rr?r@s r.r?z/InternVLForConditionalGeneration.language_modelOszz(((r/c.|jjSr)rrBr@s r.rBz-InternVLForConditionalGeneration.vision_towerSszz&&&r/c.|jjSr)rrCr@s r.rCz6InternVLForConditionalGeneration.multi_modal_projectorWszz///r/rZrKrir*r[labelsrjr|rrkrllogits_to_keep image_sizesrYc | | n|jj} | | n|jj} | | n|jj} ||n|jj}||n|jj }|j d||||||||| | | d||d|}|d}t|tr t| dn|}|j|dd|ddf}d}| 4|jd|| |jjjd|}t|||j|j |j"|j$S)ai Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", torch_dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```NT)rZrrKrir*r[rOrPrjr|rrkrlrr)rzrr)ryrzr*r9rr;r)rbr|rrorOrPrrrslicer} loss_functionr2rrxr*r9rr;)r*rZrrKrir*r[rOrPrrjr|rrkrlrrrYrr9 slice_indicesrzrys r.r<z(InternVLForConditionalGeneration.forward[sr2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD.#33!//)) ' ; ;   r/c Nt ||f|||||d|} |ddk(r|| d<| S)N)r*r[rKrlrrr)r"prepare_inputs_for_generation) r*rZr*r[rrKrlrrY model_inputsr-s r.rz>InternVLForConditionalGeneration.prepare_inputs_for_generationsVw<  +')))    !  !,8L (r/r)NNNNNNNNNNNNNNrN)NNNNNN)'rBrCrDrr_tied_weights_keysrr#r"rFr$ModulerrLrNr&r<rrrrsrtrYpropertyr?rBrCrrrurr rrrr>rxr<rrErFs@r.r|r|s"8-"?#, &" ++~ 1/ryy(( AE8<  ''  'uS$s)^'<=  )1  ))''00'+*.1537+/59@D8<-1$(,0/3&*5934.2#i ##i ''i !. i u//0 i "% i   1 12i 'uS$s)^'<=i )1 i ))*i D>i $D>i 'tni d^i !!1!12i c5<</0!i "ell+#i $+,%i & u44 5'i i \r/r|)rrr(r>r|)r)Bcollections.abcr dataclassesrtypingrrrr&torch.nnr$ activationsr cache_utilsr generationr integrationsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrrrautorconfiguration_internvlrrrr rrvr_rarrrrrrrrrrr(r.r:r>rxr|__all__rr/r.rs.!,, ! )7B9ddF&aaHY'JBIIJ(J6% II% <<% % << % U\\* %  %%6HbiiHVKOKK2  +E  !7BII!7L[7ryy[7|  3H I-/4-/`# BII# L2 72 2 j 'o ' '"))$  <"9 <  <  M+M  M`  <[<  <2 B'>B  BJ r/