rhdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddlmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*e&jVe,Z-ee$dGdde#Z.Gdde j^Z0Gdde j^Z1Gdde j^Z2de1iZ3Gdde j^Z4Gdd e j^Z5Gd!d"e j^Z6Gd#d$eZ7Gd%d&e j^Z8e$Gd'd(eZ9Gd)d*e j^Z:Gd+d,e j^Z; dId-e j^d.ejxd/ejxd0ejxd1eejxd2e=d3e=fd4Z>Gd5d6e j^Z?Gd7d8eZ@Gd9d:e j^ZAGd;dd?e9ZCGd@dAe j^ZDe$dBGdCdDe9ZEe$dEGdFdGe9eZFgdHZGy)JzPyTorch GIT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringcan_return_tuplelogging torch_int) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)GitVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r#rtorch FloatTensor__annotations__r$r%tupler&w/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/git/modeling_git.pyr"r"4sr 15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r0r"c eZdZdZfdZ d deejdeejdeejde dejf dZ xZ S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.cBt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_t|dd|_|j#dt%j&|jj)ddy) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr8register_bufferr+arangeexpandselfconfig __class__s r1r@zGitEmbeddings.__init__Js !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej  r0 input_idsr: inputs_embedspast_key_values_lengthreturncJ||j}n|jdd}|d}||jdd|||zf}||j|}n|}|jdk(r|j |}||z }|j |}|j |}|S)Nr<rr9)sizer:rEr8rGrHrL) rRrUr:rVrW input_shape seq_length embeddingsrGs r1forwardzGitEmbeddings.forwardYs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  --i8J&J  ' ': 5"&":":<"H  - -J^^J/ \\*- r0)NNNr) r'r(r)r*r@rr+ LongTensorr,intTensorr^ __classcell__rTs@r1r3r3GssE  "153759&' E,,-u//0  1 12  !$   r0r3ceZdZd fd Z d dej deejdeejdeedee dee de ej fd Z xZ S) GitSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd||_|-tjd|jjd|j|_t|j|jz |_ |j|jz|_ t|jj|jj z dzd z|_|j$|xj"|j$zc_t'j(|j|j|_t'j(|j|j|_t'j(|j|j|_t'j0|j2|_|xs t7|d d |_|j8d k(s|j8d k(rG|j:|_t'j<d|j:zd z |j|_yy)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr8r9 relative_keyrelative_key_query) r?r@rCnum_attention_headshasattr ValueError layer_idxlogger warning_oncerTr'r`attention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerJattention_probs_dropout_probrLrMr8rFrAdistance_embeddingrRrSr8rorTs r1r@zGitSelfAttention.__init__xs(     : : :a ?PVXhHi#F$6$6#78 445Q8 #      !8!8 9:,,  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rr0r%attention_mask head_maskpast_key_valueoutput_attentionspixel_values_presentrXc d|j\}}} |j|j|d|j|jj dd} |r |j nd} |j|j|d|j|jj dd} |j|j|d|j|jj dd} ||j| dddd| dddf| dddd| dddf|j\}}tj| ddddd| ddf|gd} tj| ddddd| ddf|gd} tj| | j dd}|jdk(s|jdk(r| jd| jd}}|Dtj|dz tj |j" jdd}n@tj$|tj |j" jdd}tj$|tj |j" jdd}||z }|j'||j(zdz }|j+| j, }|jdk(rtj.d | |}||z}nE|jdk(r6tj.d | |}tj.d | |}||z|z}|t1j2|jz }|||z}t4j6j9|d}|j;|}|||z}tj|| }|j=dddd j?}|jAdd|jBfz}|j|}||fS)Nr<rrirdimrjrkdtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr)"shaperzviewrlrr transposerwr{r|updateror+catmatmulr8tensorlongrrOr~rFtoreinsummathsqrtr functionalsoftmaxrLpermute contiguousrZrs)rRr%rrrrr batch_sizer\_ query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_pastattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r1r^zGitSelfAttention.forwards%2$7$7! J JJ} % T*b$":":D"OUVWI))[Aww1A%BDT$U[\]K!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L)!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC o--r0NNNNNFF) r'r(r)r@r+rarr,r boolr.r^rbrcs@r1rerews uJ7;15*.,1/4R.||R.!!2!23R.E--. R. ! R. $D> R.'tnR. u|| R.r0recneZdZfdZdej dej dej fdZxZS) GitSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr6) r?r@rryrCdenserHrIrJrKrLrQs r1r@zGitSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r0r% input_tensorrXcr|j|}|j|}|j||z}|SNrrLrHrRr%rs r1r^zGitSelfOutput.forward7 =1  ]3 }|'CD r0r'r(r)r@r+rar^rbrcs@r1rr1> U\\RWR^R^r0reagerceZdZd fd ZdZ d dej deejdeejdee dee dee d e ej fd Z xZ S) GitAttentionct|t|j||||_t ||_t|_y)N)r8ro) r?r@GIT_SELF_ATTENTION_CLASSES_attn_implementationrRroutputset pruned_headsrs r1r@zGitAttention.__init__sE .v/J/JK ,Cy  $F+ Er0c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrrRrlrrrrrzr{r|rrrsunion)rRheadsindexs r1 prune_headszGitAttention.prune_heads s u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r0r%rrrrrrXc`|j||||||\}}|j||} | |fSr)rRr) rRr%rrrrr attn_outputself_attn_weightsattention_outputs r1r^zGitAttention.forwardsI*.      * & & ;;{MB!222r0rr)r'r(r)r@rr+rarr,r rr.r^rbrcs@r1rrs";*7;15*.,1/43||3!!2!233E--. 3 ! 3 $D> 3'tn3 u|| 3r0rcVeZdZfdZdej dej fdZxZS)GitIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r?r@rryrCintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrQs r1r@zGitIntermediate.__init__6s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r0r%rXcJ|j|}|j|}|Sr)rrrRr%s r1r^zGitIntermediate.forward>s& =1 00? r0rrcs@r1rr5s#9U\\ellr0rcneZdZfdZdej dej dej fdZxZS) GitOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r?r@rryrrCrrHrIrJrKrLrQs r1r@zGitOutput.__init__Fs` YYv779K9KL f&8&8f>S>STzz&"<"<= r0r%rrXcr|j|}|j|}|j||z}|Srrrs r1r^zGitOutput.forwardLrr0rrcs@r1rrErr0rceZdZd fd Z d dej deejdeejdeedee dee de ej fd Z d Z xZ S) GitLayerct||j|_d|_t |||_t ||_t||_ y)Nr)ro) r?r@chunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rRrSrorTs r1r@zGitLayer.__init__TsK '-'E'E$%f B+F3' r0r%rrrrrrXc|j||||||\}}t|j|j|j|} | |fS)N)rrr)rrfeed_forward_chunkrr) rRr%rrrrrrself_attention_weights layer_outputs r1r^zGitLayer.forward\sg48>>   /)!5 4B4 001  # #T%A%A4CSCSUe 333r0cL|j|}|j||}|Sr)rr)rRrintermediate_outputrs r1rzGitLayer.feed_forward_chunkts,"//0@A{{#68HI r0rr)r'r(r)r@r+rarr,r rr.r^rrbrcs@r1rrSs(7;15*.,1/44||4!!2!234E--. 4 ! 4 $D> 4'tn4 u|| 40r0rceZdZfdZ d dej deejdeejdeee e e ejfdee dee dee d ee d ee d ee ej e ffd Z xZS) GitEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wNF) r?r@rSr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rRrSirTs r1r@zGitEncoder.__init__{sP  ]]vG_G_A`#aAHVQ$7#ab &+#$bsA$r%rrpast_key_values use_cacheroutput_hidden_statesr return_dictrXc |jr%|jr|rtjdd}t |t dt fs td|r | t}|rdnd} |rdnd} t|jD]4\} } |r| |fz} ||| nd}| ||||||}|d}|s,| |dfz} 6|r| |fz} | std||| | fDSt||| | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzBThe `past_key_values` should be either a `Cache` object or `None`.r/rrc3$K|]}|| ywrr/).0vs r1 z%GitEncoder.forward..s  = sr$rr%r&) rtrainingrprqrtyper rnr enumeraterr.r)rRr%rrrrrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r1r^zGitEncoder.forwardsJ  & &4==##p" /DJ+>?ab b 0*nO"6BD$5b4(4 POA|#$58H$H!.7.CilO(!$ M*!,M &9]1=M B $D>B 'tnB 'tnB d^B  uU\\"$;; <B r0rc&eZdZUeed<dZdZdZy)GitPreTrainedModelrSgitTct|trtjj |j d|j jtjj |jj|j jtjj |jj|j jt|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j 2|jj|j jyyt|tj"rJ|jjj|jjj%dyy)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsrinitnormal_class_embeddingrSinitializer_rangepatch_embeddingweightposition_embeddingrydatabiaszero_rAr5rHfill_)rRmodules r1 _init_weightsz GitPreTrainedModel._init_weightss f1 2 GGOOF22$++B_B_O ` GGOOF2299t{{?\?\O ] GGOOF55<<$++B_B_O ` fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r0N)r'r(r)rr-base_model_prefixsupports_gradient_checkpointingr r/r0r1r r s &*#*r0r ceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) rrSct|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_tj"|j |j|_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestriderrirr:r;r=)r?r@rSrC embed_dimrurvr Parameterr+randnrConv2d num_channelsr num_patches num_positionsrArrNrOrPrQs r1r@zGitVisionEmbeddings.__init__s   ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr0r]heightwidthrXc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr<g?rribicubicF)rZmode align_cornersr)rrr unsqueezer+jit is_tracingr:rvrreshaperrr interpolaterr) rRr]r0r1r.rr/class_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r1interpolate_pos_encodingz,GitVisionEmbeddings.interpolate_pos_encodingsv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr0 pixel_valuesc `|j\}}}}|sJ||jk7s||jk7r,td|d|d|jd|jd |jjj }|j|j |}|jdjdd}|jj|dd} tj| |gd } |r| |j| ||z} | S| |j|jz} | S) NzInput image size (*z) doesn't match model ().rrirr<r)rrurnrrrrflattenrrrPr+rr@rr:) rRrAr@rrr0r1 target_dtype patch_embeds class_embedsr]s r1r^zGitVisionEmbeddings.forward"s6'3'9'9$ Avu'Vt-F%SWSbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr0F) r'r(r)rr@r+rar`r@r,r^rbrcs@r1rrsdqq,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr0rcVeZdZfdZdej dej fdZxZS) GitVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r?r@rSr r activation_fnrryrCrfc1fc2rQs r1r@zGitVisionMLP.__init__6sd  #F$5$5699V//1I1IJ99V55v7I7IJr0r%rXcl|j|}|j|}|j|}|Sr)rNrMrOrs r1r^zGitVisionMLP.forward=s4/ **=9 / r0rrcs@r1rKrK5s$KU\\ellr0rKrrzr{r|rscalingrLc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)Nr<r)rr)prrri) r+rrrrrfloat32rrrLrr) rrzr{r|rrQrLkwargs attn_weightsrs r1eager_attention_forwardrWEs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r0ceZdZdZfdZ d dej deej deej deede ej eej ff dZ xZ S) GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rDgF)r?r@rSrCr)rl num_headshead_dimrnscaleattention_dropoutrL is_causalrryk_projv_projq_projout_projrQs r1r@zGitVisionAttention.__init___s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r0r%rcausal_attention_maskrrXc |j\}}}|j|}|j|} |j|} |j |||j |j jdd}| j |||j |j jdd} | j |||j |j jdd} |jjdk7r||||z}n| |}n |du|_ t} |jjdk7rN|jjdk(r|rtjdnt|jj} | ||| | ||j|j|j sdn |j" \} } | j%|||j'} |j)| } |sd} | | fS) z#Input shape: Batch x Time x Channelrriflash_attention_2Nrsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)r_rQrL)rrbr`rarr[r\rrSrr_rWrprqrr]rrLr9rrc)rRr%rrdrrr\r)querieskeysvaluesattention_interfacerrVs r1r^zGitVisionAttention.forwardss-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc ;; + +/B B).C.O!/2G!G&2!62$>DN(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0  LL((r0)NNF) r'r(r)r*r@r+rarrr.r^rbrcs@r1rYrY\s}GB.268<,1 5)||5)!.5) ( 5 5) $D> 5) u||Xell33 4 5)r0rYc eZdZdeffd Z d dej dej dej deede ejf dZ xZ S) GitVisionEncoderLayerrScDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yr) r?r@rCr)rY self_attnrrHrI layer_norm1rKmlp layer_norm2rQs r1r@zGitVisionEncoderLayer.__init__sm +++F3<<F & u  ! &r0rmceZdZdZdeffd Ze d deejdeejdee dee dee d e e e ff d ZxZS) GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rSct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) r?r@rSrrrrrmlayersr)rRrSrrTs r1r@zGitVisionEncoder.__init__sP  mmERXRjRjLk$lq%:6%B$lm &+#%msA#rrdrrrrXcj||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|} t |j D]*\} } |r|| fz}| | |||} | d} |s"|| dfz},|r|| fz}t | ||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr/)rrrr$r%r&)rSrruse_return_dictrryr) rRrVrrdrrrencoder_statesall_attentionsr%idx encoder_layerr s r1r^zGitVisionEncoder.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj &1%>N+>Vd  r0)NNNNN)r'r(r)r*rr@rrr+rarrr.rr^rbrcs@r1rwrws,, 268<,0/3&*D !.D  ( 5 D $D> D 'tn D d^D  uo% &D D r0rwceZdZdeffd Ze d deejdee dee dee dee de e e ff d Z xZS) GitVisionTransformerrSc t|||_|j}t ||_t j||j|_ t||_ t j||j|_ yr) r?r@rSrCrr]rrHrI pre_layrnormrwencoderpost_layernorm)rRrSr)rTs r1r@zGitVisionTransformer.__init__8sj  && -f5LL8M8MN'/  ll9&:O:OPr0rArrr@rrXc||n|jj}||n|jj}||n|jj}| t d|j ||}|j |}|j||||}|d}|j|}|s |f|ddzSt||j|jS)Nz You have to specify pixel_valuesr@)rVrrrrrr{) rSrrr|rnr]rrrrr%r&) rRrArrr@rr%encoder_outputsr$s r1r^zGitVisionTransformer.forwardBs2C1N-TXT_T_TqTq$8$D $++JjJj &1%& 'tn & #+4. & d^ &  uo% && & r0rzY The vision model from CLIP, used in GIT, without any head or projection on top. ceZdZUeed<dZdeffd ZdejfdZ e d de e jde ede eded e edeeeff d ZxZS) GitVisionModelrSrAcdt||t||_|j yr)r?r@r vision_model post_initrQs r1r@zGitVisionModel.__init__vs'  08 r0rXcB|jjjSr)rr]rrRs r1get_input_embeddingsz#GitVisionModel.get_input_embeddings|s  ++;;;r0rrr@rcb||n|jj}|j|||||S)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```)rArrr@r)rSr|r)rRrArrr@rs r1r^zGitVisionModel.forwardsA8&1%# 'tn # #' # d^ #  uo% &# # r0rc\eZdZdeffd Zdej dej fdZxZS) GitProjectionrSc0t|||_tjtj |j j|jtj|j|j j|_ yr) r?r@rSr SequentialryrtrCrHrIvisual_projectionrQs r1r@zGitProjection.__init__sf  !# IIf**668J8J K LL++1E1E1T1T U" r0r]rXc$|j|Sr)r)rRr]s r1r^zGitProjection.forwards%%j11r0) r'r(r)rr@r+rar^rbrcs@r1rrs* y 2%,,25<<2r0rzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states ceZdZfdZdZdZdZdedejdejdejfd Z dd Z e dd eejd eejd eejdeejdeejdeejdeeeeej&fdeedeedeededeedeeejeffdZxZS)GitModelclt||_t|_t j |_t|_ t|_ j6tjfdtjD|_|j#y)Nc3K|]B}tjtjddjj Dyw)rN)rr*r+zerosrtrC)rrrSs r1rz$GitModel.__init__..s;; U[[Av/C/C/O/OPQ;sAA )r?r@rSr3r]rrt image_encoderrrrrrxr ParameterListrimg_temperal_embeddingrrQs `r1r@zGitModel.__init__s   '/+F,@,@A!&) !.v!6  * * 6*,*:*:;v>>?;+D ' r0c.|jjSrr]rErs r1rzGitModel.get_input_embeddingss...r0c&||j_yrr)rRr|s r1set_input_embeddingszGitModel.set_input_embeddingss*/'r0c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rRheads_to_prunerrs r1 _prune_headszGitModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr0rZrrrXctjtj||||d}|j|dk(t d}|S)Nrrr)diagonal-inf)r+triuones masked_fillfloat)rRrZrrmasks r1_generate_future_maskzGitModel._generate_future_masksAzz%**T4eLWXY 5=9 r0c|jd}|jd}|j}|j} tj||f|| } tj |||zft d|j| } tj||f| |j} |dkDrAtj|jd|jd|zf| |j}tj| | fd} tj| |j| fd}tj| |fddddf}|4tj |jd|jdfd|}|jtjk7r td tj||j }t d||<|j|jd||z||z|zf}|j}|ddddd|f}|dddddf}||z|ddddd|f<|dddddddf}|S) NrrrrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr+rfullrrrrrn zeros_likerPclone)rRtgtmemorytgt_maskrWmemory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrs r1create_attention_maskzGitModel.create_attention_masksF))A,\\!_  ;; J7eTJJ #99 : &M::  kk j !?? "A %{{"HNN1$58N$NOH yy(K0a8 9hkk%&89qA#iiu 1=dAgF " *&+jj&,,q/6<>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer<z5You have to specify either input_ids or inputs_embedsrrrrz#pixel_values must be of rank 4 or 5)rUr:rVrWrir)rrrrW)tgt_len)rrrrrrrrr)%rSrrrr|rn%warn_if_padding_and_no_attention_maskrZrr get_seq_length get_head_maskrndimrr$rrrappendr+rrr]rrrrepeatrrr rrrrr%r&)rRrUrr:rArrVrrrrr@rr[r\rWprojected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr%rcombined_attention_maskexpanded_attn_maskrsequence_outputs r1r^zGitModel.forwards7J2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y. Q#..*K  &',,.s3KTU U ^ "#  &"/59 ..0$335 #&&y$++2O2OP $(!  #  A%"&"4"4 ;S#5### ""a'"$!&|'9'9!',>$Q 1a%78Sk-?-''**T-H-H-SS)#**+@A B#())O"C!!FGG(,(>(>(O %??%'#9 +  % ,(- !''*A/?/E/Ea/HI&,,'..) %%>$D$D  ! !! $(A(F(Fq(I I1a% !  #<>N"OUVW --j:J:P:PRbRiRij#'"<"< ,#9 #=#   %"< 0 6 6 B"b!(() &)%71?U>U>VXY8Y%Z"'1{1~o.?+a.AR(RSWiiS,, 2+/!5#!-T!9'  *!,#%(;; ;&-+;;)77&11   r0r) NNNNNNNNNNFN)r'r(r)r@rrrr`r+rrrarrrrrr listr,rr.rr^rbrcs@r1rrs &/0C#ekk5<<\a\h\h 0#d-115/3/3,004KO$(,0/3).&*c ELL)c !.c u||, c u||, c ELL) c   -c "%tE4E4E/F(F"GHc D>c $D>c 'tnc #'c d^c  uU\\"$>> ?c c r0rz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c eZdZdgZfdZdZdZe ddee jdee jdee jdee jd ee jd ee jd ee jd ee e e e jfd eedeedeededeede ee jeffdZ ddZxZS)GitForCausalLMz output.weightct||t||_t j |j |j|_|jyr) r?r@rrrryrCrBrrrQs r1r@zGitForCausalLM.__init__sF  F#ii 2 2F4E4EF  r0c|jSrrrs r1get_output_embeddingsz$GitForCausalLM.get_output_embeddingss {{r0c||_yrr)rRnew_embeddingss r1set_output_embeddingsz$GitForCausalLM.set_output_embeddingss $ r0rUrr:rArrVlabelsrrrrr@rrXc | | n|jj} |d} |j|||||||| | | | |  }|d}|j|}d}||jjj dj jj}|dd|dddfj}|ddddfj}|j|jd|jj|jdfd|jji|}| s|f|ddz}||f|zS|St|||j|j|j S) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rr:rArrVrrrrr@rrr<rrB)losslogitsrr%r&)rSr|rrrrrrRrwr loss_functionrrBrrr%r&)rRrUrr:rArrVrrrrrr@rrUrurrrnum_image_tokensshifted_logitsrs r1r^zGitForCausalLM.forwardsj&1%IIKNAqrE]--/F%4%%##B (>(>? B ;;11 DY,F)-)9TGf$ Ev E%#33!//))   r0c |B|j}|jd|kDr|}n|jddz }|dd|df}|j}||j|}|||jd||dS)NrrA)rUrrArr)rrnew_onesget) rRrUrrrrU past_lengthremove_prefix_lengthr[s r1prepare_inputs_for_generationz,GitForCausalLM.prepare_inputs_for_generations  &)88:Kq!K/'2$(1q'9A'=$!!%9%:":;I oo  !&// A $D>A 'tnA #'A d^A uU\\"$:: ;!A A HOS r0r)rrr r)r)Hr*r dataclassesrtypingrrrr+torch.utils.checkpointr activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersrmodeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrrconfiguration_gitrr get_loggerr'rpr"rr3rerrrrrrrr rrKrarrWrYrmrwrrrrr__all__r/r0r1rs  !,, !.)B9 Gll:   H %  ?; ?   ?-BII-`u.ryyu.r BII   .3299.3d bii   $)$NI I X***6P"))Pf 299 .% II% <<% % << % U\\* %  %%.L)L)`/6/fT ryyT n3 2993 l 2 '2  2 j 2BII 2 ~ !~  ~ B s 's  s l Qr0