rh)1dZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0e'jbe2Z3ee%dGdde#Z4Gdde jjZ6 dGde jjde jnde jnde jndee jnde8d e8fd!Z9Gd"d#e jjZ:Gd$d%e jjZ;Gd&d'eZGd,d-e=Z?Gd.d/e jjZ@Gd0d1e jjZAGd2d3e jjZBGd4d5e jjZCGd6d7e jjZDGd8d9eZEGd:d;e jjZFGd<d=e jjZGGd>d?e=ZHe%d@GdAdBe=ZIe%dCGdDdEe=eZJgdFZKy)HzPyTorch InstructBLIP model.N) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigzQ Class defining the outputs of [`InstructBlipForConditionalGeneration`]. ) custom_introceZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<deefd Zy) /InstructBlipForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsreturncHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)r+r,r-N)getattrto_tuple).0kselfs /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/instructblip/modeling_instructblip.py zKInstructBlipForConditionalGenerationModelOutput.to_tuple..JsB WW Gq!**, - s-0)tuplekeysr6s`r7r3z8InstructBlipForConditionalGenerationModelOutput.to_tupleIs% YY[    )__name__ __module__ __qualname____doc__r)rr9torch FloatTensor__annotations__r*r+r,r-rr3r<r7r(r(.s 04D(5**+ ,315FHU5,,- .526NHU../6:>OXeE$5$567>AEHU5+<+<%=>E %* r<r(ceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) InstructBlipVisionEmbeddingsconfigcpt|||_|j|_|j |_|j |_tjtjdd|j|_ tjd|j|j |j |_ |j |j zdz|_|jdz|_tjtjd|j|j|_y)Nr"r ) in_channels out_channels kernel_sizestrider)super__init__rG hidden_size embed_dim image_size patch_sizer ParameterrArandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr6rG __class__s r7rNz%InstructBlipVisionEmbeddings.__init__Ts  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"br< embeddingsheightwidthr.c|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r"Ng?rr rbicubicF)sizemode align_cornersdim)shaperZrAjit is_tracingrRrreshapepermuter functional interpolateviewcat) r6r]r^r_rXrYclass_pos_embedpatch_pos_embedrg new_height new_widthsqrt_num_positionss r7interpolate_pos_encodingz5InstructBlipVisionEmbeddings.interpolate_pos_encodingfs`!&&q)A- //55a81< yy##%+*F6UZ?** *11!RaR%811!QR%8r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr< pixel_valuesrvc|j\}}}}|jjj}|j|j |}|j dj dd}|jj|ddj |} tj| |gd} |r|j| ||} n |j} | | ddd| jdddfj |z} | S)Ndtyperr"rarf)rhrWweightrztoflatten transposerUexpandrArprvrZrc) r6rwrv batch_size_r^r_ target_dtype patch_embeds class_embedsr]rZs r7forwardz$InstructBlipVisionEmbeddings.forwards'3'9'9$ Avu++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY l;C #!%!>!>z6SX!Y !%!8!8 "4Q8L*//!:L8La5O"P"S"ST`"aa r<F) r=r>r?r%rNrATensorintrvrBboolr __classcell__r\s@r7rFrFSsmc7c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn r<rFmodulequerykeyvalueattention_maskscalingdropoutc ptj||jdd|z}|||z}tjj |d}tjj |||j}tj||} | jddj} | |fS)Nrarf)ptrainingr"r) rAmatmulr~rrmsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs r7eager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r<ceZdZdZfdZdej dedefdZ d dej de ej d e e d e ej e ej e e ej ffd Z xZ S) InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ d|_ |j|_ tj|jd|jzd|_|jretj t#j$|j}tj t#j$|j}nd}d}|Qt#j&|t#j(|d|f}tj ||j_tj|j|j|_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )bias) requires_grad)rMrNrGrOrPnum_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasrSrAzerosrp zeros_liker projection)r6rGq_biasv_biasrr\s r7rNzInstructBlipAttention.__init__su  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCr<tensorseq_lenbszc|j|||j|jjddj S)Nr"r)rorrr~r)r6rrrs r7_shapezInstructBlipAttention._shapes7{{3GQQRSUVWbbddr< hidden_states head_maskoutput_attentionsr.c |j\}}}|j|}|j||d|j||jzj ddddd}|d|d|d} } } t } |j jdk7rN|j jdk(r|rtjdnt|j j} | || | | fd |jsd n |j|jd |\} }| j||d j} |j| } |r| |f}|S| d f}|S) z#Input shape: Batch x Time x Channelr rrr"eagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.N)rrrra)rcrrkrrlrrG_attn_implementationlogger warning_oncerrrrrr)r6rrrrrtgt_lenrP mixed_qkv query_states key_states value_statesattention_interfacerroutputss r7rzInstructBlipAttention.forwardsr#0"4"4"6WiHH]+ %%c7At~~yTXTbTbGbckk q!Q 2;1y|YWX\,j (? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     %  #}}C$2H2HJJ %  % ! \"))#w;FFH ook2 1B; -JUVZH[r<NF)r=r>r?r@rNrArrrrrr9rrrs@r7rrsGD>eU\\eCece -1,1 ,||,ELL),$D> , u||Xell3XeELL>Q5RR S ,r<rcVeZdZfdZdej dej fdZxZS)InstructBlipMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rMrNrGr hidden_act activation_fnrrrOintermediate_sizefc1fc2r[s r7rNzInstructBlipMLP.__init__ sd  #F$5$5699V//1I1IJ99V55v7I7IJr<rr.cl|j|}|j|}|j|}|Sr)rrrr6rs r7rzInstructBlipMLP.forwards4/ **=9 / r<r=r>r?rNrArrrrs@r7rr s$KU\\ellr<rc eZdZdeffd Z ddej dej deede ejfdZ xZ S) InstructBlipEncoderLayerrGcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) rMrNrOrPr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r[s r7rNz!InstructBlipEncoderLayer.__init__sm ++.v6<<Fr?r#rNrArrrr9rBrrrs@r7rrsYS1S-2 $||$ $$D> $ u  ! $r<rcBeZdZUeed<dZdZdZdZdZ dZ dZ gdZ dZ y)InstructBlipPreTrainedModelrGblipT)InstructBlipQFormerEmbeddingsr%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputc|jj}t|tjtj frY|j jjd||j%|jjjyyt|tjr(|j jjd|yt|tjrJ|jjj|j jjdyt|trYtjj!|j"d|tjj!|j$d|yt|t&t(fr%|j*jjyy)zInitialize the weightsr)meanstdN?)rGinitializer_range isinstancerrrVr{datanormal_rzero_ Embeddingrfill_rFinit trunc_normal_rZrU$InstructBlipForConditionalGenerationInstructBlipModel query_tokens)r6rfactors r7 _init_weightsz)InstructBlipPreTrainedModel._init_weights^sI.. fryy"))4 5 MM   & &CV & <{{&   &&('  - MM   & &CV & <  - KK   " " $ MM   $ $S )  < = GG ! !&";";#6 ! R GG ! !&"8"8s ! O !EGX Y Z    $ $ * * ,[r<N)r=r>r?r#rCbase_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrrDr<r7rrKs> &*#"&N!-r<rc eZdZdZdeffd Z d deejdee dee dee de e e ff d Z xZS) InstructBlipEncodera Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rGct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rMrNrGr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r6rGrr\s r7rNzInstructBlipEncoder.__init__}sP  mmuU[UmUmOn$o!%=f%E$op &+#%psA#rroutput_hidden_states return_dictr.c||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|}t |j D])\} } |r||fz}| |||} | d}|s!|| dfz}+|r||fz}|st d|||fDSt|||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrD)rrrr"c3&K|] }|| ywrrDr4vs r7r8z.InstructBlipEncoder.forward..seqWXWdes)last_hidden_stater attentions)rGrr use_return_dict enumerater r9r) r6 inputs_embedsrrr r encoder_statesall_attentionsridx encoder_layer layer_outputss r7rzInstructBlipEncoder.forwards82C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r<)NNNN)r=r>r?r@r#rNrrArrrr9rrrrs@r7rrssy,1,26,0/3&* < !.< $D> < 'tn < d^ <  uo% &< r<rceZdZUdZeed<deffd Ze d dee jdee dee dee de de e eff d Zd ZxZS) InstructBlipVisionModelrwrGct||||_|j}t ||_t ||_tj||j|_ |jyr) rMrNrGrOrFr]rencoderrrrpost_layernorm post_init)r6rGrPr\s r7rNz InstructBlipVisionModel.__init__s]   && 6v>*62  ll9&:O:OP r<rr r rvr.c||n|jj}||n|jj}||n|jj}| t d|j ||}|j ||||}|d}|j|}|dddddf} |j| } |s || f|ddzSt|| |j|jS)Nz You have to specify pixel_values)rv)rrr r rr")r pooler_outputrr) rGrr rrr]rr rrr) r6rwrr r rvrencoder_outputsr pooled_outputs r7rzInstructBlipVisionModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%r?main_input_namer%rCrNrrrArBrrr9rrr'rrs@r7rrs$O $$ 7 59,0/3&*). ( u001( $D>( 'tn ( d^ ( #' (  u00 1( ( Tr<rcPeZdZdfd ZdZdZdZdZdZ d dZ xZ S) rct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj |j"|_t'|dd|_|j(dk(s|j(dk(rF|j*|_tj,d|j*zd z |j|_d |_y) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryrr"F)rMrNrGrOrhasattrrrattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrr2r-max_position_embeddingsrdistance_embeddingsave_attentionr6rGis_cross_attentionr\s r7rNz.InstructBlipQFormerMultiHeadAttention.__init__s     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD ##r<c||_yrattn_gradients)r6r>s r7save_attn_gradientsz9InstructBlipQFormerMultiHeadAttention.save_attn_gradientss ,r<c|jSrr=r;s r7get_attn_gradientsz8InstructBlipQFormerMultiHeadAttention.get_attn_gradients!s"""r<c||_yr attention_map)r6rDs r7save_attention_mapz8InstructBlipQFormerMultiHeadAttention.save_attention_map$s *r<c|jSrrCr;s r7get_attention_mapz7InstructBlipQFormerMultiHeadAttention.get_attention_map's!!!r<c|jdd|j|jfz}|j|}|j ddddS)Nrarrr"r )rcrr2rorl)r6x new_x_shapes r7transpose_for_scoresz:InstructBlipQFormerMultiHeadAttention.transpose_for_scores*sNffhsmt'?'?AYAY&ZZ AFFK yyAq!$$r<c|du}|rC|j|j|}|j|j|} |}n@|j|j|}|j|j|} |j|} |j| } t j | |j dd} |jdk(s|jdk(rF|jd} t j| tj|jjdd}t j| tj|jjdd}||z }|j||jzdz }|j| j }|jdk(rt j"d| |}| |z} nE|jdk(r6t j"d| |}t j"d ||}| |z|z} | t%j&|j(z } | j }|| |z} t+j,d | j|}|r8|j.r,|j1||j3|j4|j7|}|||z}t j || }|j9d d dd j;}|jdd|j<fz}|j|}|r||f}|S|f}|S)Nrarr/r0r"rzdeviceryzbhld,lrd->bhlrzbhrd,lrd->bhlrrfrrr )rKrrrrArr~r-rcarangelongrNror8r7r|rzeinsummathsqrtr2rSoftmaxr9rE register_hookr?rrlrr3)r6rrrencoder_hidden_statesencoder_attention_maskrr; key_layer value_layermixed_query_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtypeattention_probsattention_probs_dropped context_layernew_context_layer_shapers r7rz-InstructBlipQFormerMultiHeadAttention.forward/s83$> 11$((;P2QRI33DJJ?T4UVK3N11$((=2IJI33DJJ}4MNK JJ}5//0AB !<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||AABXY $"5"5  # #O 4  ) )$*B*B C#',,"?  &= &I # %S"S* **,CD 6G=/2O\M]r<rr() r=r>r?rNr?rArErGrKrrrs@r7rrs8$8-#+"%"#Kr<rcneZdZfdZdej dej dej fdZxZS)rc(t|tj|j|j|_tj |j|j|_tj|j|_ yr) rMrNrrrOdenserrr5hidden_dropout_probrr[s r7rNz&InstructBlipQFormerSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r<r input_tensorr.cr|j|}|j|}|j||z}|Srrlrrr6rrns r7rz%InstructBlipQFormerSelfOutput.forward7 =1  ]3 }|'CD r<rrs@r7rr~1> U\\RWR^R^r<rceZdZd fd ZdZ d dej deejdeejdeejdeejdee d e ej fd Z xZ S) InstructBlipQFormerAttentionct|t|||_t ||_t |_yr)rMrNr attentionroutputset pruned_headsr:s r7rNz%InstructBlipQFormerAttention.__init__s3 >vGYZ3F; Er<c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr"rf)lenrrwrr2rzrrrrrxrlr3union)r6headsindexs r7 prune_headsz(InstructBlipQFormerAttention.prune_headss u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r<rrrrVrWrr.cp|j||||||}|j|d|}|f|ddz} | S)N)rrrrVrWrrr")rwrx) r6rrrrVrWr self_outputsattention_outputrs r7rz$InstructBlipQFormerAttention.forwardsY~~')"7#9/ &  ;;|A F#% QR(88r<rr() r=r>r?rNrrArrrBrr9rrrs@r7rurus" ;*7;15=A>B,1||!!2!23E--.  ((9(9:  !)):): ; $D> u|| r<rucVeZdZfdZdej dej fdZxZS)InstructBlipQFormerIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rMrNrrrOrrlrrstrr intermediate_act_fnr[s r7rNz(InstructBlipQFormerIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r<rr.cJ|j|}|j|}|Sr)rlrrs r7rz'InstructBlipQFormerIntermediate.forwards& =1 00? r<rrs@r7rrs#9U\\ellr<rcneZdZfdZdej dej dej fdZxZS)InstructBlipQFormerOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rMrNrrrrOrlrrr5rmrr[s r7rNz"InstructBlipQFormerOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r<rrnr.cr|j|}|j|}|j||z}|Srrprqs r7rz!InstructBlipQFormerOutput.forwardrrr<rrs@r7rrrsr<rc>eZdZfdZ ddZdZdZxZS)InstructBlipQFormerLayercft||j|_d|_t ||_||_||jzdk(rt |d|_d|_ nd|_ t||_ t||_ t||_t||_y)Nr"rT)r;F)rMrNchunk_size_feed_forward seq_len_dimrurw layer_idxcross_attention_frequencycrossattentionhas_cross_attentionr intermediaterrxintermediate_query output_queryr6rGrr\s r7rNz!InstructBlipQFormerLayer.__init__s '-'E'E$5f=" v77 71 <">vZ^"_D '+D $',D $;FC/7 "A&"I5f=r<c |j||||}|d} |dd} |dkDr| ddd|ddf} |jr1| td|j| |||||} | d} | | ddz} t |j |j |j| } | jd|kDrt |j|j |j| dd|dddfj| j}tj| |gd} n,t |j|j |j| } | f| z} | S)N)rrrrr"z>encoder_hidden_states must be given for cross-attention layers)rrrVrWrrf)rwrrrrfeed_forward_chunk_queryrrrhfeed_forward_chunkr|rNrArp)r6rrrrVrWr query_lengthself_attention_outputsrrquery_attention_outputcross_attention_outputs layer_outputlayer_output_texts r7rz InstructBlipQFormerLayer.forwards"& )/ "0"  2!4(, ! %5a,6I%J "''(0$%eff*.*=*=*#1'*?+A&7 +>+'*A)C&!$;AB$??4--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,8I)JPQR 4'',,   L  /G+r<cL|j|}|j||}|Sr)rrxr6rintermediate_outputrs r7rz+InstructBlipQFormerLayer.feed_forward_chunk/s,"//0@A{{#68HI r<cL|j|}|j||}|Sr)rrrs r7rz1InstructBlipQFormerLayer.feed_forward_chunk_query4s."556FG(()<>NO r<)NNNNFr)r=r>r?rNrrrrrs@r7rrs,>."#=~ r<rc6eZdZfdZ ddZxZS)InstructBlipQFormerEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wr) rMrNrGrrrr rlayerr rs r7rNz#InstructBlipQFormerEncoder.__init__<sW  ]]JOPVPhPhJi jY %fi 8 j  ',# ksA$c |rdnd} |rdnd} |rdnd} t|jjD]^} |j| }|r| |fz} ||| nd}|||||||| }|d}|s:| |dfz} | dkDsI|jsV| |dfz} `|r| |fz} |st d|| | | fDSt || | | S)NrD)rWrrrr"rc3$K|]}|| ywrrDrs r7r8z5InstructBlipQFormerEncoder.forward..os  = s)rrrcross_attentions)rrGr rrr9r)r6rrrrVrWrr r rall_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_maskrs r7rz"InstructBlipQFormerEncoder.forwardDs0#7BD$5b4%6rDt{{445 VA::a=L#$58H$H!.7.CilO(%'="3)M*!,M &9]1=Mr?rNrrrs@r7rr;s',"#": r<rc2eZdZdZfdZ ddZxZS)rz;Construct the embeddings from word and position embeddings.cPt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|j!dt#j$|jj'ddt)|dd|_||_y) N) padding_idxr position_ids)r"raF) persistentr-r.)rMrNrr vocab_sizerO pad_token_idword_embeddingsr7position_embeddingsrr layernormr5rmrregister_bufferrArOrr2r-rGr[s r7rNz&InstructBlipQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ r<c||jd}nd}|&|jdd|||zfj}|k|j|}|jdk(r/|j |j |j}||z}|tj||fd}n|}|j |jjj}|j|}|j|}|S)Nr"rr.rf)rcrclonerr-rr|rNrArprr{rzr)r6 input_idsr query_embedspast_key_values_lengthr]r]rs r7rz%InstructBlipQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J++z9&*&>&>|zO`O`?a&b#'*== '"YY j'AqI %J]]4>>#8#8#>#>? ^^J/ \\*- r<)NNNr)r=r>r?r@rNrrrs@r7rrsE$ r<rceZdZdZdZdZdZdZdeffd Z dZ dZ dZ dde jd eed e j"d ed e jf d Z dde j(dee j,dee j(dee jdee j,dee j,dee j,deedeedeed eee j,effdZxZS)InstructBlipQFormerModelz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. FrGct||||_t||_t ||_|jyr)rMrNrGrr]rrr!r[s r7rNz!InstructBlipQFormerModel.__init__s9   7?1&9  r<c.|jjSrr]rr;s r7r'z-InstructBlipQFormerModel.get_input_embeddingss...r<c&||j_yrrr6rs r7set_input_embeddingsz-InstructBlipQFormerModel.set_input_embeddingss*/'r<c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrwr)r6heads_to_prunerr~s r7 _prune_headsz%InstructBlipQFormerModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr<r input_shaperN has_queryr.c|jdk(r|dddddddf}n=|jdk(r|ddddddf}ntd|d|jd|j|j}d|z d z}|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nrz!Wrong shape for input_ids (shape z) or attention_mask (shape )ryrg)rgrrhr|rz)r6rrrNrextended_attention_masks r7get_extended_attention_maskz4InstructBlipQFormerModel.get_extended_attention_masks.    1 $&4Qa]&C #    !Q &'5QdA5E&F #3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&r<rrrrrVrWrr r c D||n|jj}| | n|jj} | | n|jj} | | t d||j dnd} |j |||} | jdd} | \}}| j}|tj||f|}|j|| |}|t|tr|dj\}}}n|j\}}}||f}t|tr|Dcgc]}|j|}}n?|)tj||}|j|}n|j|}nd}|j||jj }|j#| |||||| | |  }|d}|dddddf}| s ||f|ddzSt%|||j&|j(|j*|j, Scc}w) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Cache` of length `config.n_layers` with each tuple having 4 tensors of: shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nz7You have to specify query_embeds when input_ids is Noner"r)rrrra)rN)rrrVrWrr r r)rr#past_key_valuesrrr)rGrr rrrhr]rcrNrAonesrrlistinvert_attention_mask get_head_maskr rrrrrr)r6rrrrrrVrWrr r rembedding_outputrrr]rNrencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapemaskencoder_extended_attention_maskr$sequence_outputr%s r7rz InstructBlipQFormerModel.forwards@2C1N-TXT_T_TqTq$8$D $++JjJj &1%S^`f"g ! ,/6AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2wX\43M3Md3S2w/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y$++2O2OP ,, 2"7#B/!5#%'  *!,'1a0 #]3oab6II I;-'+;;)77&11,==   C3xs-Hr) NNNNNNNNN)r=r>r?r@rrrrr$rNr'rrrArr9rrNrr LongTensorrrBrrrrrs@r7rrs #( N8/0C )' )'3Z)' )'  )'  )'\7;37/315=A>B,0/3&*n ##n !!2!23n u//0 n u||, n E--. n  ((9(9:n !)):): ;n $D>n 'tnn d^n  uU&&')UU Vn r<rz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. c#eZdZdZdgZdeffd ZdZdZdZ dZ d e jd e jfd Zee dde jd e jd ee jd ee jdee jdee jdee jd ee j$deedeedeededeedeedeeeffdZxZS)rrwrrGc(t||t|j|_t j tjd|j|jj|_ t|j|_t j|jj|j j|_t%j&|j |_|j(j*/|j*j-|j(j*|j(j./|j.j-|j(j.|j1yNr")rMrNr vision_config vision_modelrrSrArnum_query_tokensqformer_configrOrrqformerr text_configlanguage_projectionr from_configlanguage_modelrextend_keep_in_fp32_modulesr!r[s r7rNzInstructBlipModel.__init__|s  3F4H4HILLQ8O8OQWQfQfQrQr)st/0E0EF #%99V-B-B-N-NPVPbPbPnPn#o '33F4F4FG    0 0 <  " " ) )$*=*=*O*O P    4 4 @  & & - -d.A.A.W.W X r<c6|jjSrrr'r;s r7r'z&InstructBlipModel.get_input_embeddings""7799r<c:|jj|yrrrrs r7rz&InstructBlipModel.set_input_embeddings 007r<c|jjs_|jj|jj_|jj|jj _yyrrGuse_decoder_only_language_modelrsharedr embed_tokensdecoderr;s r7 _tie_weightszInstructBlipModel._tie_weightsT{{::7;7J7J7Q7QD   ' ' 47;7J7J7Q7QD   ' ' 4;r<c|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r"raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_mapr|rAcuda device_countrwarningr1rrio_same_devicer6rs r7_preprocess_acceleratez(InstructBlipModel._preprocess_acceleratey ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3:>D   ( ( 7 4r<rrc|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j|j }|SzY Obtains multimodal placeholdr mask from `input_ids` or `inputs_embeds`. rMra r'rArrGimage_token_idrPrNall unsqueeze expand_asr|r6rrspecial_image_masks r7get_placeholder_maskz&InstructBlipModel.get_placeholder_mask  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H /99"=GG VYYZgZnZno!!r<qformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskrr r rv use_cacherr.c r| | n|jj} |j|| | | | }|d}tj|j ddtj |j}|jj|jddd}tj|j ddtj |j}|tj|}tj||gd}|j|||||| | | }|dddd|j dddf}|7|jj|}|tj|}|j!|}|j#|j|j$}|j'|| }|j)||}|jj*r|jd ||| | | | d |}n|jd ||||| | | | d |}t-||| S)a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)rwrr r rvrrarMr"rf)rrrrVrWrr r rrrrr r r)rrrrrr r rr1rD)rGrrrArrcrPrNrrrh ones_likerprrr'rr|rzrmasked_scatterrr()r6rwrrrrrrrrr r rvrrr+ image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsrrs r7rzInstructBlipModel.forwardszb&1%->q-A+A1(DE  FD//DDFyQM%!&!;!% 8 8 F 5 8 89M9M}ObOb c!66yP]6^%445GI^_ ;; 6 6)d))+-"3%9'# G*d)) +-"3'="3%9'#  G?))#*  r<) NNNNNNNNNFN)r=r>r?r)rr#rNr'rrr rArrBrrrrrrrr rr9r(rrrs@r7rrss %O+,1&:8R ?( "e.>.> "uO`O` " >B15598<=A04,0/3&*).$(w ''w !,,w !))9)9 : w E--. w !!1!12 w $E$4$45w !))9)9 :w   -w $D>w 'tnw d^w #'w D>w -.w uEE F!w w r<ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c%eZdZUeed<dZdZdgZdeffd ZdZ dZ dZ d e jfd Zd Zd Zd ZdZ d!dej(dej*deej*deedeef dZdej*dej(fdZee d"dej(dej(deej*deej(deej*deej*deej*deej(deedeedeej*deededeedeed eee ff dZ!ejD d#dej(deej*deej*deej*deej*deej(ded ej*fd Z#xZ$S)$rrGrwTrc\t||tj|j|_t jtjd|j|jj|_ tj|j|_t j |jj|j"j|_|j&r t)j*|j"}nt-j*|j"}|j.%|j.j1|j.|j2%|j2j1|j2||_|j7yr)rMrNr _from_configrrrrSrArrrrOrrrrrrrr rr!rrrrr!)r6rGrr\s r7rNz-InstructBlipForConditionalGeneration.__init__Is3  3@@AUAUVLLQ8O8OQWQfQfQrQr)st/<P>PQN2>>v?Q?QRN  + + 7  " " ) ).*J*J K  / / ;  & & - -n.R.R S, r<c6|jjSrrr;s r7r'z9InstructBlipForConditionalGeneration.get_input_embeddingscrr<c:|jj|yrrrs r7rz9InstructBlipForConditionalGeneration.set_input_embeddingsfrr<c:|jj|yr)rset_output_embeddings)r6new_embeddingss r7r.z:InstructBlipForConditionalGeneration.set_output_embeddingsis 11.Ar<r.c6|jjSr)rget_output_embeddingsr;s r7r1z:InstructBlipForConditionalGeneration.get_output_embeddingsls""88::r<c6|jjSr)r get_encoderr;s r7r3z0InstructBlipForConditionalGeneration.get_encodero""..00r<c6|jjSr)r get_decoderr;s r7r6z0InstructBlipForConditionalGeneration.get_decoderrr4r<c|jjs_|jj|jj_|jj|jj _yyrrr;s r7rz1InstructBlipForConditionalGeneration._tie_weightsvrr<c|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyrrr s r7r z;InstructBlipForConditionalGeneration._preprocess_accelerate|r r<rrrvr c|j||d}|d}tj|jddtj|j }|j j|jddd} tj| jddtj|j } |tj|}tj| |gd}|j||| ||d } | dddd| jdddf} |j| } |r| || fS| S) a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)rwrvr rNrarMr"rf)rrrrVrWr ) rrArrcrPrNrrrhr rprr)r6rwrrrvr r+r"r#rr$r%r&r's r7get_image_featuresz7InstructBlipForConditionalGeneration.get_image_featuressY"**%%=+  &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!&,@BX+Y_`!a '1%".#7 % %Q'+A\->->q-A+A1(DE !% 8 8 F (.-G G$$r<rrc|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j|j }|Srrrs r7rz9InstructBlipForConditionalGeneration.get_placeholder_maskrr<rrrrr labelsrrc H| | n|jj} |j|||| d\}}}| s|jn|}| s|jn|}||j |}|t j |}|j|j|j}|j||}|j||}|jjre|jd ||| | | |d|}| r |jn|d}d}| w|jd || |jj j"d|}nB|jd ||||| | | | |d |}| r |j$n|d}| r |jn|d }t'||||| S) a+ qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import requests >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```NTrrrvr rrr)r*r<r) rrrrrr r r<rr")r)r*r+r,r-rD)rGrr:r3r'rAr r|rNrzrr!rrr* loss_functionrrr)r()r6rwrrrrrrrrr r<r rvrrr'r+r%rrr*r)s r7rz,InstructBlipForConditionalGeneration.forwardsp&1%8C ..0  7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ ;; 6 6)d))+-"3%9'# G(3W^^ FD!)t))!&T[[=T=T=_=_ci *d)) +-"3'="3%9'#  G$/7<))#*   r<c Bt|dr|j|jd} |j||||d\} } } |||jj g|jj z} | |jjjgz}tj|gtj|j}|j| d}|j|}|tj|}| j!|j|j"} |j%||}|j'|| }||d}|j(jj*s||d <|j(j,d i||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rrTr>rMr"r)rrrrD)r1r rhr:rGimage_token_indexrr bos_token_idrArrPrNrepeatr'r r|rzrr!ris_encoder_decodergenerate)r6rwrrrrrrvgenerate_kwargsrr'r+r% image_tokens start_tokensrinputsrs r7rEz-InstructBlipForConditionalGeneration.generategsD 4 )  ' ' )!''* ?C?V?V /#9%= @W@ <~}   $ = =>A]A]] +t{{/F/F/S/S.TT !LL,uzzR^ReRef %,,Z; 7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ #0NS""))<<"+F; .$%%..KK?Kr<)NFF) NNNNNNNNNNFN)NNNNNF)%r=r>r?r#rCr)rrrNr'rr.rModuler1r3r6rr rArBrrrr:rrrrrrr9r(rno_gradrErrs@r7rr9s $O!+,14:8B;ryy;11R ?0>B38&+ /%''/%!++/%!))9)9 : /% #+4. /% d^ /%b "e.>.> "uO`O` " >B15598<=A59,0/3-1&*).$(S ''S !,,S !))9)9 : S E--. S !!1!12 S $E$4$45S !))9)9 :S   1 12S $D>S 'tnS ))*S d^S #'S D>S +,!S " uEE F#S S jU]]_9==A045959).C''C$E$4$45C!))9)9 : C E,,- C !!1!12 C  1 12C#'C   CCr<r)rrrrr)r)Lr@rR dataclassesrtypingrrrrrAtorch.utils.checkpointr activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrrutilsrrrrrrautorr r!configuration_instructblipr#r$r% get_loggerr=rr(rJrFrfloatrrrrrrrrrrurrrrrrrr__all__rDr<r7r]sr" !11 !)B9 G&lljjIIoo   H %   k