rhddlZddlmZmZddlZddlZddlmZmZmZddl m Z ddl m Z ddl mZmZmZddlmZdd lmZdd lmZmZmZdd lmZmZmZmZdd lmZd dl m!Z!ejDe#Z$GddejJZ&GddejJZ'GddejJZ(GddejJZ)GddejJZ*GddejJZ+GddeZ,GddejJZ-Gdd ejJZ.Gd!d"ejJZ/Gd#d$ejJZ0Gd%d&ejJZ1Gd'd(eZ2Gd)d*e2Z3Gd+d,e2eZ4gd-Z5y).N)OptionalUnion)Tensordevicenn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging)BlipTextConfigc eZdZdZfdZ d deejdeejdeejde dejf dZ xZ S) BlipTextEmbeddingsz;Construct the embeddings from word and position embeddings.cPt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|jdt!j"|jj%ddt'|dd|_||_y) N) padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr#configselfr8 __class__s ~/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/blip/modeling_blip_text.pyr&zBlipTextEmbeddings.__init__3s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ input_idsr inputs_embedspast_key_values_lengthreturncH||j}n|jdd}|d}||jdd|||zf}||j|}|}|jdk(r|j |}||z }|j |}|j |}|S)Nr!rr$)sizer r+r#r-r.r2) r:r>r r?r@ input_shape seq_length embeddingsr-s r<forwardzBlipTextEmbeddings.forwardEs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL   00;M"  ' ': 5"&":":<"H  - -J^^J/ \\*- r=)NNNr) __name__ __module__ __qualname____doc__r&rr4 LongTensor FloatTensorintrrG __classcell__r;s@r<rr0ssE(153759&' E,,-u//0  1 12  !$   r=rc0eZdZdfd ZdZdZdZdZ ddejde ejde ejd e ejd e ejd e e d e e d e ejdeejfdZxZS)BlipTextSelfAttentionct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ ||_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj"|j$|_t)|dd|_|j*dk(s|j*dk(rG|j,|_tj.d|j,zd z |j|_yy) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)r#r$ relative_keyrelative_key_queryr)r%r&r8r)num_attention_headshasattr ValueErrorrNattention_head_size all_head_size layer_idxrLinearqueryencoder_hidden_sizekeyvaluer0attention_probs_dropout_probr2r7r#r,r'distance_embeddingr:r8is_cross_attentionr]r;s r<r&zBlipTextSelfAttention.__init__es     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rr=c||_yNattn_gradients)r:rjs r<save_attn_gradientsz)BlipTextSelfAttention.save_attn_gradientss ,r=c|jSrhrir:s r<get_attn_gradientsz(BlipTextSelfAttention.get_attn_gradientss"""r=c||_yrh attention_map)r:rqs r<save_attention_mapz(BlipTextSelfAttention.save_attention_maps *r=c|jSrhrprms r<get_attention_mapz'BlipTextSelfAttention.get_attention_maps!!!r= hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionscache_positionrAc |j\} } } |j|j| d|j|jj dd} |du} | r|n|}|St |trA|jj|j}| r |j}n|j}n|}| r|n|}| rK|IrGj|jj}|j|jj}n|j!|j| d|j|jj dd}|j#|j| d|j|jj dd}|D| s|nd}j%|||jd|i\}}| rd|j|j<t'j(| |j dd}|j*dk(s|j*dk(rF|j-d} t'j.| t&j0|j2 jdd}t'j.| t&j0|j2 jdd}||z }|j5||j6zdz }|j9| j: }|j*dk(rt'j<d | |}||z}nE|j*dk(r6t'j<d | |}t'j<d ||}||z|z}|t?j@|jz }|||j9|j2z}tCjDd |}|jG|}|||z}t'j(||}|jIddddjK}|j-dd|jLfz}|j|}||fS)Nr!rrWr|TrUrV)dtyperrzbhld,lrd->bhlrzbhrd,lrd->bhlrdimrr )'shaper_viewrXr[ transpose isinstancer is_updatedgetr]cross_attention_cacheself_attention_cachelayerskeysvaluesrarbupdater4matmulr#rCr5longrrdr,toreinsummathsqrtrSoftmaxr2permute contiguousr\)r:rurvrwrxryrzr{r| batch_sizerE_ query_layerrfrcurr_past_key_valuecurrent_states key_layer value_layerattention_scoresposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probsattention_probs_dropped context_layernew_context_layer_shapes r<rGzBlipTextSelfAttention.forwards%2$7$7! J JJ} % T*b$":":D3E/>  %.*=>+66::4>>J %*8*N*N'*8*M*M'&4#2D.- ."<+224>>BGGI-44T^^DKKK(j"d&>&>@X@XY1a   >*j"d&>&>@X@XY1a  )7It)<)C)C{DNN=M~<^*& ;&@DN--dnn=!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||#',,"?  &= &I # %S"S* **,CD o--r=rhNNNNNFN)rHrIrJr&rkrnrrrtr4rrrMr booltuplerGrOrPs@r<rRrRdsu8-#+" 7;15=A>B*.,115g.||g.!!2!23g.E--. g. ((9(9: g. !)):): ; g.!g.$D>g.!.g. u|| g.r=rRcneZdZfdZdej dej dej fdZxZS)BlipTextSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr) r%r&rr^r)denser.r/r0r1r2r9s r<r&zBlipTextSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r=ru input_tensorrAcr|j|}|j|}|j||z}|Srhrr2r.r:rurs r<rGzBlipTextSelfOutput.forward7 =1  ]3 }|'CD r=rHrIrJr&r4rrGrOrPs@r<rr1> U\\RWR^R^r=rceZdZd fd ZdZ d dej deejdeejdeejdee dee d eej d e ej fd Z xZ S)BlipTextAttentionct|t||||_t ||_t |_y)Nr])r%r&rRr:routputset pruned_headsres r<r&zBlipTextAttention.__init__s6 )&2DPYZ (0 Er=c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrr:rXr[rrr_rarbrrr\union)r:headsindexs r< prune_headszBlipTextAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r=rurvrwrxrzr{r|rAc r|j|||||||}|j|d|} | f|ddz} | S)Nrvrwrxrzr{r|rr)r:r) r:rurvrwrxrzr{r| self_outputsattention_outputoutputss r<rGzBlipTextAttention.forward s\yy )"7)/)!  ;;|A F#% QR(88r=)FN)NNNNFN)rHrIrJr&rr4rrrMr rrrGrOrPs@r<rrs" ;*7;15=A*.,115||!!2!23E--.  ((9(9:  ! $D>!. u|| r=rcVeZdZfdZdej dej fdZxZS)BlipTextIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yrh) r%r&rr^r)intermediate_sizerr hidden_actstrr intermediate_act_fnr9s r<r&zBlipTextIntermediate.__init__:s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r=rurAcJ|j|}|j|}|Srh)rrr:rus r<rGzBlipTextIntermediate.forwardBs& =1 00? r=rrPs@r<rr9s#9U\\ellr=rcneZdZfdZdej dej dej fdZxZS)BlipTextOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r%r&rr^rr)rr.r/r0r1r2r9s r<r&zBlipTextOutput.__init__Js` YYv779K9KL f&8&8f>S>STzz&"<"<= r=rurrAcr|j|}|j|}|j||z}|Srhrrs r<rGzBlipTextOutput.forwardPrr=rrPs@r<rrIrr=rceZdZfdZ d dej deejdeejdeejdeejdeedee d eej d e ej fd Z d Z xZ S) BlipTextLayercLt|||_|j|_d|_t |||_||_|jjr't ||jj||_ t||_ t||_ y)Nrr)rfr])r%r&r8chunk_size_feed_forward seq_len_dimr attention layer_num is_decodercrossattentionr intermediaterr)r:r8rr;s r<r&zBlipTextLayer.__init__Xs  '-'E'E$*6YG" ;; ! !"34;;+A+AY#D 18$V, r=rurvrwrxryrzr{r|rAc |j||||||} | d} | dd} |%|j| ||||||} | d} | | ddz} t|j|j|j | } | f| zS)N)rvrwr{rzr|rrr)rrrfeed_forward_chunkrr)r:rurvrwrxryrzr{r|self_attention_outputsrrcross_attention_outputs layer_outputs r<rGzBlipTextLayer.forwardfs"& )/)) "0" 2!4(, ,&*&9&9 5#&;-"3-':' # 7q9  7 ;;G0  # #T%A%A4CSCSUe ((r=cL|j|}|j||}|Srh)rr)r:rintermediate_outputrs r<rz BlipTextLayer.feed_forward_chunks,"//0@A{{#68HI r=r)rHrIrJr&r4rrrMr rrrGrrOrPs@r<rrWs -"7;15=A>B*.,115%)||%)!!2!23%)E--. %) ((9(9: %) !)):): ; %)!%)$D>%)!.%) u|| %)Nr=rcdeZdZfdZ ddej deejdeejdeejdeejdeeeejdee d ee d ee d ee d eej d e eej e ffdZ xZ S)BlipTextEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}w)NF) r%r&r8r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r:r8ir;s r<r&zBlipTextEncoder.__init__sP  ]]eFLdLdFe#fM&!$<#fg &+#$gsA$rurvrwrxrypast_key_values use_cacher{output_hidden_states return_dictr|rAc |jr%|jr|rtjdd}d} |rat |t s-tj dd} tj|}n$t |trt|t}| rdnd} |rdnd}|r|dnd}t|jjD]N}|j|}| r| |fz} |||nd}||||||||| }|d}|s:||dfz}|F||dfz}P| r| |fz} | r|j}| std ||| ||fDSt!||| || S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.TrrrWc3$K|]}|| ywrhr).0vs r< z*BlipTextEncoder.forward..s = s)last_hidden_staterru attentionscross_attentions)rtrainingloggerwarningrr warning_oncer from_legacy_cacher rr8rrto_legacy_cacherr)r:rurvrwrxryrrr{rrr|return_legacy_cacheall_hidden_statesall_self_attentionsall_cross_attentionsr layer_modulelayer_head_mask layer_outputss r<rGzBlipTextEncoder.forwards  & &4==p" # ou5##` '+#"5"G"G"XO\:"5o|~"V"6BD$5b4%6;P;\rbft{{445 VA::a=L#$58H$H!.7.CilO(%&! M*!,M &9]1=MBEI$(,1/4&*15X ||X !!2!23X E--. X ((9(9: X !)):): ; X "%e.?.?(@"ABX D>X $D>X 'tnX d^X !.X  uU\\"$MM NX r=rcVeZdZfdZdej dej fdZxZS)BlipTextPoolerct|tj|j|j|_tj |_yrh)r%r&rr^r)rTanh activationr9s r<r&zBlipTextPooler.__init__s9 YYv1163E3EF '')r=rurAc\|dddf}|j|}|j|}|S)Nr)rr)r:rufirst_token_tensor pooled_outputs r<rGzBlipTextPooler.forwards6+1a40 #56  6 r=rrPs@r<rrs#$ U\\ellr=rcVeZdZfdZdej dej fdZxZS)BlipTextPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r%r&rr^r)rrrrr transform_act_fnr.r/r9s r<r&z(BlipTextPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr=rurAcl|j|}|j|}|j|}|Srh)rrr.rs r<rGz'BlipTextPredictionHeadTransform.forwards4 =1 --m< }5 r=rrPs@r<rrs$UU\\ellr=rc*eZdZfdZdZdZxZS)BlipTextLMPredictionHeadcHt|t||_t j |j |jd|_t jtj|j|_ |j|j_ y)NF)bias) r%r&r transformrr^r)r(decoder Parameterr4zerosrr9s r<r&z!BlipTextLMPredictionHead.__init__sm 8@yy!3!3V5F5FUS LLV->->!?@ !II r=c:|j|j_yrh)rrrms r< _tie_weightsz%BlipTextLMPredictionHead._tie_weights's II r=cJ|j|}|j|}|Srh)rrrs r<rGz BlipTextLMPredictionHead.forward*s$}5  ]3 r=)rHrIrJr&r#rGrOrPs@r<rrs &&r=rcVeZdZfdZdej dej fdZxZS)BlipTextOnlyMLMHeadcBt|t||_yrh)r%r&r predictionsr9s r<r&zBlipTextOnlyMLMHead.__init__2s 3F;r=sequence_outputrAc(|j|}|Srh)r()r:r)prediction_scoress r<rGzBlipTextOnlyMLMHead.forward6s ,,_=  r=rrPs@r<r&r&1s#<!u||! !r=r&c*eZdZUdZeed<dZgZdZy)BlipTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r8bertc.t|tjtjfr<|jj j d|jjnct|tjrI|jj j|jj jdt|tjr2|j%|jj jyyy)zInitialize the weightsg)meanstd?N) rrr^r'weightdatanormal_r8initializer_ranger.rzero_fill_)r:modules r< _init_weightsz%BlipTextPreTrainedModel._init_weightsFs fryy",,7 8 MM   & &CT[[5R5R & S  - KK   " " $ MM   $ $S ) fbii (V[[-D KK   " " $.E (r=N) rHrIrJrKr__annotations__base_model_prefix_no_split_modulesr:rr=r<r-r-<s   %r=r-c# eZdZdZdfd ZdZdZdZdede e de d e d ef d Z dd eejdeejd eejdeejdeejdeejdeejdeejdeeej"dee dee dee dee d ee deejd ee ejeff dZxZS) BlipTextModela& The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and `is_decoder` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ct||||_t||_t ||_|r t|nd|_|jyrh) r%r&r8rrFrencoderrpooler post_init)r:r8add_pooling_layerr;s r<r&zBlipTextModel.__init__]sI   ,V4&v. 0AnV,t  r=c.|jjSrhrFr+rms r<get_input_embeddingsz"BlipTextModel.get_input_embeddingsgs...r=c&||j_yrhrF)r:rbs r<set_input_embeddingsz"BlipTextModel.set_input_embeddingsjs*/'r=c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrArrr)r:heads_to_prunerrs r< _prune_headszBlipTextModel._prune_headsnsE +002 CLE5 LL  u % / / ; ;E B Cr=rvrDrrrAc|jdk(r|dddddddf}n5|jdk(r|r|\}}tj||}|ddddfj||d|ddddfk} | j |j } | j d|j dkr[|j d| j dz } tjtj||| f|| j | gd} | dddddddf|ddddddfz}n*|ddddddf}ntd |d |j d |j |j }d |z dz}|S)a= Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r NrWrr)rrr!)axisz!Wrong shape for input_ids (shape z) or attention_mask (shape )rr2g) rr4r5repeatrrrcatonesrZ) r:rvrDrrextended_attention_maskrrEseq_ids causal_maskprefix_seq_lens r<get_extended_attention_maskz)BlipTextModel.get_extended_attention_maskvs&    1 $&4Qa]&C #    !Q &)4& J,,z&A%dD!m4;;J TUVZabfhikoboZpp )nn^-A-AB $$Q'.*>*>q*AA%3%9%9!%<{?P?PQR?S%SN"'))!JJ!+Z HQW_j_p_p(   #K+6aq!m*D~VWY]_cefVfGg*g'*8D$9I*J'3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&r=r>r rwr?encoder_embedsrxryrrr{rrr|c | | n|jj} | | n|jj} | | n|jj} |r| | n|jj} nd} | | t d|4|j |||j}|\}}|j}nY|%|jdd}|\}}|j}n2|%|jdd}|\}}|j}n t dd}| 5t| ts| ddjdn| j}|)tj|||zfj|}|j!||||}|t|t"r|dj\}}}n|j\}}}||f}t|t"r|Dcgc]}|j%|}}n?|)tj||}|j%|}n|j%|}nd}|j'||jj(}||j+|||| }n|}|j-|||||| | | | | | }|d}|j.|j/|nd}| s ||f|d dzSt1|||j2|j4|j6|j8 Scc}w) a. encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timer!zGYou have to specify either input_ids or inputs_embeds or encoder_embedsrr~rO)r>r r?r@) rvrwrxryrrr{rrr|r)r pooler_outputrrurr)r8r{ruse_return_dictrrZ%warn_if_padding_and_no_attention_maskrCrrr rget_seq_lengthr4rTrrYlistinvert_attention_mask get_head_maskrrFrArBrrrurr) r:r>rvr rwr?rZrxryrrr{rrrr|rDrrErr@rUencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapemaskencoder_extended_attention_maskembedding_outputencoder_outputsr)rs r<rGzBlipTextModel.forwardsvH2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K%0 "J %%F  &',,.s3K%0 "J "))F  '(--/4K%0 "J #**Ffg g!"  &"/59 "1%++B/$335 #  !"ZZZBX5X(YZ]]^deN150P0P K1  ! ,/6AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2wX\43M3Md3S2w/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y$++2O2OP  !##)+'= /   . ,, 2"7#B+/!5#)'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   [3xsK9)T)NNNNNNNNNNNNNFN)rHrIrJrKr&rGrIrMrrrNrrrYrr4r`rMrrrGrOrPs@r<r?r?Ts/0C<'$<'38:<'GM<'[_<' <'@-115/3,004158<9==A$(,0/3&*%*15!P ELL)P !.P u||, P ELL) P   - P !.P  ( 5P !) 6P "$u'8'8"9:P D>P $D>P 'tnP d^P TNP !.!P " uU\\"$PP Q#P r=r?c'eZdZfdZdZdZdZdZ ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee e jdee dee dee dee dee dee dee dee jdeee jeff$dZdfd ZxZS)BlipTextLMHeadModelct||t|d|_t ||_|j |_y)NF)rD)r%r&r?r.r&clslabel_smoothingr9s r<r&zBlipTextLMHeadModel.__init__Is8  !&EB &v.%55r=c6|jjSrh)r.rGrms r<rGz(BlipTextLMHeadModel.get_input_embeddingsPsyy--//r=c:|jj|yrh)r.rIr:new_embeddingss r<rIz(BlipTextLMHeadModel.set_input_embeddingsSs &&~6r=cB|jjjSrh)rmr(rrms r<get_output_embeddingsz)BlipTextLMHeadModel.get_output_embeddingsVsxx##+++r=c||jj_|j|jj_yrh)rmr(rrrqs r<set_output_embeddingsz)BlipTextLMHeadModel.set_output_embeddingsYs,'5$$2$7$7!r=r>rvr rwr?rxrylabelsrrr{rr return_logitsr reductionr|rAc4| | n|jj} |d} |j|||||||| | | | | ||}|d}|j|}|r|ddddddfj Sd}||ddddddfj }|ddddfj j |j }t||j}||jd|jj|jd}|dk(r0|j|jddjd}| s|f|d dz}||f|zS|St|||j|j|j |j" S) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NF) rvr rwr?rxryrrr{rrrr|rr!r)ryrnnonerW)losslogitsrrurr)r8r]r.rmrrrrrnrr(rCsumrrrurr)r:r>rvr rwr?rxryrwrrr{rrrxrryr|rr)r+lm_lossshifted_prediction_scoresloss_fctrs r<rGzBlipTextLMHeadModel.forward]sT&1%rrv model_kwargs model_inputsr;s r<rz1BlipTextLMHeadModel.prepare_inputs_for_generations>w<  +)   &* \"r=)NNNNNNNNNNNNNFTr0N)NN)rHrIrJr&rGrIrtrvrr4rr`rrrrrrGrrOrPs@r<rkrkHs607,8 -115/3,0048<9=)-8<$(,0/3&*(-%)#)15%Z ELL)Z !.Z u||, Z ELL) Z   - Z  ( 5Z !) 6Z &Z "$u||"45Z D>Z $D>Z 'tnZ d^Z  ~Z TN!Z "C=#Z $!.%Z & uU\\"$EE F'Z x  r=rk)r?rkr-)6rtypingrrr4torch.utils.checkpointrrrtorch.nnr activationsr cache_utilsr r r generationrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrrrutilsrconfiguration_blipr get_loggerrHrModulerrRrrrrrrrrrr&r-r?rk__all__rr=r<rsW" " $$%!CC)9  .   H %00hP.BIIP.h  . .d 299  RYY 9.9z_ bii_ F RYY  bii$ryy0!"))!%o%0p +p h|1?|~ Nr=