rhZdZddlZddlZddlmZddlmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!ddl"m#Z#e!jHe%Z&ee dGddeZ'Gdde jPZ)Gdde jPZ*Gdde jPZ+Gdde jPZ,Gdde jPZ-Gdde jPZ.Gd d!e jPZ/Gd"d#e jPZ0Gd$d%eZ1Gd&d'e jPZ2e Gd(d)eZ3e Gd*d+e3Z4Gd,d-e jPZ5e d.Gd/d0e3Z6Gd1d2e jPZ7Gd3d4e jPZ8e d5Gd6d7e3Z9e d8Gd9d:e3Z:e d;Gd<d=e3Z;e Gd>d?e3ZBM8Du'8'8!9:;B;?JeE$5$5678?r*rc4eZdZdZfdZddZ ddZxZS)ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. c,t|t||_t j t jdd|j|_ t||_ |jj}t j t jd|dz|j|_ t j|j|j|_t j"|j$|_||_yNr)super__init__TextEmbeddingstext_embeddingsr Parameterr$zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings Embeddingmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfrBr: __class__s r+r1zViltEmbeddings.__init__Os  .f5ekk!Q8J8J&KL 3F ;++77 #%<< A{QPVPbPb0c#d %'\\&2Q2QSYSeSe%f"zz&"<"<=  r*cL |jjjj\}}}}|j|}|dddddddfj }t j j||jd|jdfj}|dddfjddddf} |dddfjddddf} |j\} } } }|jj|jjz}|jddddddfjddjd| ||}t!j"t%| | Dcgc]R\}}t j j't j j|||fdd d||z d| |z fTc}}d}|j)djdd}|j)djdd}t!j*t-t!j.|jd t!j.|jd d d j1|j2}|ddddddddf}|j5|jd|jdd d d }|j)dd}|j)d}|dks|t7|t8s| | z}|j;}n| | z}t=|j;|}|j?d}d|z j?d}|dddfjA}|Dcgc]}||dddf|k(}}|Dcgc]}||dddf|k(}}|Dcgc]}|jCd}}|Dcgc]}|jCd}}|Dcgc]}||z  }}g} tEt%|||D]\}!\}}"}#|#dkrOt!jFt!jH|j |}$| jK||!|$^t!jFt!jH|"j |#d}%| jKt!j"||!||!|%gdt!j"| d} || dddf| dddffj| d | }|| dddf| dddffj| d }|| dddf| dddffj| d d}|| dddf| dddffj| d | }|jLj5| d d }&t!j"|&|fd}t!j"|jdddddfdddddfj5| d d |fd}||z}|jO|}t!j"t!jH|jddj1||gd}|||| |fffScc}}wcc}wcc}wcc}wcc}wcc}w)Nr)sizerrdimbilinearT)rGmode align_cornersij)indexingdeviceF)as_tuple) replacement)(r9 projectionweightshapefloatr functional interpolatelongsumrB image_size patch_sizer; transposeviewr$catzippadflattenstackrarangetorRexpand isinstanceintmaxminnonzerouniquerG enumerate multinomialonesappendr7rA)'rC pixel_values pixel_maskmax_image_length_phpwxx_maskx_hx_w batch_size num_channelsheightwidth patch_dim spatial_poshw pos_embed patch_indexeffective_resolution valid_idx non_valid_idx unique_rowsu valid_row_idxnon_valid_row_idxv valid_numsnon_valid_numspad_numsselectinvp valid_choice pad_choice cls_tokenss' r+ visual_embedzViltEmbeddings.visual_embed^sR,,77>>DD 1b"  ! !, /AtQM*002**6QWWQZ8P*QVVXQTl1%ad+QTl1%ad+23''/ L&%KK**dkk.D.DD ..q!"ax8BB1aHMMaQ]_hjst II SM Aq !!MM--#V'&* .  1fqj1  %%a(221a8 IIaL " "1a (kk U\\&,,r"23U\\&,,rBR5S^b cik "FMM" " "$aA"56 !((a&,,q/2rSUV !))!Q/ " a #3#;:N^`cCd $'9 3779 #&9 "#7#;#;#=?OP NNEN2 V,,e,< 1o,,. BMNQ9QT?a#78N NNYZ]=A+>!+CDZZ)67AaffQi7 7->?!&&)??2<=Q$q(==&s:~x'PQ fMAz2qAv$00A1D1D1FHXY  mA.|<="..uzz"~/C/C/EqVZ[  eiiq)9;LQ;OPZ;[(\bcde  f6q) fQTlF1a4L( ) . .z2| Lq!t fQTl2388RH!&A,q!t "<=BB:rSTU fQTlF1a4L89>>z2|\ ^^**:r2> IIz1o1 -II  % %aAg .q$z : A A*bRT UW` agh   M LLOEJJv||A:==fEvNTUV&;888S POZ7?=s%?AZ Z +ZZ%Z Z!c (|j|||} |-|j|||jj\}} } n|j d} |d}| |j t j|t j| jz} ||j t j| |t j| jz}t j| |gd} t j|| gd} | | fS)N) input_idstoken_type_ids inputs_embeds)rurdtyperRrH) r3rrBrurdr>r$ zeros_liker[rR full_likera)rCrattention_maskrrsrtr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmaskss r+forwardzViltEmbeddings.forwards**m+  595F5Fj4;;;W;W6G6 2L+{%,,Q/K '#$ !D$>$>   ^5::kFXFX Y%  $d&@&@ OOK)=UZZXcXjXj k'  YY \:B  >;7Q?5  r*))r)r r!r"r#r1rr __classcell__rDs@r+r-r-Fs V9B'!r*r-c*eZdZdZfdZddZxZS)r2zGConstruct the embeddings from word, position and token_type embeddings.c>t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_t#|dd|_|j'dt)j*|jj-dd|j'd t)j.|j0j3t(j4 dy) N) padding_idxepsposition_embedding_typeabsolute position_ids)rrNF) persistentrr)r0r1rr< vocab_sizer6 pad_token_idword_embeddingsmax_position_embeddingsr;type_vocab_sizer> LayerNormlayer_norm_epsr?r@rAgetattrrregister_bufferr$rfrhr5rrGr[rCrBrDs r+r1zTextEmbeddings.__init__s/ !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r*cT||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} || z} |jdk(r|j|} | | z } |j| } |j| } | S)NrNrrrrr)rGrhasattrrrhr$r5r[rRrr>rr;rrA) rCrrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr>rr;s r+rzTextEmbeddings.forwards=  #..*K',,.s3K ^  ,,Q  ^ J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- r*)NNNNr r!r"r#r1rrrs@r+r2r2sQ & r*r2c(eZdZdZfdZdZxZS)r8z# Image to Patch Embedding. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)r0r1r]r^r~r6ri collectionsabcIterabler:rConv2drU)rCrBr]r^r~r6r:rDs r+r1zViltPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir*c|j\}}}}||jk7r td|jjj }|j|j |}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rWr~ ValueErrorrUrVrrg)rCrsr}r~rr target_dtyperys r+rzViltPatchEmbeddings.forward-si2>2D2D/ L&% 4,, ,w --33 OOLOO,O? @r*rrs@r+r8r8s jr*r8c&eZdZfdZddZxZS)ViltSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r0r1r6num_attention_headsrrrjattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer?attention_probs_dropout_probrArs r+r1zViltSelfAttention.__init__9s.     : : :a ?PVXhHi"6#5#5"67334A7  $*#=#= #&v'9'9F,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD 6G=/2O\M]r*NNFr r!r"r1rrrs@r+rr8sG$,r*rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rBreturnNct|tj|j|j|_tj |j|_yN) r0r1rrr6denser?r@rArs r+r1zViltSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r*r input_tensorcJ|j|}|j|}|SrrrArCrrs r+rzViltSelfOutput.forwards$ =1  ]3 r*) r r!r"r#rr1r$Tensorrrrs@r+rr{sD >z>d> U\\RWR^R^r*rc,eZdZfdZdZddZxZS) ViltAttentionct|t||_t ||_t |_yr)r0r1r attentionroutputset pruned_headsrs r+r1zViltAttention.__init__s0 *62$V, Er*c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)NrrrH)lenrrrrrrrrrrrrunion)rCheadsindexs r+ prune_headszViltAttention.prune_headss u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r*cj|j||||}|j|d|}|f|ddz}|S)Nrr)rr)rCrrrr self_outputsattention_outputrs r+rzViltAttention.forwardsE~~m^YPab ;;|A F#% QR(88r*r)r r!r"r1r rrrs@r+rrs" ;$r*rc`eZdZdeddffd Zdej dej fdZxZS)ViltIntermediaterBrNct|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r0r1rrr6intermediate_sizerri hidden_actstrr intermediate_act_fnrs r+r1zViltIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r*rcJ|j|}|j|}|Sr)rrrCrs r+rzViltIntermediate.forwards& =1 00? r* r r!r"rr1r$rrrrs@r+rrs19z9d9U\\ellr*rcxeZdZdeddffd Zdej dej dej fdZxZS) ViltOutputrBrNct|tj|j|j |_tj|j|_ yr) r0r1rrrr6rr?r@rArs r+r1zViltOutput.__init__sB YYv779K9KL zz&"<"<= r*rrcT|j|}|j|}||z}|Srrrs r+rzViltOutput.forwards. =1  ]3 % 4 r*rrs@r+rrs?>z>d> U\\RWR^R^r*rc*eZdZdZfdZddZxZS) ViltLayerz?This corresponds to the Block class in the timm implementation.crt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nrr)r0r1chunk_size_feed_forward seq_len_dimrrr intermediaterrrrr6rlayernorm_beforelayernorm_afterrs r+r1zViltLayer.__init__s '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr*c|j|j||||}|d}|dd}||j|jz}|j |}|j |}|j ||}|f|z}|S)N)rrr)rr%rgrRr&r$r) rCrrrrself_attention_outputsrr layer_outputs r+rzViltLayer.forwards!%  ! !- 0  / "0"  2!4(,)=+;+;z&ViltEncoder.forward..smq_`_lms)last_hidden_staterr)ror0r(r ) rCrrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r+rzViltEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO(YjkM)!,M &9]1=M.  - KK   " " $ MM   $ $S ).r*N) r r!r"rr&base_model_prefixsupports_gradient_checkpointing_no_split_modulesrJr)r*r+r?r?"s% &*#)+>?*r*r?ceZdZdfd ZdZdZdZe ddee jdee jdee jdee jd ee jd ee jd ee jd ee jd ee dee dee dee deeee jffdZxZS) ViltModelct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r0r1rBr-rr+encoderrrr6r layernorm ViltPoolerpooler post_init)rCrBadd_pooling_layerrDs r+r1zViltModel.__init__<sk   (0"6* f&8&8f>S>ST,=j(4  r*cB|jjjSrrr3rrCs r+get_input_embeddingszViltModel.get_input_embeddingsMs..>>>r*c:||jj_yrrX)rCrs r+set_input_embeddingszViltModel.set_input_embeddingsPs:?''7r*c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrQr0rr )rCheads_to_pruner0r s r+ _prune_headszViltModel._prune_headsSsE +002 CLE5 LL  u % / / ; ;E B Cr*rrrrsrtrrrrrr7r8rc ~| | n|jj} | | n|jj} | | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \}}| |jn |j}|tj||f|}| | t d| | t d||jdn|jd}||k7r t d |Btj||jj|jjf|}|j||jj}|j|||||||| \}}|j|| }|j!|||| | | }|d}|j#|}|j$|j%|nd}| s ||f|d dzSt'|||j(|j* S)ak image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerNz5You have to specify either input_ids or inputs_embedsrQzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrr7r8r)r6 pooler_outputrr)rBrr7use_return_dictr%warn_if_padding_and_no_attention_maskrGrRr$rqrWr] get_head_maskr/rget_extended_attention_maskrQrRrTr rr)rCrrrrsrtrrrrrr7r8rtext_batch_sizerrRimage_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputs r+rzViltModel.forward[sT2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U&1#%.%:!! @T@T  !"ZZ/:)FPVWN  # (@ef f  !l&:VW W4@4L<--a0R^RdRdefRg  .`a a  %5t{{7M7Mt{{OeOe$fouvJ&&y$++2O2OP +/??       !5,; , (.150P0PQ_al0m,, 2/!5# ' *!,..98< 8O O4UY #]3oab6II I)-')77&11   r*)T NNNNNNNNNNNN)r r!r"r1rZr\r`rrr$ LongTensorr%rjboolrr r(rrrs@r+rOrO:s`"?@C156:594815155948.2,0/3&*t E,,-t !!2!23t !!1!12 t u001 t U--. t E--.t   1 12t u001t 'smt $D>t 'tnt d^t  )51B1B+CC Dt t r*rOc$eZdZfdZdZxZS)rSct|tj|j|j|_tj |_yr)r0r1rrr6rTanh activationrs r+r1zViltPooler.__init__s9 YYv1163E3EF '')r*c\|dddf}|j|}|j|}|S)Nr)rrt)rCrfirst_token_tensorrms r+rzViltPooler.forwards6+1a40 #56  6 r*rrs@r+rSrSs $ r*rSzU ViLT Model with a language modeling head on top as done during pretraining. ceZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee jdee dee dee de eee jffdZxZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biasct||t||_t ||_|j yr)r0r1rOr@ ViltMLMHead mlm_scorerUrs r+r1zViltForMaskedLM.__init__s4  f% $V, r*c.|jjSr)r{decoderrYs r+get_output_embeddingsz%ViltForMaskedLM.get_output_embeddingss~~%%%r*c\||j_|j|j_yr)r{r}r)rCnew_embeddingss r+set_output_embeddingsz%ViltForMaskedLM.set_output_embeddingss !/,11r*rrrrsrtrrrlabelsrr7r8rc F| | n|jj} |j||||||||| | |  } | dd\}}||jdn|jd}|ddd|f|dd|df}}|j |}d}| at }| j |j} ||jd|jj| jd}| s|f| ddz}||f|zS|St||| j| jS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrsrtrrrrr7r8rFrrNrrrr) rBrcr@rWr{rrgrRr`rr rr)rCrrrrsrtrrrrrr7r8rrlrm text_seq_len text_featuresrv mlm_logitsmasked_lm_lossloss_fctrs r+rzViltForMaskedLM.forwards]R&1%o 'tno d^o  ~uU%6%677 8o o r*rxc$eZdZfdZdZxZS)ViltPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ y)Nr) r0r1rrr6rrirrr transform_act_fnrrrs r+r1z$ViltPredictionHeadTransform.__init__ns{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr*cl|j|}|j|}|j|}|Sr)rrrrs r+rz#ViltPredictionHeadTransform.forwardws4 =1 --m< }5 r*rrs@r+rrmsUr*rc,eZdZdfd ZdZdZxZS)rzc|t|||_t||_t j |j|jd|_ t jtj|j|_ |||j_|j|j_ y)NFr)r0r1rBr transformrrr6rr}r4r$r5rrV)rCrBrVrDs r+r1zViltMLMHead.__init__s  4V<yy!3!3V5F5FUS LLV->->!?@  "(DLL !II r*c:|j|j_yr)rr}rYs r+ _tie_weightszViltMLMHead._tie_weightss II r*cJ|j|}|j|}|Sr)rr})rCrys r+rzViltMLMHead.forwards" NN1  LLOr*r)r r!r"r1rrrrs@r+rzrz~s &&r*rzz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. ceZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ejffdZ xZS)ViltForQuestionAnsweringc t|||j|_t||_t j t j|j|jdzt j|jdzt jt j|jdz|j|_ |jy)NrF) r0r1 num_labelsrOr@r Sequentialrr6rGELU classifierrUrs r+r1z!ViltForQuestionAnswering.__init__s   ++f% -- IIf((&*<*>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrrrFr)rBrcr@rbrrgrRrrY binary_cross_entropy_with_logitsrWrrr)rCrrrrsrtrrrrrr7r8rrbrrrs r+rz ViltForQuestionAnswering.forwards b&1%T 'tnT d^T  'u/@/@)AA BT T r*rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. ceZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ejffdZ xZS)ViltForImageAndTextRetrievalct||t||_t j |j d|_|jyr/) r0r1rOr@rrr6 rank_outputrUrs r+r1z%ViltForImageAndTextRetrieval.__init__ sC  f% 99V%7%7; r*rrrrsrtrrrrrr7r8rc D| | n|jj} d} | td|j||||||||| | |  }| r |jn|d}|j |}| s|f|ddz}| | f|zS|St | ||j|jS)ad image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rrrFr) rBrcNotImplementedErrorr@rbrrrr)rCrrrrsrtrrrrrr7r8rrrbrrs r+rz$ViltForImageAndTextRetrieval.forwardsZ&1%M 'tnM d^M  'u/@/@)AA BM M r*rzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. ceZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ejffdZ xZS)"ViltForImagesAndTextClassificationc t|||j|_t||_|j }t jt j|j|z|j|zt j|j|zt jt j|j|z|j|_ |jyr)r0r1rrOr@ num_imagesrrrr6rrrrU)rCrBrrDs r+r1z+ViltForImagesAndTextClassification.__init__ms   ++f% && -- IIf((:5v7I7IJ7V W LL++j8 9 GGI IIf((:5v7H7H I   r*rrrrsrtrrrrrr7r8rc | | n|jj} | | n|jj} | | n|jj} | |jdk(r|j d}| |jdk(r|j d}||j dnd} | ||j dnd} | |jjk7r tdg}| rgnd}| rgnd}t| D]}|j|||||dd|ddddddfnd||dd|ddddfnd||||dd|ddddfnd|dz| | |  }| r |jn|d}|j|| r|j|j| s|j|jtj |d}|j#|}d}| Wt%}| j'|j(} ||j+d|j,| j+d}| s|||f}||f|zS|St/|||| S) a3 image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrrz\Make sure to match the number of images in the model with the number of images in the input.) rrrsrtrrrrrr7r8rNrHr)rBrr7rcndim unsqueezerWrrr.r@rbrrrrr$rarrrgrRr`rr)rCrrrrsrtrrrrrr7r8rpooler_outputsrrrrrbrmrrrrs r+rz*ViltForImagesAndTextClassification.forwards^2C1N-TXT_T_TqTq$8$D $++JjJj &1%'11!4L  # (9(9Q(>'11!4L.:.F\''*D  2>2J++A.PTJ // /n 2 ,R$ z" 6Aii--p 'tnp d^p  7u?P?P9QQ Rp p r*rceZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ejffdZ xZS)ViltForTokenClassificationc0t|||j|_t|d|_t j |j|_t j|j|j|_ |jy)NF)rV) r0r1rrOr@rr?r@rArr6rrUrs r+r1z#ViltForTokenClassification.__init__sk   ++f> zz&"<"<= ))F$6$68I8IJ r*rrrrsrtrrrrrr7r8rc 4| | n|jj} |j||||||||| | |  } | d}||jdn|jd}|j |}|j |ddd|f}d}| Wt }| j|j} ||jd|j| jd}| s|f| ddz}||f|zS|St||| j| jS)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrrrrNrFr)rBrcr@rWrArrrgrRr`rrrr)rCrrrrsrtrrrrrr7r8rrltext_input_sizerrrrs r+rz"ViltForTokenClassification.forwards?0&1% E,,-> !!2!23> !!1!12 > u001 > U--. > E--.>   1 12> u001> ))*> $D>> 'tn> d^>  $eE,=,=&>> ?> > r*r)rrrrxrr rOr?)>r#collections.abcrr dataclassesrtypingrrr$torch.utils.checkpointrtorch.nnr activationsr modeling_layersr modeling_outputsr r r rrrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_viltr get_loggerr loggerrModuler-r2r8rrrrrr r+r?rOrSrxrrzrrrr__all__r)r*r+rsf !" %!9.[[,*   H %  @{@  @$W!RYYW!t6RYY6r"))>? ?FRYY$BIIF ryy "  #*#L) "))) X*/**.U #U U p   C )C  C L"))"")), g 2g  g T Z #6Z  Z z D )<D  D NL !4L L ^ r*