rh$dZddlZddlmZddlmZmZmZddl Z ddl Z ddl m Z ddl m Z mZmZddlmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZmZdd lm Z m!Z!m"Z"m#Z#ddl$m%Z%e"jLe'Z(Gdde jRZ*Gdde jRZ+ d>de jRde jXde jXde jXdee jXde-de-fdZ.Gdde jRZ/Gdde jRZ0Gd d!e jRZ1Gd"d#e jRZ2Gd$d%e jRZ3Gd&d'eZ4Gd(d)e jRZ5e!Gd*d+eZ6e!Gd,d-e6Z7Gd.d/e jRZ8e!d01Gd2d3e6Z9e!d41Gd5d6e6Z:ee!d71Gd8d9e Z;e!d:1Gd;d "55<!>z6SX!Y "44 \\*- r7)FNF)__name__ __module__ __qualname____doc__rboolr$r&TensorintrSr BoolTensorrc __classcell__r5s@r6rr+s ,z ,4 ,D ,&M5<<&M&MUX&M]b]i]i&MV7;). ll"%"2"23#'   r7rcZeZdZdZfdZdej dej fdZxZS)r,z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)r#r$ image_sizer3 num_channelsr( isinstance collectionsabcIterabler.rConv2d projection)r4rrsr3rtr(r.r5s r6r$zDeiTPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir7rTr c|j\}}}}||jk7r td|j|j dj dd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r"r)rDrt ValueErrorrzflatten transpose)r4rTr\rtr9r:xs r6rczDeiTPatchEmbeddings.forwardsa2>2D2D/ L&% 4,, ,w  OOL ) 1 1! 4 > >q! Dr7) rerfrgrhr$r&rjrcrmrns@r6r,r,s) jELLU\\r7r,modulequerykeyvalueattention_maskscalingr2c tj||jdd|z}tjj |dtj j|j}tjj|||j}|||z}tj||} | jddj} | |fS)Nr<)rCdtype)ptrainingrr") r&matmulr~rrIsoftmaxfloat32torr2r contiguous) rrrrrrr2kwargs attn_weights attn_outputs r6eager_attention_forwardrs<<s}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#n4 ,,|U3K''1-88:K  $$r7c eZdZdeddffd Z ddeejdede e ejejfe ejffdZ xZ S) DeiTSelfAttentionrr Nc2t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |j|_ |jdz|_ d|_ tj|j|j|j |_tj|j|j|j |_tj|j|j|j |_y) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)r#r$r(num_attention_headshasattrr|rrkattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasrrrr4rr5s r6r$zDeiTSelfAttention.__init__sF     : : :a ?PVXhHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=/2O\M]r7rd) rerfrgrr$rr&rjrirtuplercrmrns@r6rrsr]z]d].-1"' 1ELL)1 1 uU\\5<</0% 2EE F 1r7rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rr Nct|tj|j|j|_tj |j|_yN) r#r$rrr(denser0r1r2rs r6r$zDeiTSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r7r input_tensorcJ|j|}|j|}|Srrr2r4rrs r6rczDeiTSelfOutput.forwards$ =1  ]3 r7) rerfrgrhrr$r&rjrcrmrns@r6rrsD >z>d> U\\RWR^R^r7rc eZdZdeddffd ZdeeddfdZ d dejde ejd e de e ejejfe ejffd ZxZS) DeiTAttentionrr Nct|t||_t ||_t |_yr)r#r$r attentionroutputset pruned_headsrs r6r$zDeiTAttention.__init__"s0 *62$V, Er7headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)NrrrB)lenrrrrrrrrrrrrunion)r4rindexs r6 prune_headszDeiTAttention.prune_heads(s u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r7rrrch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r4rrr self_outputsattention_outputrs r6rczDeiTAttention.forward:sE ~~mY@QR ;;|A F#% QR(88r7rd)rerfrgrr$rrkrr&rjrrirrrcrmrns@r6rr!s"z"d" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F r7rc`eZdZdeddffd Zdej dej fdZxZS)DeiTIntermediaterr Nct|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r#r$rrr(intermediate_sizerru hidden_actstrr intermediate_act_fnrs r6r$zDeiTIntermediate.__init__Js] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r7rcJ|j|}|j|}|Sr)rr)r4rs r6rczDeiTIntermediate.forwardRs& =1 00? r7 rerfrgrr$r&rjrcrmrns@r6rrIs19z9d9U\\ellr7rcxeZdZdeddffd Zdej dej dej fdZxZS) DeiTOutputrr Nct|tj|j|j |_tj|j|_ yr) r#r$rrrr(rr0r1r2rs r6r$zDeiTOutput.__init__[sB YYv779K9KL zz&"<"<= r7rrcT|j|}|j|}||z}|Srrrs r6rczDeiTOutput.forward`s. =1  ]3 % 4 r7rrns@r6rrZs?>z>d> U\\RWR^R^r7rc eZdZdZdeddffd Z d dejdeejde de e ejejfe ejffd Z xZ S) DeiTLayerz?This corresponds to the Block class in the timm implementation.rr Ncrt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nreps)r#r$chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr(layer_norm_epslayernorm_beforelayernorm_afterrs r6r$zDeiTLayer.__init__ms '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr7rrrc|j|j|||}|d}|dd}||z}|j|}|j|}|j ||}|f|z}|S)N)rrr)rrrrr)r4rrrself_attention_outputsrr layer_outputs r6rczDeiTLayer.forwardws "&  ! !- 0 /"0"  2!4(,)=8 ++M: ((6 {{<? /G+r7rd)rerfrgrhrr$r&rjrrirrrcrmrns@r6rrjsI[z[d[-1"' ||ELL)  uU\\5<</0% 2EE F r7rceZdZdeddffd Z d dej deej deded ede e e ff d Z xZ S) DeiTEncoderrr Nct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wrd) r#r$rr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r4rr[r5s r6r$zDeiTEncoder.__init__sN  ]]uVE]E]?^#_!If$5#_` &+#$`sA#rrroutput_hidden_states return_dictc|rdnd}|rdnd}t|jD]1\}} |r||fz}|||nd} | || |} | d}|s)|| dfz}3|r||fz}|std|||fDSt|||S)Nrrc3&K|] }|| ywrr).0vs r6 z&DeiTEncoder.forward..smq_`_lms)last_hidden_stater attentions) enumeraterrr) r4rrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r6rczDeiTEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO(IZ[M)!,M &9]1=MS>ST,=j(4  r7c.|jjSr)r8r-)r4s r6get_input_embeddingszDeiTModel.get_input_embeddingss///r7c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr rrr)r4heads_to_prunerrs r6 _prune_headszDeiTModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr7rTrUrrrrrSc||n|jj}||n|jj}||n|jj}| t d|j ||jj }|jjjjj}|j|k7r|j|}|j|||} |j| ||||} | d} |j| } |j|j| nd} |s| | | fn| f} | | ddzSt!| | | j"| j$S)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rUrS)rrrrrr)r pooler_outputrr)rrruse_return_dictr| get_head_maskrr8r-rzr rrr r!r#rrr)r4rTrUrrrrrSexpected_dtypeembedding_outputencoder_outputssequence_output pooled_output head_outputss r6rczDeiTModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%:L?? /Tl+ ,, /!5# ' *!,..98< 8O O4UY ?L?XO];_n^pL/!""55 5)-')77&11   r7)TFNNNNNNF)rerfrgrrir$r,r&r*rrr&rjrlrrrrcrmrns@r6rrszd[_lp&0&90C046:,0,0/3&*).; u||,; "%"2"23; ELL) ; $D> ; 'tn ; d^; #';  u00 1; ; r7rc*eZdZdeffd ZdZxZS)r"rct|tj|j|j |_t|j|_ yr) r#r$rrr(pooler_output_sizerr pooler_act activationrs r6r$zDeiTPooler.__init__Bs> YYv1163L3LM  !2!23r7c\|dddf}|j|}|j|}|S)Nr)rr:)r4rfirst_token_tensorr3s r6rczDeiTPooler.forwardGs6+1a40 #56  6 r7)rerfrgrr$rcrmrns@r6r"r"As4z4 r7r"ad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introceZdZdeddffd Ze d deejdeejdeejdee d ee d ee d e de e e ffd ZxZS)DeiTForMaskedImageModelingrr NcNt||t|dd|_t j t j |j|jdz|jzdt j|j|_ |jy)NFT)rrr"r) in_channels out_channelsrq) r#r$rrr Sequentialryr(encoder_stridert PixelShuffledecoderr$rs r6r$z#DeiTForMaskedImageModeling.__init__]s  fdS }} II"..#22A58K8KK  OOF11 2    r7rTrUrrrrrSc ||n|jj}|j|||||||}|d} | ddddf} | j\} } } t | dzx} }| j dddj | | | |} |j| }d}||jj|jjz}|j d||}|j|jjdj|jjdjdj}tjj||d }||zj!|j!d zz |jj"z }|s|f|ddz}||f|zS|St%|||j&|j( S) a; bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```N)rUrrrrrSrrr<r=r"none) reductiongh㈵>)lossreconstructionrr)rr-rrDrkrHrGrFrsr3repeat_interleaverYrrrIl1_losssumrtrrr)r4rTrUrrrrrSrr2r\sequence_lengthrtr9r:reconstructed_pixel_valuesmasked_im_lossr?r_reconstruction_lossrs r6rcz"DeiTForMaskedImageModeling.forwardnsL&1% R 'tn R d^R #'R  u// 0R R r7r?z DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. ceZdZdeddffd Ze d deejdeejdeejdee d ee d ee d e de e e ffd Z xZS)DeiTForImageClassificationrr Nc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jyNF)rr) r#r$ num_labelsrrrrr(Identity classifierr$rs r6r$z#DeiTForImageClassification.__init__ss   ++f> OUN_N_bcNc"))F$6$68I8IJikititiv r7rTrlabelsrrrrScb||n|jj}|j||||||}|d} |j| dddddf} d} ||j | j }|jj |jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |jdk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd|j|jd} n,|jj dk(rt!} | | |} |s| f|ddz} | | f| zS| St#| | |j$|j& S) aZ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: Polaroid camera, Polaroid Land camera ```NrrrrrSrr regressionsingle_label_classificationmulti_label_classificationr<)rJlogitsrr)rr-rrYrdevice problem_typerWrr&longrkr squeezer rKrrrr)r4rTrrZrrrrSrr2r`rJloss_fctrs r6rcz"DeiTForImageClassification.forwardsT&1% Y 'tn Y d^Y #'Y  u++ ,Y Y r7rTzC Output type of [`DeiTForImageClassificationWithTeacher`]. ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<y)+DeiTForImageClassificationWithTeacherOutputaj logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr` cls_logitsdistillation_logitsrr)rerfrgrhr`rr& FloatTensorrrhrirrrrr7r6rgrg4s} +/FHU&& '..2J**+27;%"3"34;8