rh|pdZddlmZmZmZddlZddlZddlmZddlm Z m Z ddl m Z ddl mZdd lmZmZmZdd lmZmZdd lmZmZdd lmZmZmZd dlmZej>e Z!GddejDZ#GddejDZ$ d4dejDdejJdejJdejJdeejJde&de&fdZ'GddejDZ(GddejDZ)Gdd ejDZ*Gd!d"ejDZ+Gd#d$ejDZ,Gd%d&eZ-Gd'd(ejDZ.Gd)d*ejDZ/eGd+d,eZ0eGd-d.e0Z1ed/0Gd1d2e0Z2gd3Z3y)5zPyTorch ViViT model.)CallableOptionalUnionN)nn)CrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) VivitConfigc0eZdZdZfdZddefdZxZS)VivitTubeletEmbeddingsa Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). ct||j|_|j|_|j|_|j|j dz|j|j dzz|j|j dzz|_|j|_tj|j|j|j|j|_ y)Nrr) kernel_sizestride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_size embed_dimrConv3d num_channels projectionselfconfig __class__s {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/vivit/modeling_vivit.pyrzVivitTubeletEmbeddings.__init__/s  ++ ++ -- __ 2 2$//!"44 6$//!"44 6   ++))   !3!3ATAT]c]p]p interpolate_pos_encodingc \|j\}}}}}|sP||jk7s||jk7r2td|d|d|jdd|jdd |jddddd }|j |}|j dj dd}|S) NzImage image size (*z) doesn't match model (rrz).rr )shaper! ValueErrorpermuter)flatten transpose) r+ pixel_valuesr0 batch_sizer r(heightwidthxs r.forwardzVivitTubeletEmbeddings.forward?s>J>P>P; J fe'Vt-F%SWSbSbJb$VHAeW4KDOO\]L^K__`aeapapqras`ttvw  $++Aq!Q: OOL ) IIaL " "1a (r/F)__name__ __module__ __qualname____doc__rboolr> __classcell__r-s@r.rr$s dr/rcpeZdZdZfdZdej dededej fdZd de fd Z xZ S) VivitEmbeddingsz Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. ct|tjt j dd|j |_t||_ tjt j d|jjdz|j |_ tj|j|_|jdd|_||_y)Nr)rrr Parametertorchzerosr% cls_tokenrpatch_embeddingsr$position_embeddingsDropouthidden_dropout_probdropoutr"r#r,r*s r.rzVivitEmbeddings.__init__Ws ekk!Q8J8J&KL 6v >#%<< KK400<zVivitEmbeddings.forwards>J>P>P; J fe**&>>J\\*- r/r?) r@rArBrCrrKTensorintr0rDr>rErFs@r.rHrHPsL &D5<<&D&DUX&D]b]i]i&DPdr/rHmodulequerykeyvalueattention_maskscalingrRc tj||jdd|z}tjj |dtj j|j}tjj|||j}|||z}tj||} | jddj} | |fS)NrV)r\dtype)ptrainingrr) rKmatmulr8rr`softmaxfloat32torxrRrz contiguous) rprqrrrsrtrurRkwargs attn_weights attn_outputs r.eager_attention_forwardrs<<s}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#n4 ,,|U3K''1-88:K  $$r/c eZdZdeddffd Z ddeejdede e ejejfe ejffdZ xZ S) VivitSelfAttentionr,rTNc2t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |j|_ |jdz|_ d|_ tj|j|j|j |_tj|j|j|j |_tj|j|j|j |_y) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)rrr%num_attention_headshasattrr5r,roattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probru is_causalrLinearqkv_biasrqrrrsr*s r.rzVivitSelfAttention.__init__sF     : : :a ?PVXhHi"6#5#5"67334A7   #)#=#= #&v'9'9FzVivitSelfAttention.forwards %2$7$7! J HH] # T*b$":":DO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=/2O\M]r/NF) r@rArBrrrrKrnrDrtupler>rErFs@r.rrsr]{]t].-1"' 1ELL)1 1 uU\\5<</0% 2EE F 1r/rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r,rTNct|tj|j|j|_tj |j|_yN) rrrrr%denserPrQrRr*s r.rzVivitSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r/r input_tensorcJ|j|}|j|}|SrrrRr+rrs r.r>zVivitSelfOutput.forwards$ =1  ]3 r/) r@rArBrCrrrKrnr>rErFs@r.rr sD >{>t> U\\RWR^R^r/rc eZdZdeddffd ZdeeddfdZ d dejde ejd e de e ejejfe ejffd ZxZS) VivitAttentionr,rTNct|t||_t ||_t |_yr)rrr attentionroutputset pruned_headsr*s r.rzVivitAttention.__init__s0 +F3%f- Er/headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr[)lenrrrrrrrqrrrsrrrunion)r+rindexs r. prune_headszVivitAttention.prune_heads#s u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r/rrrch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r+rrr self_outputsattention_outputrs r.r>zVivitAttention.forward5sE ~~mY@QR ;;|A F#% QR(88r/r)r@rArBrrrrorrKrnrrDrrr>rErFs@r.rrs"{"t" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F r/rc$eZdZfdZdZxZS)VivitIntermediatecPt|tj|j|j |_tj|j|_ t|jtrt|j|_y|j|_yr)rrrrr%intermediate_sizerrPrQrR isinstance hidden_actstrr intermediate_act_fnr*s r.rzVivitIntermediate.__init__Dss YYv1163K3KL zz&"<"<= f'' -'-f.?.?'@D $'-'8'8D $r/cl|j|}|j|}|j|}|Sr)rrrR)r+rs r.r>zVivitIntermediate.forwardMs4 =1 00?  ]3 r/r@rArBrr>rErFs@r.rrCs 9r/rc$eZdZfdZdZxZS) VivitOutputct|tj|j|j |_tj|j|_ yr) rrrrrr%rrPrQrRr*s r.rzVivitOutput.__init__VsB YYv779K9KL zz&"<"<= r/cT|j|}|j|}||z}|Srrrs r.r>zVivitOutput.forward[s. =1  ]3 % 4 r/rrFs@r.rrUs > r/rc*eZdZdZfdZddZxZS) VivitLayerzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.crt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nreps)rrchunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr%layer_norm_epslayernorm_beforelayernorm_afterr*s r.rzVivitLayer.__init__hs '-'E'E$'/-f5!&) " V-?-?VEZEZ [!||F,>,>FDYDYZr/c|j|j|||}|d}|dd}||z}|j|}|j|}|j ||}|f|z}|S)N)rrr)rrrrr)r+rrrself_attention_outputsrr layer_outputs r.r>zVivitLayer.forwardrs!%  ! !- 0 / "0"  2!4(,)=8 ++M: ((6 {{<? /G+r/r)r@rArBrCrr>rErFs@r.rresX[r/rc.eZdZfdZ ddZxZS) VivitEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rrr,r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r+r,rr-s r.rzVivitEncoder.__init__sN  ]]fF^F^@_#`1Jv$6#`a &+#$asA#c|rdnd}|rdnd}t|jD]1\}} |r||fz}|||nd} | || |} | d}|s)|| dfz}3|r||fz}|std|||fDSt|||S)Nrrc3&K|] }|| ywrr).0vs r. z'VivitEncoder.forward..smq_`_lms)last_hidden_stater attentions) enumeraterrr ) r+rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r.r>zVivitEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO(IZ[M)!,M &9]1=MzVivitPooler.forwards6+1a40 #56  6 r/rrFs@r.rrs $ r/rc>eZdZUeed<dZdZdZgZdZ dZ dZ dZ dZ y)VivitPreTrainedModelr,vivitr9Tct|tjtjfrm|jj j d|jj|j%|jj jyyt|tjrz|jj j d|jj|j2|jj |jjyyt|tjrJ|jj j|jj jdyt|trI|j j j|j"j jyy)zInitialize the weightsr)meanstdNg?)rrrr'weightdatanormal_r,initializer_rangerzero_ Embedding padding_idxrfill_rHrMrO)r+rps r. _init_weightsz"VivitPreTrainedModel._init_weightssI fryy"))4 5 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  0    ! ! ' ' )  & & + + 1 1 31r/N)r@rArBr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrrr/r.rrs; $O&*#N"&4r/rceZdZd fd ZdZdZe d deejdeejdee dee de d ee d e e eje ffd ZxZS) VivitModelct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)rrr,rHrSrencoderrrr%r layernormrpooler post_init)r+r,add_pooling_layerr-s r.rzVivitModel.__init__sk   )&1#F+ f&8&8f>S>ST->k&)D  r/c.|jjSr)rSrN)r+s r.get_input_embeddingszVivitModel.get_input_embeddingss///r/c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrr)r+heads_to_prunerrs r. _prune_headszVivitModel._prune_headssE+002 CLE5 LL  u % / / ; ;E B Cr/r9rrrr0rrTc||n|jj}||n|jj}||n|jj}| t d|j ||jj }|j||}|j|||||}|d} |j| } |j|j| nd} |s | | f|ddzSt| | |j|jS)a Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesrk)rrrrrr)r pooler_outputrr)r,rruse_return_dictr5 get_head_maskrrSrrrr rr) r+r9rrrr0rembedding_outputencoder_outputssequence_outputrs r.r>zVivitModel.forwards*h2C1N-TXT_T_TqTq$8$D $++JjJj &1%rErFs@r.rrs"0 C5915,0/3).&*s u001s E--.s $D> s 'tn s #' s d^s  uU&&')CC Ds s r/ra ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ) custom_introceZdZfdZe d deejdeejdeejdee dee de dee d e e eje ffd Z xZS) VivitForVideoClassificationc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jy)NF)rr) rr num_labelsrrrrr%Identity classifierrr*s r.rz$VivitForVideoClassification.__init__ss   ++%@ OUN_N_bcNc"))F$6$68I8IJikititiv r/r9rlabelsrrr0rrTc||n|jj}|j||||||}|d} |j| dddddf} d} |}|jdk(r2t } | | j d|j d} n 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```N)rrrr0rrrrVr)losslogitsrr) r,r(rr4r2rrbrrrr)r+r9rr5rrr0rrr,r8r7loss_fctrs r.r>z#VivitForVideoClassification.forwardsz&1%rErFs@r.r0r0~s 5915-1,0/3).&*~ u001~ E--.~ ))* ~ $D> ~ 'tn ~ #'~ d^~  uU&&')>> ?~ ~ r/r0)rrr0)r)4rCtypingrrrrKtorch.utils.checkpointrtorch.nnrr activationsr modeling_layersr modeling_outputsr r rmodeling_utilsrr pytorch_utilsrrutilsrrrconfiguration_vivitr get_loggerr@rModulerrHrnfloatrrrrrrrrrrrr0__all__rr/r.rIs,, .!9bbFQ77,   H %)RYY)XLbiiLn% II% <<% % << % U\\* %  %%>FFTbii&$RYY$N $ "))  $+$N( 299( V ")) 4?44>T %T T n  L "6L  L ^ Pr/