rhLddlmZddlmZmZmZddlZddlmZddlm Z m Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZdd lmZddlmZeedGddeZ GddejBZ"GddejBZ#Gdde Z$GddejBZ%GddejBZ&GddejBZ'd Z(dQd!Z)d"ejTd#e+d$ejTfd%Z, dRd&ejBd'ejTd(ejTd)ejTd*eejTd+e-d,e-d-eefd.Z.Gd/d0ejBZ/Gd1d2ejBZ0Gd3d4ejBZ1Gd5d6e Z2Gd7d8ejBZ3Gd9d:ejBZ4Gd;deZ6ed?Gd@dAe6Z7dBejTdCe+d)ee8e-e+fd$ejTfdDZ9 dSdEee+ejTfdFee+ejTfdGe8dHeejtdIeejvd$ejTf dJZgdPZ?y)U) dataclass)CallableOptionalUnionN)nn)ACT2CLSACT2FN)GradientCheckpointingLayer)BackboneOutput)ROPE_INIT_FUNCTIONS)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple torch_int)check_model_inputs)EfficientLoFTRConfiga Base class for outputs of keypoint matching models. Due to the nature of keypoint detection and matching, the number of keypoints is not fixed and can vary from image to image, which makes batching non-trivial. In the batch of images, the maximum number of matches is set as the dimension of the matches and matching scores. The mask tensor is used to indicate which values in the keypoints, matches and matching_scores tensors are keypoint matching information. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<y)KeypointMatchingOutputa matches (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Index of keypoint matched in the other image. matching_scores (`torch.FloatTensor` of shape `(batch_size, 2, num_matches)`): Scores of predicted matches. keypoints (`torch.FloatTensor` of shape `(batch_size, num_keypoints, 2)`): Absolute (x, y) coordinates of predicted keypoints in a given image. hidden_states (`tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(batch_size, 2, num_channels, num_keypoints)`, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) attentions (`tuple[torch.FloatTensor, ...]`, *optional*): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, 2, num_heads, num_keypoints, num_keypoints)`, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) Nmatchesmatching_scores keypoints hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrtupler /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/efficientloftr/modeling_efficientloftr.pyrr%s| ,0GXe'' (/37OXe//07-1Ix))*18-?)] )),7 '' 2 #++Aq!Q7 %%aAq1  ,/ $ ) 4333r*ra) r!r"r#rr6r%rcrr(r`rerfs@r+rrsU.3.,9=4||4 ( 54 u||U\\) * 4r*rc|ddddf}|ddddf}tj| |gdjd}|S)N.r2rr1rT)r%stackflatten)rLx1x2rot_xs r+ rotate_halfrsL 3!8B 319B KK"b r * 2 22 6E Lr*c6|j}|j}|j}|j|}|j|}||zt||zz}||zt||zz}|j ||j |fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )rW)rWr9r>rr_) qkr]r\rM unsqueeze_dimrWq_embedk_embeds r+apply_rotary_pos_embrs( GGE  A  A -- &C -- &C3w;q>C/0G3w;q>C/0G ::E: "GJJUJ$; ;;r*rn_reprNc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)shaper:reshape)rrbatchnum_key_value_headsslenhead_dims r+ repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr*modulequerykeyvalueattention_maskscalingdropoutkwargsc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr2rrr1)rUrW)ptrainingr)rnum_key_value_groupsr%matmul transposerr functionalsoftmaxfloat32r_rWrr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r+eager_attention_forwardr&s3 ; ;$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r*rrposition_embeddingsrrNc |j\}}}|jdd}|j|j||d|} |du} | r|n|} |j| j||d|} |j | j||d|j j dd} ||\}}t| | ||d\} } | j||d|j j dd} | j||d|j j dd} t}|jjdk7rt|jj}||| | | fd|jsdn |j|jd|\}}|jg|dj!}|j#|}||fS)Nr1rr2)reager)rrr)rrviewrrrrrrr._attn_implementationrrrrrrr)rBrrrr batch_sizeseq_lenrU input_shaperrcurrent_statesrrr]r\attention_interfacerrs r+r`zEfficientLoFTRAttention.forward[s$1#6#6 GS#))#2. {{=166z7BPST 2$>2D.-[[055j'2sS {{>277 GRQUQ^Q^_iijkmno  **HC';L*VY[^no'p $L*#((Wb$--PZZ[\^_` __Z"dmmLVVWXZ[\ (? ;; + +w 6"9$++:Z:Z"[ $7     %  #}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r*ry)r!r"r#r$rrr6r%rcrr(rrr`rerfs@r+rr@sG 3  49=KO *)||*) ( 5*)&eELL%,,,F&GH *) +, *) u||Xell33 4 *)r*rc\eZdZdeffd Zdej dej fdZxZS)EfficientLoFTRMLPr.c0t||j}|j}t j |dz|d|_t|j|_ t j ||d|_ t j||_ y)Nr2Fr) r5r6r@intermediate_sizerrfc1r mlp_activation_functionrrfc2r layer_normrBr.r@rrKs r+r6zEfficientLoFTRMLP.__init__su (( "4499[1_.?eL !?!?@99. %H,,{3r*rrNc|j|}|j|}|j|}|j|}|Sra)rrrrr)rBrs r+r`zEfficientLoFTRMLP.forwards@/  6 /  6 r*) r!r"r#rr6r%rcr`rerfs@r+rrs+434U\\ellr*rc eZdZdedeffd Z d dejdeejdee ejejfde e dejf d Z xZ S) !EfficientLoFTRAggregatedAttentionr.rct||j|_t||_t |||_t||_yra) r5r6rr aggregationr attentionrmlprs r+r6z*EfficientLoFTRAggregatedAttention.__init__sC )/)I)I&9&A0C$V,r*rrrrrNc \|j\}}}}|j||\}} |j\}} } }|j|d|}| j|d|} |j|| fd|i|\} }| j ddd} | j||| | } t j jj| |jdd} t j|| gd } | j ddd d} |j| }|j dd dd}||z}|S) Nr1rrr2rbilinearF scale_factormode align_cornersrTr) rr rr rr%rr interpolatercatr )rBrrrrr embed_dimrDaggregated_hidden_states aggregated_encoder_hidden_states aggregated_h aggregated_wrintermediate_states output_statess r+r`z)EfficientLoFTRAggregatedAttention.forwardso'4&9&9# Iq!FJEUEU 0F B "B,D+I+I(<q$<#C#CJPRT]#^ +K+S+ST^`bdm+n(' $ , !4    Q"))!Q2 !))*i|\ hh))55 d&D&D:ej6 $ii (D!L199!Q1E!45 %--aAq9 % 5 r*ry)r!r"r#rrr6r%rcrr(rrr`rerfs@r+rrs-3--9=KO '||' ( 5'&eELL%,,,F&GH ' +, '  'r*rceZdZdedeffd Zdejdeejejfde e dejfdZ xZ S) *EfficientLoFTRLocalFeatureTransformerLayerr.rcft|t|||_t|||_yra)r5r6rself_attentioncross_attentionrs r+r6z3EfficientLoFTRLocalFeatureTransformerLayer.__init__s, ? R@Sr*rrrrNc B|j\}}}}}|jd|||}|j|fd|i|}|jdd|||} | jd} | jd|||} |j|| fi|}|j|d|||}|S)Nr1rr2r)rrrflipr ) rBrrrrrDrheightwidthrs r+r`z2EfficientLoFTRLocalFeatureTransformerLayer.forwards 3@2E2E/ Ay&%%--b)VUK +++MmObmflm - 5 5b!YPU V 5 : :1 = 5 = =b)VUZ [,,,]+,   r*rceZdZdeffd Zdej deej ej fdee dej fdZ xZ S)%EfficientLoFTRLocalFeatureTransformerr.c t|tjt |j Dcgc]}t ||c}|_ycc}w)N)r)r5r6rrrnum_attention_layersrlayers)rBr.irKs r+r6z.EfficientLoFTRLocalFeatureTransformer.__init__sM mmv::; ;6QO    sArrrrNc >|jD] }||fd|i|}|S)Nr)r))rBrrrlayers r+r`z-EfficientLoFTRLocalFeatureTransformer.forwards5 [[ dE!-cEXc\bcM dr*) r!r"r#rr6r%rcr(rrr`rerfs@r+r&r&s[ 3 ||#5<<#=>+,   r*r&c|eZdZdededeffd ZdejdejdejfdZxZ S) EfficientLoFTROutConvBlockr.r@rcNt|tj||dddd|_tj||dddd|_tj ||_t|j|_ tj||dddd|_ y)NrrFrurvrjrkr) r5r6rrl out_conv1 out_conv2rn batch_normr rrr out_conv3rs r+r6z#EfficientLoFTROutConvBlock.__init__s ;0AqYZdelqr 0aST[` ..):;!&"@"@AC#4kqYZdelqrr*rresidual_statesrNc|j|}||z}|j|}|j|}|j|}|j |}t j j|ddd}|SNg@rFr)r1r2r3rrr4rrr)rBrr5s r+r`z"EfficientLoFTROutConvBlock.forward s|..9)M9..9///:///:..9--33 #Je4 r*rrfs@r+r.r.sL s3 s# sZ] s U\\ ELL UZUaUa r*r.ceZdZdeffd Zdej deej dej fdZdej deej de ej ej ffd Z xZ S) EfficientLoFTRFineFusionLayerr.cpt||j|_|j}t j |d|ddddd|_t j|_tdt|D]3}t|||||dz }|jj|5y)NrrFr0) r5r6fine_kernel_sizefine_fusion_dimsrrlout_convrout_conv_layersrrr.r)rBr.r<r*r=rKs r+r6z&EfficientLoFTRFineFusionLayer.__init__s  & 7 7!22 Q !1!!4!AWX_d   "}}q#./0 2A1&:J1:MO_`ade`eOfgH  ' ' 1 2r*rr5rNc|j|}tjj|ddd}t |j D]\}}||||}|Sr7)r=rrr enumerater>)rBrr5r*r,s r+forward_pyramidz-EfficientLoFTRFineFusionLayer.forward_pyramid&so  m4  11 *E2 "$"6"67 EHAu!-1CDM Er*coarse_featuresresidual_featuresc|j\}}}}}|jd|||}tt|}|j ||}|j\}} } } |j|d| | | }|dddf} |dddf} t | |z}t jj| |j|d} | j\}}}| j|d|jdz|} | jdddd} t jj| |jdz|d} | j|d|jdzdz|} | jdddd} | | fS)a? For each image pair, compute the fine features of pixels. In both images, compute a patch of fine features center cropped around each coarse pixel. In the first image, the feature patch is kernel_size large and long. In the second image, it is (kernel_size + 2) large and long. r1r2Nrrrr) rrrreversedrArrrunfoldr;r)rBrBrCrrDr coarse_height coarse_width fine_featuresfine_embed_dim fine_height fine_widthfine_features_0fine_features_1rvrs r+r`z%EfficientLoFTRFineFusionLayer.forward4sAP@U@U= Ay-)11"iP\] *;!<=,,_>OP 5B5H5H2>; %--j!^[Zde '1-'1-[M12--.. )>)>vWX/ (-- 1g)11*b$BWBWYZBZ\cd)11!Q1=--.. )>)>)B6[\/ *11*b4CXCX[\C\abBbdkl)11!Q1=//r*) r!r"r#rr6r%rcrrAr(r`rerfs@r+r9r9s 23 2 || ell+   '0'0  -'0 u||U\\) * '0r*r9ceZdZdZeZdZdZdZdZ dZ e e dZ dejddfd Zdej$dej$fd Zy) EfficientLoFTRPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. efficientloftr pixel_valuesT)rr rrNNc6t|tjtjtjtj frm|j jjd|jj|j%|jjjyyt|tjrJ|jjj|j jjdyy)zInitialize the weightsr)meanstdNg?)rXrrrlConv1drnweightdatanormal_r.initializer_rangerkzero_rfill_)rBrs r+ _init_weightsz+EfficientLoFTRPreTrainedModel._init_weightsps fryy"))RYYO P MM   & &CT[[5R5R & S{{&   &&('  - KK   " " $ MM   $ $S ).r*c:|dddddddfdddddddfS)aK Assuming pixel_values has shape (batch_size, 3, height, width), and that all channels values are the same, extract the first channel value to get a tensor of shape (batch_size, 1, height, width) for EfficientLoFTR. This is a workaround for the issue discussed in : https://github.com/huggingface/transformers/pull/25786#issuecomment-1730176446 Args: pixel_values: torch.FloatTensor of shape (batch_size, 3, height, width) Returns: pixel_values: torch.FloatTensor of shape (batch_size, 1, height, width) Nrr))rBrRs r+ extract_one_channel_pixel_valuesz>EfficientLoFTRPreTrainedModel.extract_one_channel_pixel_values{s$Aq!QJ'4A 66r*)r!r"r#r$r config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_supports_sdpar{r_can_record_outputsrModuler]r%r&r_r)r*r+rPrP^so (L($O&*#N2- *BII*$*7U=N=N7SXSdSd7r*rPza EfficientLoFTR model taking images as inputs and outputting the features of the images. c eZdZdeffd Zee ddejde ejde e de fdZxZS) EfficientLoFTRModelr.ct||||_t||_t ||_t||_|jy)N)r.) r5r6r.rbackboner&local_feature_transformerr- rotary_emb post_initrBr.rKs r+r6zEfficientLoFTRModel.__init__sF   +F3 )Nv)V&7vF r*rRlabelsrrNc | td|jdk7s|jddk7r td|j\}}}}}|j |dz|||}|j |}|j |} | d} | dd} | jdd\} } }|j| \}}|j|dzdddj |dzd| }|j|dzdddj |dzd| }||f}| j |d| | |} |j| fd |i|} | ft| z} t| S)  Examples: ```python >>> from transformers import AutoImageProcessor, AutoModel >>> import torch >>> from PIL import Image >>> import requests >>> url = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/phototourism_sample_images/london_bridge_78916675_4568141288.jpg?raw=true" >>> image1 = Image.open(requests.get(url, stream=True).raw) >>> url = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/phototourism_sample_images/london_bridge_19481797_2295892421.jpg?raw=true" >>> image2 = Image.open(requests.get(url, stream=True).raw) >>> images = [image1, image2] >>> processor = AutoImageProcessor.from_pretrained("zju-community/efficient_loftr") >>> model = AutoModel.from_pretrained("zju-community/efficient_loftr") >>> with torch.no_grad(): >>> inputs = processor(images, return_tensors="pt") >>> outputs = model(**inputs) ```Nz>EfficientLoFTR is not trainable, no labels should be provided.rr2zOInput must be a 5D tensor of shape (batch_size, 2, num_channels, height, width)r1r) feature_maps) ValueErrorndimsizerrr_rkrmr:rlr(r )rBrRrprrrDchannelsr#r$featuresrBrCcoarse_embed_dimrGrHr]r\rs r+r`zEfficientLoFTRModel.forwards<  ]^ ^    !\%6%6q%9Q%>no o1=1C1C. Ax#++JNHfeT <<\J ==."2,$SbM8G8M8Mbc8R5-???3SjjaR4<>> create_meshgrid(2, 2) tensor([[[[-1., -1.], [ 1., -1.]], [[-1., 1.], [ 1., 1.]]]]) >>> create_meshgrid(2, 2, normalized_coordinates=False) tensor([[[[0., 0.], [1., 0.]], [[0., 1.], [1., 1.]]]]) rrrV?r2ij)indexingr1rT)r%linspacermeshgridrr>)r#r$rrCrWxsysgrids r+create_meshgridrsb 519eF% HB 6A:vfE JBEAI$ )FQJ#% * ;;u~~b"t<" ED <<1a * *1 -D Kr*inputc|j\}}}}t||||j}|j|j}|dj d}|dj d}|j ||d} tj|| zdd} tj|| zdd} tj| | gd} | j ||dS)aL Copied from kornia library : kornia/geometry/subpix/dsnt.py:76 Compute the expectation of coordinate values using spatial probabilities. The input heatmap is assumed to represent a valid spatial probability distribution, which can be achieved using :func:`~kornia.geometry.subpixel.spatial_softmax2d`. Args: input (`torch.Tensor` of shape `(batch_size, embed_dim, height, width)`): The input tensor representing dense spatial probabilities. normalized_coordinates (`bool`): Whether to return the coordinates normalized in the range of :math:`[-1, 1]`. Otherwise, it will return the coordinates in the range of the input shape. Returns: output (`torch.Tensor` of shape `(batch_size, embed_dim, 2)`) Expected value of the 2D coordinates. Output order of the coordinates is (x, y). Examples: >>> heatmaps = torch.tensor([[[ ... [0., 0., 0.], ... [0., 0., 0.], ... [0., 1., 0.]]]]) >>> spatial_expectation2d(heatmaps, False) tensor([[[1., 2.]]]) .rr1.rTkeepdimr2) rrrCr_rWrrr%sumr) rrrrr#r$rpos_xpos_y input_flat expected_y expected_xoutputs r+spatial_expectation2dr/s8,1;;(J 65 65*@%,, OD 775;; D L  $E L  $EJ 26J5:-r4@J5:-r4@J YY J/ 4F ;;z9a 00r*z[ EfficientLoFTR model taking images as inputs and outputting the matching of them. c eZdZdZdeffd ZdejdeejejffdZ dejde deejejejffd Z d ejd ejd e d e deejejff dZ dejdejd ejd e d e dejf dZdejdejd ejd e dejf dZee ddej$deej(deedefdZxZS)!EfficientLoFTRForKeypointMatchingaEfficientLoFTR dense image matcher Given two images, we determine the correspondences by: 1. Extracting coarse and fine features through a backbone 2. Transforming coarse features through self and cross attention 3. Matching coarse features to obtain coarse coordinates of matches 4. Obtaining full resolution fine features by fusing transformed and backbone coarse features 5. Refining the coarse matches using fine feature patches centered at each coarse match in a two-stage refinement Yifan Wang, Xingyi He, Sida Peng, Dongli Tan and Xiaowei Zhou. Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed In CVPR, 2024. https://arxiv.org/abs/2403.04765 r.ct||||_t||_t ||_|jyra)r5r6r.rirQr9refinement_layerrnros r+r6z*EfficientLoFTRForKeypointMatching.__init__ss;   1&9 =f E r*scoresrNc|j\}}}}}|j|||z||z}|jddj}|jddj}||jj kD} | j |||||} t| |jjd} | j |||z||z} | ||k(z||k(z} || z} | jd\} } | jd\} }tj| |gj |dd}tj| | gd}tj|dkD|d}||fS) ai Based on a keypoint score matrix, compute the best keypoint matches between the first and second image. Since each image pair can have different number of matches, the matches are concatenated together for all pair in the batch and a batch_indices tensor is returned to specify which match belong to which element in the batch. Note: This step can be done as a postprocessing step, because does not involve any model weights/params. However, we keep it in the modeling code for consistency with other keypoint matching models AND for easier torch.compile/torch.export (all ops are in torch). Args: scores (`torch.Tensor` of shape `(batch_size, height_0, width_0, height_1, width_1)`): Scores of keypoints Returns: matched_indices (`torch.Tensor` of shape `(2, num_matches)`): Indices representing which pixel in the first image matches which pixel in the second image matching_scores (`torch.Tensor` of shape `(num_matches,)`): Scores of each match r2TrrFr1rTr) rrmaxvaluesr.coarse_matching_thresholdrrcoarse_matching_border_removalr%rrwhere)rBrrheight0width0height1width1max_0max_1mask masked_scoresmatching_scores_0 max_indices_0matching_scores_1 max_indices_1matching_indicesrs r+_get_matches_from_scoresz:EfficientLoFTRForKeypointMatching._get_matches_from_scores|s{*8>||4 GVWfZ6)97V;KL 1d +22 1d +22 ===||J&I4!K!KUS||J&(8'F:JKv'6U?; +8+<+squeezer)rBrrrrr num_keypointsrDr;rindices indices_0 indices_1rdelta_0delta_1fine_matches_0fine_matches_1 fine_matchess r+_get_first_stage_fine_matchingz@EfficientLoFTRForKeypointMatching._get_first_stage_fine_matchingsJ+:*?*?' M1a$%5s%:;)11*mRP))O<)$// ..   #("))!''  '1,-3||Aq"a(// M2rR,,tQ (;(;B(?(F(Fr2rST(UV^^_`a,,tQ (;(;B(?(F(Fr2rST(UV^^_`a1!Q$7'J:NN1!Q$7'J:NN++y)4!<{{NN#CK  $$r*rrc|j\}}}}t|dz} |dddf} |dddf} | | z} | | z} tj|| jj |ddd}tj|| jj d|dd}| d} | d} | d} t ddd| j jtj}|d }| |d z} | |d z} |j |||| d z| d z}|||| | | f}|j ||d}tjj||jjz d}|j ||dd}t|dd}|dddf}|dddf|dz|zz}tj||gd}|S)a For the given position in their respective fine windows, retrieve the 3x3 fine confidences around this position. After applying softmax to these confidences, compute the 2D spatial expected coordinates. Shift the first stage fine matching with these expected coordinates. Note: This step can be done as a postprocessing step, because does not involve any model weights/params. However, we keep it in the modeling code for consistency with other keypoint matching models AND for easier torch.compile/torch.export (all ops are in torch). Args: indices (`torch.Tensor` of shape `(batch_size, 2, num_keypoints)`): Indices representing the position of each keypoint in the fine window fine_matches (`torch.Tensor` of shape `(2, num_matches, 2)`): Coordinates of matched keypoints after the first fine stage fine_confidence (`torch.Tensor` of shape `(num_matches, fine_window_size, fine_window_size)`): Second stage confidence of matching fine features between the first and the second image fine_window_size (`int`): Size of the window used to refine matches fine_scale (`float`): Scale between the size of fine features and coarse features Returns: fine_matches (`torch.Tensor` of shape `(2, num_matches, 2)`): Coordinates of matched keypoints after the second fine stage rNrr)rCrrT)rrC)N.rrr2 r1rT)rrr%arangerCrrr_longrrrr.!fine_matching_regress_temperaturerr)rBrrrrrrrrDr;rr indices_1_i indices_1_j batch_indicesmatches_indicesdeltaheatmapfine_coordinates_normalizedrrs r+_get_second_stage_fine_matchingzAEfficientLoFTRForKeypointMatching._get_second_stage_fine_matching&sD+:*?*?' M1a$%5s%:;AqDM AqDM #33 "22  Z 8H8HIQQR\^_abdef ,,}Y=M=MNVVWXZgijlmni( !), !), 1T)JZJZ[^^_d_i_iji !E&M1 !E&M1 )11  '79IA9MO_bcOc *-)U`bm*mn)11*mQO--// dkkKK KQS0 "))*mQJ&;GT&J1&M#%ad+%ad+/Jf/UXb/bc{{NN#CK r*rMrNc|j\}}}}|jj} t|dz} t j ||| z d} t j ||| z d} | d}| d}||jddzz }||jddzz }||j ddz} tjj| dtjj| dz} | j|||| dz| dz} | dddddf} | j||||}|j||||\}}| d}| d}|| dzz }||j ddz}|j|||||}|S)a For each coarse pixel with a corresponding window of fine features, compute the matching confidence between fine features in the first image and the second image. Fine features are sliced in two part : - The first part used for the first stage are the first fine_hidden_size - config.fine_matching_slicedim (64 - 8 = 56 by default) features. - The second part used for the second stage are the last config.fine_matching_slicedim (8 by default) features. Each part is used to compute a fine confidence tensor of the following shape : (batch_size, (coarse_height * coarse_width), fine_window_size, fine_window_size) They correspond to the score between each fine pixel in the first image and each fine pixel in the second image. Args: fine_features_0 (`torch.Tensor` of shape `(num_matches, fine_kernel_size ** 2, fine_kernel_size ** 2)`): Fine features from the first image fine_features_1 (`torch.Tensor` of shape `(num_matches, (fine_kernel_size + 2) ** 2, (fine_kernel_size + 2) ** 2)`): Fine features from the second image coarse_matched_keypoints (`torch.Tensor` of shape `(2, num_matches, 2)`): Keypoint coordinates found in coarse matching for the first and second image fine_scale (`int`): Scale between the size of fine features and coarse features Returns: fine_coordinates (`torch.Tensor` of shape `(2, num_matches, 2)`): Matched keypoint between the first and the second image. All matched keypoints are concatenated in the second dimension. rr1rrrr2.) rr.fine_matching_slice_dimrr%splitrrrrrrr)rBrMrNrrrrrrJrr;split_fine_features_0split_fine_features_1rfirst_stage_fine_confidence fine_indicesrsecond_stage_fine_confidencefine_coordinatess r+_fine_matchingz0EfficientLoFTRForKeypointMatching._fine_matchingqsJGVF[F[C M#3^"&++"E"E$%5s%:;!& O^Ne=egi j % O^Ne=egi j02/2*O,A,A",E,LL)O,A,A",E,LL*O,E,Eb",MM--//CbmmF[F[\kmnFoo)11  '79IA9MO_bcOc *#qtQrT/:&5&=&=  '79I' #&*%H%H ' $   & " l02/2*,CS,HH(79R9RSUWY9Z'Z$??   (     r*rRrprc | td|j|fi|}|j}|d}|jdd\}}} |j\} } } } }| |z }|j ||\}}}|dd}|j ||\}}|j\} } }t j| d}|||dddff}|||dddff}t||z}| |z }|j||||}|dddddddf|z |dddddddf<|dddddddf| z |dddddddf<t||||j|jS)rrNz9SuperGlue is not trainable, no labels should be provided.rrtrr)rrrrr ) rvrQrurrrr%rrrrrr )rBrRrpr model_outputsrzrBr{rGrHrrDryr#r$rcoarse_keypointscoarse_matching_scorescoarse_matched_indicesrCrMrNrrrKrmatching_keypointss r+r`z)EfficientLoFTRForKeypointMatching.forwards<  XY Y)<(;(;L(SF(S  --#1+8G8M8Mbc8R5-1=1C1C. Ax - KOK`K` \L H02H %QRL+/+@+@Rc+d(5::1m Z0; )-9OPQSTPT9U*UV)-9OPQSTPT9U*UV <= k) !00/Sceop);Aq!QJ)G%)O1aA:&);Aq!QJ)G&)P1aA:&%*2('55$//   r*ra)r!r"r#r$rr6r%rcr(rr9rrrrrrrr&rrdrrrr`rerfs@r+rr_s 331u||31ellTYT`T`F`@a31j1;$||1;;@1; u||U\\5<<7 81;f@%@%#(,,@% @%  @% u||U\\) * @%DIIllI I  I  I IVZ Z Z #(,, Z  Z  Z x.2D ''D ))*D +, D D D r*r)rPrirr)r)FNN)T)@ dataclassesrtypingrrrr%r activationsr r modeling_layersr modeling_outputsr modeling_rope_utilsr modeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrconfiguration_efficientloftrrrrgr-rhr{rrrrrrcrrr9rrrrrr&r.r9rPriboolrrCrWrrr__all__r)r*r+rs1",, *9.6F&0> :[: :0$BII$P"))* :< &"))*#4RYY#4N<> UU\\ U# U%,, U(% II% <<% % << % U\\* %  %%'(%4E)biiE)P $0 0f1K8BII*2D0BIID0N*7O*7*7Z H57H5  H5V StUTWGWAX]b]i]i6$)%)#' 8 #u||# $8 ell" #8!8 U\\ " 8 EKK 8  \\ 8v-1-1t-1W\WcWc-1` o (Eo  o d hr*