rhddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl m cmZddl mZm Z ddlmZddlmZmZmZddlmZdd lmZmZdd lmZmZmZd d l m!Z!erdd l"m#Z#er ddl$m%Z%ddl&m'Z'eedGddeZ( dLde j de j de j fdZ)dededefdZ*de j de j de j fdZ+Gdde jXZ-dedede.defdZ/de j de j de.de j fd Z0Gd!d"e jXZ1Gd#d$e jXZ2Gd%d&e jXZ3 dMd'e jXd(e j d)e j d*e j d+ee j d,e4d-e4fd.Z5Gd/d0e jXZ6Gd1d2e jXZ7dNd3e j d4e4d5e8de j fd6Z9Gd7d8e jXZ:Gd9d:e jXZ;Gd;deZ=Gd?d@e j|Z?GdAdBe jXZ@GdCdDe jXZAGdEdFe jXZBeGdGdHeZCedIGdJdKeCZDdHdKgZEy)ON) dataclass)CallableOptionalUnion)Tensornn)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)auto_docstringcan_return_tupleis_accelerate_available) EomtConfig)linear_sum_assignment) PartialState)reducea Class for outputs of [`EomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~EomtImageProcessor.post_process_semantic_segmentation`] or [`~EomtImageProcessor.post_process_instance_segmentation`] or [`~EomtImageProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~EomtImageProcessor] for details regarding usage. ) custom_introc<eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dZeeej"ed <y) "EomtForUniversalSegmentationOutputa+ loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segementation. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_state hidden_states attentions patch_offsets)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrr tupler!r"listry/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/eomt/modeling_eomt.pyrr0s*)-D(5$$ %,8<(5#4#45<8<(5#4#45<59x 1 1298r? numerator denominatorrs r.pair_wise_dice_lossrI{s|^^  % %a (FELL22I**R.D)FJJrN47,CCK A +/2 2D Kr-c\|jd}tjd}||tj|}||tj |}tj ||z |j}tj ||z d|z j}||z}|S)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnone reduction)shaperBCEWithLogitsLossr' ones_like zeros_likerDrE) r>r?height_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negrs r.$pair_wise_sigmoid_cross_entropy_lossrXs||A$$v6I&vuv/FG&vu/?/?/GH||25EEvxxPH||25EEF ~~VH h D Kr-c eZdZdZ d dedededeffd Zejdejdejd ejd ejd e e e f d Z xZ S)EomtHungarianMatcheraqThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). cost_class cost_mask cost_dice num_pointsct||dk(r|dk(r|dk(r td||_||_||_||_y)aHCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)super__init__ ValueErrorr^r[r\r])selfr[r\r]r^ __class__s r.razEomtHungarianMatcher.__init__sK"  ?yA~)q.34 4$$""r-rr mask_labels class_labelsr1c g}|jd}t|D]}||jd}||} |dd||f } ||j| } | dddf} | dddf} t j d|j d| j} | j| jddd} t| | djd} | j| jddd}t| |djd} t| | }t| | }|j|z|j| zz|j|zz}t j |t j"d }t j$|t j"d }t j&|d}t)|j+}|j-||Dcgc]O\}}t j.|tj0 t j.|tj0 fQ}}}|Scc}}w) ao Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rrANrr3deviceF align_cornersg _Bg _©dtype)rNrangesoftmaxtor'randr^rirepeatr=r9rXrIr\r[r]minimumtensormaximum nan_to_numrcpuappend as_tensorint64)rcrrrerfindices batch_sizei pred_probs pred_maskr[ target_maskr0target_coordinatespred_coordinatesr\r] cost_matrixassigned_indicesjmatched_indicess r.forwardzEomtHungarianMatcher.forwards/8*,*//2 z" -A-a088r?rprobsrGrHrs r. dice_lossrsz, NN  $ $Q 'EUV^((,,I))B-&**R.0K A +/2 2D 88: !D Kr-ctjd}|||}|jdj|z }|S)a| Args: inputs (`torch.Tensor`): A float tensor of arbitrary shape. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss. rKrLr)rrOmeanrF)r>r?rrScross_entropy_lossrs r.sigmoid_cross_entropy_lossr5sD$$v6I"662  " "1 % ) ) +i 7D Kr-ceZdZdedeeefffd Zdeee dee fdZ dee de e e ffdZ d e d ee d e ejdeee ffd Zd ejdeejd e ejde deeejff dZdZdZdejdejfdZdejde de dedejf dZ dd ejd ejdeejd eejdeeeejfdeeejff dZd ejdej2dejfdZxZS)EomtLossconfig weight_dictct|t|dg|j|_||_|j |_tj|jdz}|j |d<|jd||j|_ |j|_ |j|_ t|j|j |j"|j|_y)aH The Eomt Loss. The loss is computed very similar to DETR. The process happens in two steps: 1) we compute hungarian assignment between ground truth masks and the outputs of the model 2) we supervise each pair of matched ground-truth / prediction (supervise class and mask) Args: config (`EomtConfig`): The configuration for Eomt model also containing loss calculation specific parameters. weight_dict (`dict[str, float]`): A dictionary of weights to be applied to the different losses. scipyrrA empty_weight)r[r]r\r^N)r`rar num_labelsrno_object_weighteos_coefr'onesregister_buffertrain_num_pointsr^oversample_ratioimportance_sample_ratiorZ class_weight dice_weight mask_weightmatcher)rcrrrrds r.razEomtLoss.__init__Js $ * ++&// zz$//A"56 == R ^\:!11 & 7 7'-'E'E$+**((((   r-sizesr1cn|d}|ddD]'}t|D]\}}t|||||<)|S)Nrr) enumeratemax)rcrmaxessublistindexitems r. _max_by_axiszEomtLoss._max_by_axismsSaQRy 7G(1 7 t"5<6e  7 7 r-tensorsc`|j|Dcgc]}t|jc}}t|g|z}|\}}}}|dj} |dj } t j|| | } t j|||ft j| } t|| | D]o\}} }| d|jdd|jdd|jdfj|d|d|jdd|jdf<q| | fScc}w)Nrrmrirr3F) rr+rNlenrmrir'zerosrboolzipcopy_)rcrrtmax_size batch_shaper|_heightwidthrmripadded_tensors padding_masks padded_tensor padding_masks r._pad_images_to_max_in_batchz$EomtLoss._pad_images_to_max_in_batchus7$$w%OVd6<<&8%OP7|nx/ '2$ Avu   ""[fM J#>ejjY_` 36wP]3^ G /FM< +FLLO+->v||A->@Q&,,q/@QQ R X XY_ `AFL*6<<?*,=fll1o,== > G},,&PsD+rrfr{c |}|j\}}}tj|j}|j |} t j t||D  cgc] \} \}} | | c} }} } t j||f|jt j|j} | | | <|jdd}||| }d|i}|Scc} }} w)aCompute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )weight) fill_valuermrirr3loss_cross_entropy) rNrCrossEntropyLossr$_get_predictions_permutation_indicesr'catrfullrrzri transpose)rcrrfr{ pred_logitsr| num_queriesrrSidxtargetrtarget_classes_otarget_classespred_logits_transposedloss_celossess r. loss_labelszEomtLoss.loss_labelss"+ %0%6%6" K''t/@/@A 77@ 99-0w-G H H>66AqVAY H   %$//]h]o]o /s!,!6!6q!!<2NC&0  Is#C! rrercj|}j|}||}j|\}} ||}|dddf}|dddf}tj5j |fdj jj} t|| djd} dddt| djd} t|  |t| | |d} ~~| S#1swYExYw)aCompute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Nc&j|SN)calculate_uncertainty)logitsrcs r.z%EomtLoss.loss_masks..st99&Ar-Frjr) loss_mask loss_dice) r _get_targets_permutation_indicesrr'rsample_points_using_uncertaintyr^rrr=r9rr)rcrrer{rsrc_idxtgt_idx pred_masks target_masksrr0 point_labels point_logitsrs` r. loss_maskszEomtLoss.loss_maskss,4;;GD77@)'2 ::;G a#G,  4( #AtG, ]]_ i $ D DA%%,, ! ( 6GW\]eefghL i$J0AQVW__`ab 4L,PYZ"<yI    ) i is (ADD c tjt|Dcgc]\}\}}tj|| c}}}}tj|Dcgc]\}}| c}}}||fScc}}}wcc}}wrr'rr full_like)rcr{r}srcr batch_indicespredictions_indicess r.rz-EomtLoss._get_predictions_permutation_indicessj iX_N`"a"a{q(35??3#:"ab #iiW(E#q(EF111#b(E #A7  A> c tjt|Dcgc]\}\}}tj|| c}}}}tj|Dcgc]\}}| c}}}||fScc}}}wcc}}wrr)rcr{r}rtgtrtarget_indicess r.rz)EomtLoss._get_targets_permutation_indicessh iX_N`"a"a{q(1c5??3#:"ab #@HQC#@An,,#b#@rrc2tj| }|S)a In Eomt paper, uncertainty is estimated as L1 distance between 0.0 and the logit prediction in 'logits' for the foreground class in `classes`. Args: logits (`torch.Tensor`): A tensor of shape (R, 1, ...) for class-specific or class-agnostic, where R is the total number of predicted masks in all images and C is: the number of foreground classes. The values are logits. Returns: scores (`torch.Tensor`): A tensor of shape (R, 1, ...) that contains uncertainty scores with the most uncertain locations having the highest uncertainty score. )r'abs)rcruncertainty_scoress r.rzEomtLoss.calculate_uncertaintys %yy01!!r-r^rrc |jd}t||z}tj||d|j}t ||d} || } t||z} || z } tj | dddddf| dd} |tj|tj|j z}| |dddfz } |jd d| jd ddfj|| d}| dkDr:tj|tj|| d|jgd }|S) a This function is meant for sampling points in [0, 1] * [0, 1] coordinate space based on their uncertainty. The uncertainty is calculated for each point using the passed `uncertainty function` that takes points logit prediction as input. Args: logits (`float`): Logit predictions for P points. uncertainty_function: A function that takes logit predictions for P points and returns their uncertainties. num_points (`int`): The number of points P to sample. oversample_ratio (`int`): Oversampling parameter. importance_sample_ratio (`float`): Ratio of points that are sampled via importance sampling. Returns: point_coordinates (`torch.Tensor`): Coordinates for P sampled points. rr3rhFrjNr)kr5rrAr5) rNrr'rqrir=topkarangelongviewr)rcruncertainty_functionr^rr num_boxesnum_points_sampledr0rpoint_uncertaintiesnum_uncertain_pointsnum_random_pointsrshifts r.rz(EomtLoss.sample_points_using_uncertaintysI<LLO  .>!>?"JJy2DaPVP]P]^#F,=US 2<@"#:Z#GH&)==jj,Q1W59MSTUVWX"U\\)5::V\VcVc%dd uQW~-222q9#((2,/JOOPY[oqrs q % "EJJy:KQW]WdWd$ef! ! r-auxiliary_predictionsc|j||||}|j||dj}i|j|||||j |||}|jt |D]\\} } | d}| d}|j ||||} | jD cic] \} } | d| | } } } |j| ^|Scc} } w)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], then it contains the logits from the inner layers of the EomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrhrrr) r get_num_masksrirrrritemsupdate)rcrrrerfrr{rrr aux_outputs loss_dictkeyvalues r.rzEomtLoss.forward:s H,,35I;Xde&&|LO CsB#r)r#r$r%rdictstrrrar+rrrr*rnparrayrr'rrrrrrrrirrrs@r.rrIs! z! S%Z8H! F$tCy/d3i-4<-E&RX.DY-" $* :>v, QVWYW_W_Q`  c6k  D<#ll<%,,'<rxx <  < c5<< <|2 - "ELL"U\\""5! 5! 5!  5! "' 5! 5!zDH 5#ll5$ll5%,,' 5 5<<( 5 (S%,,->(?@ 5 c5<< 5n %,,  QVQ]Q] r-rcZeZdZdZfdZdej dej fdZxZS)EomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr kernel_sizestride)r`ra image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)rcrrrrrr$rds r.razEomtPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir- pixel_valuesr1c|jd}||jk7rtd|jd|d|j|j dj dd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r3)rNrrbr&rCr)rcr'r embeddingss r.rzEomtPatchEmbeddings.forwardsz#))!, 4,, ,!../yaI __\2::1=GG1M r-) r#r$r%r&rar'rrrrs@r.rrs) jELLU\\r-rcdeZdZdZdeddffd ZdejdejfdZxZ S)EomtEmbeddingszM Construct the CLS token, mask token, position and patch embeddings. rr1Nct|||_|j|_t j t jdd|j|_ t j t jd|j|j|_ t||_|jj}t j |j"|_d|jz|_t j(||j|_|j-dt j.|j1ddy)Nr position_ids)rrAF) persistent)r`rarrr Parameterr'randnr cls_tokenrnum_register_tokensregister_tokensrpatch_embeddingsr$Dropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsrrexpand)rcrr$rds r.razEomtEmbeddings.__init__s   ++ekk!Q8J8J&KL!||EKK6;U;UW]WiWi,jk 3F ;++77 zz&"<"<= !"V%?%?!?#%<< V=O=O#P  ^U\\+-F-M-Mg-Vchir-r'c|j\}}}}|jjjj}|j|j |}|j j|dd}|jj|dd}||j|jz}tj|||gd}|j|}|S)NrlrArr)rNr5r&rrmrpr2r<r4r;r.r'rr8)rcr'r|r target_dtyper* cls_tokensr4s r.rzEomtEmbeddings.forwards*00 Aq!,,77>>DD ** ..55j"bI$":":4;L;L"MM YY OZHaP \\*- r-) r#r$r%r&rrar'rrrrs@r.r,r,s9jzjdj ELL U\\ r-r,modulequeryr r attention_maskscalingr8c tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)NrA)r5rm)ptrainingrr3) r'rDrrr7rofloat32rprmr8rG contiguous) r@rAr r rBrCr8r; attn_weights attn_outputs r.eager_attention_forwardrLs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r-c eZdZdZfdZ ddej deej deej eej ffdZ xZ S) EomtAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)r`rarr embed_dimnum_attention_heads num_headshead_dimrbscaleattention_dropoutr8 is_causalrLineark_projv_projq_projout_projrcrrds r.razEomtAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r-r rBr1c :|j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rt|jj} | |||| ||j|j|jsdn |j\} } | j!|||j#} |j%| } | | fS)z#Input shape: Batch x Time x Channelrr3eager)rVrCr8)rNrZrXrYrrRrSrrLr_attn_implementationrrVrTrGr8reshaperIr[) rcr rBr;r| seq_lengthrPquerieskeysvaluesattention_interfacerKrJs r.rzEomtAttention.forwardsa-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0 L((r-r) r#r$r%r&rar'rrr*rrrs@r.rNrNsVGB.26$)||$)!.$) u||Xell33 4 $)r-rNcXeZdZdfd Zdej dej fdZxZS)EomtLayerScaler1ct|tj|jt j |jz|_yr) r`rarr0layerscale_valuer'rrlambda1r\s r.razEomtLayerScale.__init__s8 ||F$;$;ejjI[I[>\$\] r- hidden_statec ||jzSr)rkrcrls r.rzEomtLayerScale.forward!sdll**r-r1Nr#r$r%rar'rrrrs@r.rhrhs$^+ELL+U\\+r-rhinput drop_probrGc|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r_rr)rr)rNndimr'rqrmrifloor_div)rqrrrG keep_probrN random_tensoroutputs r. drop_pathrz%sCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr-cxeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) EomtDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrrr1c0t|||_yr)r`rarr)rcrrrds r.razEomtDropPath.__init__<s "r-r cDt||j|jSr)rzrrrGrcr s r.rzEomtDropPath.forward@s FFr-c d|jS)Nzp=)rrrcs r. extra_reprzEomtDropPath.extra_reprCsDNN#$$r-r) r#r$r%r&rrrar'rrrrrrs@r.r|r|9sGb#(5/#T#GU\\GellG%C%r-r|cXeZdZdfd Zdej dej fdZxZS)EomtMLPr1c~t||jx}}t|j|jz}t j ||d|_t|jtrt|j|_ n|j|_ t j ||d|_ y)NTbias)r`rarr mlp_ratiorrWfc1r  hidden_actrr activationfc2rcr in_features out_featureshidden_featuresrds r.razEomtMLP.__init__Hs %+%7%77 lf0063C3CCD99[/E f'' -$V%6%67DO$//DO99_lFr-rlcl|j|}|j|}|j|}|Sr)rrrrns r.rzEomtMLP.forwardSs2xx - |4 xx - r-rorprs@r.rrGs$ GELLU\\r-rcXeZdZdfd Zdej dej fdZxZS) EomtSwiGLUFFNr1c0t||jx}}t|j|jz}t|dzdz dzdzdz}t j |d|zd|_t j ||d|_y)Nr3r Tr) r`rarrrrrW weights_in weights_outrs r.razEomtSwiGLUFFN.__init__[s %+%7%77 lf0063C3CCD2Q67!;AAE))K_1D4P99_lNr-rlc|j|}|jdd\}}tjj ||z}|j |S)Nr3rAr)rchunkrr7silur)rcrlx1x2hiddens r.rzEomtSwiGLUFFN.forwarddsS|4 ##A2#.B##B'",''r-rorprs@r.rrZs$O(ELL(U\\(r-rc eZdZdZdeddffd Z d dejdeejde de e ejejfe ejffd Z xZ S) EomtLayerzCThis corresponds to the Block class in the original implementation.rr1Nct|tj|j|j |_t||_t||_ |jdkDrt|jntj|_tj|j|j |_|j rt#||_nt'||_t||_y)Nepsr_)r`rar LayerNormrlayer_norm_epsnorm1rN attentionrh layer_scale1drop_path_rater|Identityrznorm2use_swiglu_ffnrmlpr layer_scale2r\s r.razEomtLayer.__init__ns \\&"4"4&:O:OP &v.*62@F@U@UX[@[f&;&;(,'78=H zz-0 xx - ((6 ~~l3mC /G+r-)NF)r#r$r%r&rrar'rrrrr*rrrs@r.rrks~M3z3d3&-1"' ||ELL)  uU\\5<</0% 2EE F r-rcXeZdZdfd Zdej dej fdZxZS)EomtLayerNorm2dc*t||||y)N)relementwise_affine)r`ra)rcrraffinerds r.razEomtLayerNorm2d.__init__s 36Jr-rlr1c|jdddd}tj||j|j|j |j }|jdddd}|S)Nrr3r r)permuteF layer_normnormalized_shaperrrrns r.rzEomtLayerNorm2d.forwardsb#++Aq!Q7 ||L$2G2GVZV_V_aeaiaij #++Aq!Q7 r-)gư>Trprs@r.rrs$KELLU\\r-rc\eZdZdeffd Zdej dejfdZxZ S)EomtScaleLayerrct||j}tj||dd|_t |j|_tj||dd|d|_ t||_ y)Nr3rr rF)rpaddinggroupsr) r`rarrConvTranspose2dconv1r rrr%conv2r layernorm2drcrrrds r.razEomtScaleLayer.__init__su (( '' [aXYZ  !2!23YY     +;7r-r r1c|j|}|j|}|j|}|j|}|Sr)rrrrrs r.rzEomtScaleLayer.forwardsB =1  6  =1 ((7 r-) r#r$r%rrar'rtrrrrs@r.rrs*8z8 U\\ellr-rc\eZdZdeffd Zdej dej fdZxZS)EomtScaleBlockrct||j|_t j t |jDcgc] }t|c}|_ycc}wr) r`ranum_upscale_blocks num_blocksr ModuleListrnrblockrcrrrds r.razEomtScaleBlock.__init__sG  33]]E$//DZ#[qN6$:#[\ #[sA&r r1c8|jD] }||} |Sr)r)rcr rs r.rzEomtScaleBlock.forwards%ZZ 1E!-0M 1r- r#r$r%rrar'rrrrs@r.rrs,]z] U\\ellr-rc\eZdZdeffd Zdej dej fdZxZS) EomtMaskHeadrc t||j}tj|||_tj|||_tj|||_t|j|_ yr) r`rarrrWrrfc3r rrrs r.razEomtMaskHead.__init__sa (( 99[+699[+699[+6 !2!23r-r r1c|j|j|}|j|j|}|j|}|Sr)rrrrrs r.rzEomtMaskHead.forwardsD(?@ (?@ / r-rrs@r.rrs*4z4U\\ellr-rcZeZdZUdZeed<dZdZdZdgZ dZ dZ de jd d fd Zy ) EomtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. reomtr'FrTr@r1Nc|jj}t|tjtj tj frtjj|jtjd|jxtjj|j\}}|dkDrdtj|z nd}tjj|j| |yyt|tjrJ|jj j#d|jj j%yt|tj&rf|jj j)dd|j*2|jj |j*j%yyt|t,rGt/|dr:|j0j j#|jj2yyt|t4rtjj7|j8j j;t<j>d|j;|j8j@|j8_|jBj j%yy) N)arrr4r_)rstdrk)"rinitializer_ranger rrWr%rinitkaiming_uniform_rmathsqrtr_calculate_fan_in_and_fan_outuniform_rdatafill_zero_r:normal_ padding_idxrhhasattrrkrjr, trunc_normal_r2rpr'rHrmr4)rcr@rfan_inrbounds r. _init_weightsz!EomtPreTrainedModel._init_weightsskk++ fryy"))R5G5GH I GG $ $V]]diil $ C{{&GGAA&--P 17!DIIf--  ufe<' - MM   $ $S ) KK   " " $  - MM   & &CQ & 7!!- ""6#5#56<<>.  /vy)##))$++*F*FG*  /$&GG$9$9  %%((7cs%:%b!!''(    !  " " ' ' - - / 0r-)r#r$r%r&rr)base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attnrModulerr,r-r.rrsJ $O&+#$ N0BII0$0r-rzV The EoMT Model with head on top for instance/semantic/panoptic segmentation. c$eZdZdZdeddffd Zdededed ed eeefdeeeff d Z d eeefdefd Z e e ddede eed e eede ede ede eedefdZdZdej fdZedZxZS)EomtForUniversalSegmentationr'rr1Nct||||_|j|_t ||_t j|j|j|_ t j|j|j|_ t jt|jDcgc] }t!|c}|_t%||_t)||_t j,|j|j.dz|_|j2|j4z|j2|j4zf|_|j8|j:|j<d|_tA||j>|_!|jEdtGjH|jJ|jMycc}w)Nrr)rrr)rrattn_mask_probs)'r`rarnum_hidden_layersr,r*rrrr layernormr:rrArrnrlayersr upscale_blockr mask_headrWrclass_predictorrr grid_sizerrrrrrSrr'rr post_initrs r.raz%EomtForUniversalSegmentation.__init__sp   !'!9!9(0f&8&8f>S>ST\\&"4"4f6H6HI mmfF^F^@_$`1Yv%6$`a +F3%f-!yy););V=N=NQR=RS ++v/@/@@&BSBSW]WhWhBhi"("5"5++++.  "T=M=MN . 6;L;L0MN %%as>G*rrrerfrc|j|||||}|jjD]'\}}|jD]\} } || vs | |z} )|S)Nrrrerfr)rSrr) rcrrrerfrrr rloss_keyrs r. get_loss_dictz*EomtForUniversalSegmentation.get_loss_dict+s|(,~~!5!5#%"7 (6(  ++113 #KC"+//"3 #$(?FND # # r-rc4t|jSr)rFre)rcrs r.get_lossz%EomtForUniversalSegmentation.get_lossCs9##%&&r-output_hidden_statesrr"c ||n|jj}||n|jj}|rdnd}|rdnd}d\} } d} | td|j |} t |j D]\} }|r|| fz }| |j|jjz k(rp|jjdddddfj| jdddj| j}tj || fd} | |j|jjz k\r|j"s7|j$| |jz |jjzdkDr|j'| }|j)|\}}| |fz } | |fz } tj*| jd| jd| jd| jtj, } t/j0||j2d }|j5|j7d|j7dd}|jj8}||jj:z}|dkD| ddd||df<|j=| |j$| |jz |jjz||| j } | dddd fjd|jj>dd} | jAjC| d} || | |}|d} |s||dfz }|j'| }|r||fz }|j)|\}}| |fz } | |fz } d}|B|@d}tE| | D]/\}}|jG||||d}||jI|z }1tK|||||||S)ah mask_labels (`list[torch.Tensor]`, *optional*): list of mask labels of shape `(num_labels, height, width)` to be fed to a model class_labels (`list[torch.LongTensor]`, *optional*): list of target class labels of shape `(num_labels, height, width)` to be fed to a model. They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segementation. Nr,)r,r,z You have to specify pixel_valuesrrArr)rirmbilinear)sizemode)probnum_query_tokensencoder_start_tokensri.ger_r)rrrrr r!r")&rr rrbr*rrrrrArr<rNrprir'rrGrrpredictrrr interpolaterrrrr9_disable_attention_maskrQr masked_fillrrr r)rcr'rerfr rr"all_hidden_statesall_attentionsmasks_queries_logits_per_layerclass_queries_logits_per_layerrBr r layer_modulerAnorm_hidden_statesrrinterpolated_logitsrr layer_outputssequence_outputrrs r.rz$EomtForUniversalSegmentation.forwardFsV*%9$D $++JjJj 2C1N-TXT_T_TqTq"6BD0dIOF&(F  ?@ @ 5 !*4;;!75 6 C#!m%55!d,,t{{/E/EEE ))$1*5<<]=P=PQR=SUWY[\__`m`t`tu % 5-*@a H d,,t{{/E/EEE !5!5cD&>',,Q/1D1I1I!1Lb'#$(;;#:#: '7$//:[:['[$ObdeNeq"3#3"35I5JJK"&!=!="--cD4J4J.JT[[McMc.cd%5)=)00 ">""04 !=!D!DRIhIhjlnp!q!/!5!5!7!C!C^OUY!Z(HYZM)!,M =#3"55k5 6n..7  /!3 3 59\\/5R22&+?*AA&&+?*AA&  "|'?D>A.0N? 1:$&:!..)=)= +!-*. /  i00 12!5!5-+%'  r-c.|jjSr)r*r5rs r.get_input_embeddingsz1EomtForUniversalSegmentation.get_input_embeddingss///r-rc|ddd|jjddf}|j|}|dd|jj|jjzdddf}|j dd}|j |jddg|j}|j|}|j|}tjd||}||fS)Nrr3rrAzbqc, bchw -> bqhw) rrrr*r9rrarNrrrr'einsum)rcr query_tokens class_logits prefix_tokens mask_logitss r.rz$EomtForUniversalSegmentation.predictsa!:4;;#:#:!:A=> ++L9 q$++"9"9DOO<]<]"]"_abbc %//15 - --m.A.A!.DbZ4>>Z ~~l3 **=9 ll#6 mT L((r-c|dkr9tj|jd|||kD}d|ddd||df|<|S)Nrrrh)r'rqrN) attn_maskrrrrirandom_queriess r.rz4EomtForUniversalSegmentation._disable_attention_masksW !8"ZZ (: Rr-)NNNNN)r#r$r%rrrarrrrr rrrr+rrrr!r'r staticmethodrrrs@r.rrsQ %Ozd8$%    $CK0  c6k 0'$sF{"3''/3/3/3,004x x d6l+x tF|, x 'tn x $D> x  V -x  ,x x t0)ell) r-r)F)r_)r_F)Fcollections.abcr!r dataclassesrtypingrrrnumpyrr'torch.nn.functionalrr7rr activationsr file_utilsr r r modeling_layersrmodeling_utilsrrutilsrrrconfiguration_eomtrscipy.optimizer accelerateraccelerate.utilsrrr=rIrXrrZrrrrrr,rrLrNrhrrzr|rrrrrrrrrr__all__r,r-r.r;s, !,, !LL9FNN*4''  77  7BLQLL5:\\ \\@6,u||X]XdXd8g299gTff<u||U\\VY^c^j^j(uryyup "))B"RYY"X% II% <<% % << % U\\* %  %%.;)BII;)|+RYY+U\\eTV[VbVb( %299 %bii&(BII("0*0fbllRYY2 RYY 299"$0/$0$0N R#6R  Rj !"@ Ar-