rh dZddlZddlmZddlmZmZddlZddl Zddlm Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZmZmZd d lmZej6eZeedGddeZeedGddeZGdde j@Z!Gdde j@Z"Gdde j@Z#Gdde j@Z$d>dejJde&de'd ejJfd!Z(Gd"d#e j@Z)Gd$d%eZ*Gd&d'e j@Z+Gd(d)e j@Z,Gd*d+e j@Z-Gd,d-e j@Z.eGd.d/eZ/eGd0d1e/Z0d2ejJd3e1d ejJfd4Z2d2ejJd5e1d6e1d ejJfd7Z3Gd8d9e j@Z4ed:Gd;d2D2D/ L&% 4,, ,w  T__Q' '5DOOA4F+F$VHAeW4KDOO\]L^K__`aeapapqras`ttvw __\2::1aAF r")rrrrr.rI __classcell__r<s@r#r)r)[s j r"r)c eZdZdZdeddffd ZdededejfdZ dd ejd ejd e ejd e e dejf d Z xZS)SegGptEmbeddingszX Construct the embeddings from patch, position embeddings for input and prompt. r;returnNc~t|tjt j ddd|j |_tjt j ddd|j |_tjt j ddd|j |_ tjt j ddd|j |_ tjt j ddd|j |_ t||_ |j|jzdzdz}tjt j d||j |_tj$|j&|_y)Nrr@)r-r.r Parameterrzerosr2 mask_tokensegment_token_inputsegment_token_prompttype_token_semantictype_token_instancer)patch_embeddingspretrain_image_sizer0randnposition_embeddingsDropouthidden_dropout_probdropout)r:r; num_positionsr<s r#r.zSegGptEmbeddings.__init__s3 ,,u{{1aF@SUWX``abdeghjkle_# O#**1aA6 6"**1feR@ @r"rDprompt_pixel_valuesbool_masked_posembedding_typecR|j|}|j|}|j\}}} } |jj||| d} |j dj | j d|| d} |d| z z| | zz}||nd}|j|| } ||jz}||jz}|| z}|| z}|dk(r |j}n |dk(r |j}ntd|||z}||z}tj||fd}|S)NrarinstancesemanticzBEmbedding type should be either 'semantic' or 'instance', but got rdim)rWrArRexpand unsqueezetype_asrjrmrSrTrUrVrBrcat)r:rDrnrorpinput_embeddingsprompt_embeddingsrE patch_height patch_width_rRw pos_embedtype_embeddingrHs r#rIzSegGptEmbeddings.forwardsh 00> 112EF3C3I3I0 L+q__++J kSUV  % %b ) 1 1* = E Eb,Xcef g-Q7*q.H+9+E:11, L ,d.F.FF-0I0II,i7- 9 Z '!55N z )!55Nabpaqrs s+n<->YY 02CD!L r")NN)rrrrrr.intrTensorrmr BoolTensorstrrIrJrKs@r#rMrM~s>|>> AsA3A5<<A,7;(, +ll+#\\+"%"2"23 + ! +  +r"rMc8eZdZdZfdZdededejdejfdZdejd ejd ejd ejde eefde eefdejfd Z dd ejdejfdZ xZ S)SegGptAttentionz=Multi-head Attention block with relative position embeddings.ct||j|j}}t |t j jr|n||f}t |t j jr|n||f}|d|jz|d|jzf}|j|jz}|j|_ |dz|_ tj|j|jdz|j|_tj|j|j|_|j |_|j r|| t#dtj$t'j(d|dzdz ||_tj$t'j(d|dzdz ||_yy)NrrgrbiaszBInput size must be provided if using relative positional encoding.r@)r-r.r/r0r3r4r5r6r2num_attention_headsscalerLinearqkv_biasqkvproj use_relative_position_embeddingsrBrPrrQ rel_pos_h rel_pos_w)r:r;r/r0 input_sizehead_dimr<s r#r.zSegGptAttention.__init__s !'!2!2F4E4EJ #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq  mv'8'88*Q-6K\K\:\] %%)C)CC#)#=#= t^ 99V//1C1Ca1Gfoo^IIf00&2D2DE 060W0W-  0 0! !eff \\%++a*Q-6G!6KX*VWDN\\%++a*Q-6G!6KX*VWDN 1r"q_sizek_sizerel_posrNctdt||zdz }tj|j d|j ddj ddd|d}|j d|j dd}tj|dddft||z dz}tj|dddft||z dz}||z |dz t||z dzz}||jS) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (int): size of the query. k_size (int): size of key k. rel_pos (`torch.Tensor`): relative position embeddings (L, channel). Returns: Extracted positional embeddings according to relative positions. r@rrralinear)rcrdN?) rmaxrhrirjrArCrarangelong) r:rrr max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordss r# get_rel_poszSegGptAttention.get_rel_poss 1s6622Q67 -- OOAw}}Q/ 4 < bhwkzbhwc,wkc->bhwkN)rrArjreinsum)r:rrrrrr query_height query_width key_height key_widthrelative_position_heightrelative_position_widthrEr~rureshaped_queryrel_hrel_ws r#add_decomposed_rel_posz&SegGptAttention.add_decomposed_rel_poss>%+! k & I#'#3#3L*i#X "&"2"2; 9"U"[[ Asz<cR -~?WX -~?VW||J k:yYeAq!Q,--aAtQ6F0GG||J {(BJQZDZ[ r"rc |j\}}}}|j|j|||zd|jdj ddddd}|jd||jz||zdj d\}} } ||j z| jddz} |jr.|j| ||j|j||f||f} tjjj| tj dj#|j$} |rE| j'||j||zd} | j'||jz||zd} nd} | | zj||j||d} | j dddddj|||d} |j)| } | | fS) Nrrar@rr)dtyperu)rArrjrrCunbindr transposerrrrrrrsoftmaxfloat32torviewr)r:routput_attentionsrErFrGr~rrkeyvalue attn_weightsattn_weights_reshaped attn_outputs r#rIzSegGptAttention.forward;s'4':':$ FE1 HH] # WZ%D4L4Lb Q WQ1a #  KK:8P8P+PRX[`R`bdellmnosE *cmmB.CC  0 066eT^^T^^fe_W]_dVeLxx**22,>FDYDYZr"r ensemble_condfeature_ensemblerc|j|j||}|d}|dd}|r|jddz|k\r|j|jddzd\}} |dk(ra|jddz} | j d| d} | j ddj | } | j|j} n"| j ddj | } tj|| gd}|j||z}|} |j|}|j|}| |j|z}|f|z}|S) N)rrrr@rtraT)rukeepdim) rrrAsplitrjmean expand_asrryrrr) r:rrrrself_attention_outputsattention_outputoutputspromptinputs num_promptsresiduals r#rIzSegGptLayer.forwardsz"&  ! !- 0/"0" 2!4(,  0 6 6q 9Q >- O-334D4J4J14MQR4RXY3ZNFF!.44Q71< ;;D9CCFK'6D9CCFK$yy&&)9qA '78=H  ,,]; /  4>>-#@@  "W,r")FF)rrrrrr.rrrboolrr rIrJrKs@r#rrs[|[U[t["'"' #||## #  # uU\\5<</0% 2EE F #r"rcpeZdZdeddffd Z d dej dededed edee e ff d Z xZ S) SegGptEncoderr;rNNc t|||_tjd|j |j dDcgc]}|j}}tjt|j Dcgc]}t|||c}|_ tj|j|j|_d|_ycc}wcc}w)Nrcpu)rrF)r-r.r;rlinspacernum_hidden_layersitemr ModuleListrangerlayersrr2r layernormgradient_checkpointing)r:r;xdprir<s r#r.zSegGptEncoder.__init__s  !&63H3H&JbJbkp!qrAqvvxrrmm%PVPhPhJi$jQ[Q%@$jk f&8&8f>S>ST&+#s$js CC$rrroutput_hidden_states return_dictc6|rdnd}|rdnd}g}t|jD]\} } |r||fz}|jj| kDrdnd} | || ||} | d}| |jjk(r.|d|jddz||jddzdzdz}| |jj vr |j |j||s|| dfz}|r||fz}|std||||fDSt||||S)Nr!r@rrr`c3$K|]}|| ywrr!).0vs r# z(SegGptEncoder.forward..s=s)rrrr) enumerater r; merge_indexrA!intermediate_hidden_state_indicesappendr r r) r:rrrrrall_hidden_statesall_self_attentionsrr layer_moduler layer_outputss r#rIzSegGptEncoder.forwardsr#7BD$5b4%'"(5 POA|#$58H$H!"&!8!81!#?@ LL-=!>? &   #F F%(A$BRBRAS&TU U!1 3r"rrNcd|jdk(rWtjjj ||j |j |j|j}|S|jdk(r|j}|j}|jdd}||z jdjdd}||z tj||jzz }|j|}|j ddddf|z|jddddfz}|S)Nr#r$rT)rr@)r)r'rrr layer_normr)r&rrrrrpowsqrtr)r:r input_dtypeuss r#rIzSegGptLayerNorm.forwards     .##..q$2G2GVZV_V_aeaiaijA  !1 1''K Aq$'AQ A##At#4AQ%**Q\22A;'A AtTM*Q.1dD=1IIAr")gư>r#) rrrrr.rrrIrJrKs@r#r!r!s( 4  %,, r"r!c>eZdZfdZdej fdZxZS)SegGptDecoderHeadcTt|tj|j|jdd|_t |j|jd|_t|j|_ tj|jddd|_ y)Nrr)r+paddingr$)r)rr'T)r+r) r-r.rr8decoder_hidden_sizeconvr!rr r ract_fctheadrs r#r.zSegGptDecoderHead.__init__s II  & &  & &   )#77V=R=R`p f//0 IIf88!QUV r"rc|j|}|j|}|j|}|j|}|Sr)r6r r7r8rs r#rIzSegGptDecoderHead.forward-s@ -0 }5  ]3  -0 r")rrrr.rrrIrJrKs@r#r2r2s WU%6%6r"r2cveZdZfdZdej dej fdZdej fdZxZS) SegGptDecodercBt|tj|jt |j z|jdz|jzd|_ t||_ |j|_|j|_||_ y)Nr@Tr) r-r.rrr2lenrr0r5 decoder_embedr2 decoder_predr;rs r#r.zSegGptDecoder.__init__7s YY   V%M%M!N N   q 6#=#= =  .f5 ++#)#=#=  r"rrNc|j\}}}}|j||||j|j|j}|j dddddd}|j|d||jz||jzf}|S) Nrrrr@rrarA)rArjr0r5rC)r:rrEr|r}r~s r#_reshape_hidden_statesz$SegGptDecoder._reshape_hidden_statesCs3@3F3F0 L+q%--  k4??DOOUYUmUm &--aAq!Q? %--r<$//#A;QUQ`Q`C`a. r"cl|j|}|j|}|j|}|Sr)r>rCr?rs r#rIzSegGptDecoder.forwardOs8**=9 33MB ))-8 r") rrrr.rrrCrIrJrKs@r#r;r;6s9  E4E4E %J[J[ U%6%6r"r;cPeZdZUeed<dZdZdZddgZde jdd fd Z y ) SegGptPreTrainedModelr;modelrDTrMrmodulerNNc>|jj}t|tjtj frtj j|jjjtjd|j|jj|j_ |j%|jjjyyt|tj t"frJ|jjj|jjj%dyt|t&r tj j|j(jjtjd|j|j(j|j(_ tj j|j*jjtjd|j|j*j|j*_ yt|t,rtj j|j.jjtjd|j|j.j|j._ tjj j1|j2|tjj j1|j4|tjj j1|j6|tjj j1|j8|tjj j1|j:|yy)zInitialize the weightsr)rstdNr)rJ)r;initializer_ranger3rrr8init trunc_normal_r&datarrrrrzero_rr!fill_rrrrMrZnormal_rRrSrTrUrV)r:rHrJs r# _init_weightsz#SegGptPreTrainedModel._init_weights_skk++ fryy"))4 5"$!6!6v}}7I7I7L7LU]]7[bekn!6!o!r!r ##"FMM {{&   &&('  ? @ KK   " " $ MM   $ $S )  0$&GG$9$9  %%((7%:%b!!''(    ! %'GG$9$9  %%((7%:%b!!''(    !  0 1.0gg.C.C**//225==A/D/b++112  & & + HHMM ! !&"3"3 ! = HHMM ! !&"<"<# ! F HHMM ! !&"="=3 ! G HHMM ! !&"<"<# ! F HHMM ! !&"<"<# ! F2r") rrrrrbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrModulerRr!r"r#rFrFWs> $O&*#+];&GBII&G$&Gr"rFc6eZdZdeffd ZdefdZdeee efddfdZ e dde jd e jd e jd ee jd eed eedee j$deedeedeedeeeffdZxZS) SegGptModelr;ct||||_t||_t ||_|jyr)r-r.r;rMrHrencoder post_initrs r#r.zSegGptModel.__init__s;   *62$V,  r"rNc.|jjSr)rHrWrs r#get_input_embeddingsz SegGptModel.get_input_embeddingss///r"heads_to_pruneNc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr[layerr prune_heads)r:r_rbheadss r# _prune_headszSegGptModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr"rDrn prompt_masksrorrplabelsrrrc ||n|jj}| | n|jj} | | n|jj} ||nd}|jj j jj} |j| }|j| }tj||fd}|tj||fdntj||fd}||tjd||jj j} tj| dztj |j"} tj$| | dzz tj |j"}tj| |g}|j'd}|j ||||}|j)|||| | }|S) a prompt_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt pixel values. Prompt pixel values can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. prompt_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt mask. Prompt mask can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). feature_ensemble (`bool`, *optional*): Boolean indicating whether to use feature ensemble or not. If `True`, the model will use feature ensemble if we have at least two prompts. If `False`, the model will not use feature ensemble. This argument should be considered when doing few-shot inference on an input image i.e. more than one prompt for the same image. embedding_type (`str`, *optional*): Embedding type. Indicates whether the prompt is a semantic or instance embedding. Can be either instance or semantic. labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`, `optional`): Ground truth mask for input images. Examples: ```python >>> from transformers import SegGptImageProcessor, SegGptModel >>> from PIL import Image >>> import requests >>> image_input_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_2.jpg" >>> image_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1.jpg" >>> mask_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1_target.png" >>> image_input = Image.open(requests.get(image_input_url, stream=True).raw) >>> image_prompt = Image.open(requests.get(image_prompt_url, stream=True).raw) >>> mask_prompt = Image.open(requests.get(mask_prompt_url, stream=True).raw).convert("L") >>> checkpoint = "BAAI/seggpt-vit-large" >>> model = SegGptModel.from_pretrained(checkpoint) >>> image_processor = SegGptImageProcessor.from_pretrained(checkpoint) >>> inputs = image_processor(images=image_input, prompt_images=image_prompt, prompt_masks=mask_prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> list(outputs.last_hidden_state.shape) [1, 56, 28, 1024] ``` Fr@rtzLabels were provided, but bool_masked_pos were not. It will be set to default value. If you're training the model, make sure to provide a bool_masked_pos.rr)rpro)rrrr)r;rruse_return_dictrHrWr9r&rrrrylogger warning_oncer7rQrrr%rwr[)r:rDrnrfrorrprgrrrexpected_dtyper7bool_masked_pos_zerosbool_masked_pos_onesembedding_outputencoder_outputss r#rIzSegGptModel.forwardsv2C1N-TXT_T_TqTq$8$D $++JjJj &1%@T(UVO-77:O?? -n^m+ ,, -/!5# ' r"NNNNNNN)rrrrr.r)r^dictrlistrer rrrrrrrrr rrIrJrKs@r#rYrYs$|0&;0C4T#Y+?CDC 7;+/(,.2,0/3&*kllk#\\kll k "%"2"23 k #4. k! k**+k$D>k'tnkd^k u)) *kkr"rYtensorr0c|j\}}}}||z}||z}|j||||||f}|jdddddd}|j|||z|dzdzf}|S)NrBrr@rrrAr)rArjrC)rtr0rEr1rFrGr|r}s r#patchifyrvs.4ll+J feZ'L:%K ^^:|\:Wbdn"o^ pF ^^Aq!Q1 -F ^^:|k/I:WX=[\K\"]^ ^F Mr"r|r}c b|jd}t|jddz dz}||z|jdk7r"td|jdd|d|d |j|||||df }|j dd ddd d }|j|d||z||zf }|S)Nrrarr`rzNumber of patches z does not match patch height (z ) and width (r?rBrAr@r)rArrBrjrC)rtr|r}rEr0s r# unpatchifyrxsaJfll2&*s23Jk!V\\!_4 a 11OP\~]jkvjwwy z  ^^:|[*V`bc"d^ eF ^^Aq!Q1 -F ^^:q,2K[[eMe"f^ gF Mr"ceZdZfdZdej dej dej dej fdZxZS) SegGptLosscft||j|_|j|_yr)r-r.betar0rs r#r.zSegGptLoss.__init__*s& KK  ++r"rfr'rgroctj||fd}|dddddfjdd|jdzdz}t ||j d|jz|j d|jz}t j||d|j}||zj|jz }|S)aNComputes the L1 loss between the predicted masks and the ground truth masks. Args: prompt_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values from mask prompt. pred_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, 2*height, width)`): Predicted masks. labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Ground truth mask for input images. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: `torch.FloatTensor`: The mean L1 loss between the predicted masks and the ground truth masks. r@rtNrrnone) reductionr|) rryrepeatr0rxrArhsmooth_l1_lossr|sum)r:rfr'rgro ground_truthmaskr&s r#rIzSegGptLoss.forward/s2yy,!7Q? q!Tz*11!Q8JQ8NO$ 2 21 5 H,J\J\]^J_cgcrcrJrs LFQUQZQZ[t   "TXXZ/ r") rrrr.rrrrIrJrKs@r#rzrz)sK, !''!%%!!! ! )) !r"rzzM SegGpt model with a decoder on top for one-shot image segmentation. c eZdZdeffd Ze ddejdejdejdeejdee dee d eejd ee d ee d ee d e eeffdZxZS)SegGptForImageSegmentationr;ct||||_t||_t ||_|jyr)r-r.r;rYrGr;decoderr\rs r#r.z#SegGptForImageSegmentation.__init__Ys;    ( $V,  r"rDrnrfrorrprgrrrrNc ||n|jj}| | n|jj} | | n|jj} ||jj j j} tj| dztj|j} tj| | dzz tj|j} tj| | g}|jd}|j ||||||||| |  }| r |jn|d}tj|d}|j!|}d}| t#|j}|||||}| s)|f}| r ||dfz}|r| rdnd}|||fz}||f|z}|St%|||j&|j( S) aY prompt_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt pixel values. Prompt pixel values can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. prompt_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt mask. Prompt mask can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). feature_ensemble (`bool`, *optional*): Boolean indicating whether to use feature ensemble or not. If `True`, the model will use feature ensemble if we have at least two prompts. If `False`, the model will not use feature ensemble. This argument should be considered when doing few-shot inference on an input image i.e. more than one prompt for the same image. embedding_type (`str`, *optional*): Embedding type. Indicates whether the prompt is a semantic or instance embedding. Can be either instance or semantic. labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`, `optional`): Ground truth mask for input images. Examples: ```python >>> from transformers import SegGptImageProcessor, SegGptForImageSegmentation >>> from PIL import Image >>> import requests >>> image_input_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_2.jpg" >>> image_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1.jpg" >>> mask_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1_target.png" >>> image_input = Image.open(requests.get(image_input_url, stream=True).raw) >>> image_prompt = Image.open(requests.get(image_prompt_url, stream=True).raw) >>> mask_prompt = Image.open(requests.get(mask_prompt_url, stream=True).raw).convert("L") >>> checkpoint = "BAAI/seggpt-vit-large" >>> model = SegGptForImageSegmentation.from_pretrained(checkpoint) >>> image_processor = SegGptImageProcessor.from_pretrained(checkpoint) >>> inputs = image_processor(images=image_input, prompt_images=image_prompt, prompt_masks=mask_prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> result = image_processor.post_process_semantic_segmentation(outputs, target_sizes=[(image_input.height, image_input.width)])[0] >>> print(list(result.shape)) [170, 297] ``` Nr@rr) rDrnrfrorrprgrrrrartr)r&r'rr)r;rrrirGrHrWr7rrQrrr%ryrwrrrzr%rr)r:rDrnrfrorrprgrrrr7rmrnrrr'r&loss_fnridxs r#rIz"SegGptForImageSegmentation.forwardcsv2C1N-TXT_T_TqTq$8$D $++JjJj &1%@T(UVO-77:O**% 3%+-)/!5#  LWW%G%G\cdf\g"%*YY/Ir%R"\\"<=    -G<V_MD ]F#71:-/ /aQ73q 'tnq d^q  u33 4q q r"r)rYrFr)rF)7rcollections.abcr4 dataclassesrtypingrrrtorch.utils.checkpointrtorch.nnrrh activationsr modeling_layersr modeling_utilsr utilsr r rrconfiguration_seggptr get_loggerrrjrr%rWr)rMrrrrrrrrrr!r2r;rFrYrrvrxrzr__all__r!r"r#rsB!" $!9-DD.   H %  J+J  J,  :K:  :* BII FRryyRjK4biiK4^  U\\eTV[VbVb* %RYY %,,,^9 BII9 zbii< 0BIIB-GO-G-G`B'BBJ U\\ s u||  u|| 3 S U\\ ''T } !6}  } @ Qr"