rhHdZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZmZm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&e#jNe(Z)dZ*Gdde jVZ,Gdde jVZ-Gdde jVZ.GddeZ/e"GddeZ0e"Gdde0Z1e"dGd d!e0eZ2e"d"Gd#d$e0Z3gd%Z4y)&zPyTorch OpenAI ImageGPT model.N)AnyOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPast)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_conv1d_layer)auto_docstringlogging torch_float)ImageGPTConfigc ddl}ddl}t j j|}tjd||jj|}g}g}|D]l\} } tjd| d| |jj|| } |j| |j| jnt||D]\} } | dd} | jd} t!d | Ds| d d vr4tjd j#dj%| j|} | d d vr t'| d} | D]W} |j)d| r|jd| }n| g}|ddk(s|ddk(r t'| d} n|ddk(r t'| d} n|ddk(s|ddk(rt'| |d} t'| d} n|ddvrt'| d} t'| d} nt+| dk(r,| ddk(r$|ddk(rt'| |d} t'| d} nQ|ddk(rt'| d} t'| d} n0|dd k(rt'| d} t'| d} nt'| |d} t+|d!k\sEt-|d}| |} Zt+| dkDr| ddk(s| d dk(s| d d k(s| d dk(rn | j.| j.k(sJ tjd"| | d d#k(rbt5j6| j9|j:|j:j<| j>ddd|j:f<| d d$k(rot5j6| j9|j:|j:j<| j>dd|j:d!|j:zf<!| d d%k(ret5j6| j9|j:|j:j<| j>ddd!|j:zdf<t+| dk(rP| ddk(rH| d!dk(r@t5j6| j9|j:|j:| _| d dk(rt5j6| | _| d dk(r7t5j6| | j>d|j@dz ddf<O| d d k(r$t5j6| | j>d <{t5j6| | _|S#t$rtj dwxYw#t0$r1}|xj2| j.| j.fz c_d}~wwxYw)&z0 Load tf checkpoints in a pytorch model rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /c3$K|]}|dv yw))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/imagegpt/modeling_imagegpt.py z.load_tf_weights_in_imagegpt..Qs  n n )_stepz Skipping {})wtet transformerz [A-Za-z]+\d+z(\d+)wgweightbbiaswpewte)q_projk_projv_projc_attnr rattnc_projr-lm_headsoszInitialize PyTorch weight r6r7r8)!re tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendsqueezezipsplitanyformatjoingetattr fullmatchlenintshapeAssertionErrorargstorch from_numpyreshapen_embdTdata vocab_size)modelconfigimagegpt_checkpoint_pathr?tftf_path init_varsnamesarraysnamerVarraypointerm_name scope_namesnumes r(load_tf_weights_in_imagegptro0sy ggoo67G KK8 BC''0I E F ' e (l5'BC&&w5 T emmo& ' 5&)L3 eABxzz#    "X " KK ,,SXXd^< =  88 #g}5G 'F||OV4 hhx8 %h 1~$ A#(=!'84Q3&!'62Q5(KNe,C!';q>:!'84Q#AA!'84!'84TaDGv$5+a.H:T!';q>:!'84Q6)!'95!'84Q5(!'51!'84!';q>:;1$+a.)!#,; '> t9q=T!W.$r(f2DRTYHY]abd]ein]n  }} 333  078 8x /4/?/? fmm]c]j]j@k/l/n/nGLLOfmmO+ , "X !AFAQAQ fmmV]];Ba LLFMMA ,=== >"X !383C3CEMMRXR_R_agananDo3p3r3rGLLA -// 0 Y!^Q6 1d1g6I ++EMM&--,WXGL "X  ++E2GL "X 7<7G7G7NGLL06,,q00!3 4 "X $//6GLL  ++E2GLYL3\ LC   Q   P" 7==%++66 s#U7)V7 V W#,WWcheZdZddeedeffd ZdejdejfdZ xZ S)ImageGPTLayerNorm hidden_sizeepsct|||_tjt j ||_yN)super__init__rsr ParameterrYTensorr1)selfrrrs __class__s r(rwzImageGPTLayerNorm.__init__s. ll5<< #<= tensorreturnc|tjtjtj|dd|jzz }||j z}|S)Nr+T)axiskeepdim)rYsqrtmeansquarersr1)rzr}s r(forwardzImageGPTLayerNorm.forwardsK%**UZZ V0D2W[%\_c_g_g%ghh$++% r|)gh㈵>) __name__ __module__ __qualname__tuplerUfloatrwrYryr __classcell__r{s@r(rqrqs5>E#J>U> ellu||r|rqcBeZdZddeedeeffd ZdZddZddZ dZ dZ dd e jd eed ee jd ee jd ee jdee jdeedeedee jdefdZxZS)ImageGPTAttentionis_cross_attention layer_idxc t||j}|jdt j t j ||ftjjdd||d|jdt jdd|j|_ |j|_ |j|jz|_|j|_|j|jz|jk7r&t!d|jd |jd |j"|_||_|j&|_||_|j*|_|j$rNt-d |jz|j|_t-|j|j|_n(t-d |jz|j|_t-|j|j|_t5j6|j8|_t5j6|j<|_tA|_!y) Nr3dtyperF) persistent masked_biasgz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r>r )"rvrwmax_position_embeddingsregister_bufferrYtrilonesboolviewr}rr embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsrscale_attn_by_inverse_layer_idxrreorder_and_upcast_attnrr9q_attnr;rDropout attn_pdrop attn_dropout resid_pdrop resid_dropoutset pruned_heads)rzrarr max_positionsr{s r(rwzImageGPTAttention.__init__s 66    JJuzz=-"@ S T Y Y1m]    ]ELL,>5Q++33$..8 .. ==4>> )T^^ ;OPTP^P^O_`NN#2'  #)";";"406/U/U,"'-'E'E$  " " T^^!3T^^DDK @DK T^^!3T^^DDKT^^T^^< JJv'8'89ZZ(:(:;Er|cFt|dk(ryt||j|j|j\}}t j |||jz|d|jzzg}t|j|d|_ t|j|d|_ |j|jz|jt|z z|_|jt|z |_|jj||_y)Nrr>rdim) rTrrrrrYcatrrr9r;union)rzheadsindex index_attns r( prune_headszImageGPTAttention.prune_headss u:? 7t~~t}}^b^o^op uYYut'>T__I\@]^_ )jaH (eC  ??dnn<RUV[R\A\]#e*4 --33E:r|cDtj||jdd}|jr |t |j ddzz }|j r|t|jdzz }|js|j d|j d}}|jdddd||z |d|f} tj|jj} tj| |j|j} tj | || }|||z}t#j$d|}|j'|j}|j)|}|||z}tj||} | |fS)Nr+?rrdevicer)rYmatmul transposerrsizerrrrr3finforminr}rwhererSoftmaxtyper) rzquerykeyvalueattention_mask head_mask attn_weights query_length key_length causal_mask mask_value attn_outputs r(_attnzImageGPTAttention._attnst||E3==R+@A  " "'+ejjn6K*LLL  / /'%0B*CCL&&',zz"~sxx|*L))Aq*|*Cj*PR]S]R]$]^K\%7%78<rr )rrpermuterzr}rattn_head_size new_shapes r( _split_headszImageGPTAttention._split_heads>sEKKM#2&)^)DD i(~~aAq))r|c|jddddj}|jdd||zfz}|j|S)zS Merges attn_head_size dim and num_attn_heads dim into hidden_size rr>rr Nr)r contiguousrrrs r( _merge_headszImageGPTAttention._merge_headsFsO1a+668KKM#2&)n*D)FF {{9%%r| hidden_states layer_pastrrencoder_hidden_statesencoder_attention_mask use_cacheoutput_attentionscache_positionr~c |du} |j\} } } |St|trA|jj |j }| r |j }n|j}n|}| r|n|}| r%t|ds td|[rY|j|}j|j j}|j|j j}nQ|j|}|j|j|j d\}}|j#| d|j$|j&j)dd}|j#| d|j$|j&j)dd}n|j|j|j d\}}}|j#| d|j$|j&j)dd}|j#| d|j$|j&j)dd}|D| s| nd} j+|||j d| i\}}| rd|j|j <|j#| | |j$|j&j)dd}|j,r|j/|||||\}}n|j1|||||\}}|j3||j$|j&}|j5|}|j7|}||fS) NrzIf class is used as cross attention, the weights `q_attn` have to be defined. Please make sure to instantiate class with `ImageGPTAttention(..., is_cross_attention=True)`.r>rr+rrT)rV isinstancer is_updatedgetrcross_attention_cacheself_attention_cachehasattrrrlayerskeysvaluesr9rNrrrrrupdaterrrrr;r)rzrrrrrrrrrrrseq_lenrrcurr_past_key_valuecurrent_statesrrrrrs r(rzImageGPTAttention.forwardNs3$>'--Wa  !*&9:'2266t~~F %*4*J*J'*4*I*I'&0#2D.- 4* t %* M2)00@EE+224>>BII M2![[8>>tTU>V UhhsB FPPQRTUV 3DNNDMMJTTUVXYZ $ N ; A A$//WX A Y E3((3DNNDMMBLLQPQRCJJsB FPPQRTUVE  !3E^4N,33CQacqPrsJC!8< %%dnn5 3GQQRSUVW  ' '(,(G(GsTY[ikt(u %K(, 5#unV_(` %K'' T^^T]]S kk+. ((5 L((r|)FN)NNNNNNNFFN)rrrrrrUrwrrrrrrYryr rrrrs@r(rrs)"8D>)"V^_bVc)"V ;$)L2)h*&'+15,08<9=$),115D)||D)UOD)!. D) ELL) D) ( 5 D)!) 6D)D>D)$D>D)!.D) D)r|rcVeZdZfdZdej dej fdZxZS) ImageGPTMLPct||j}t|||_t|||_t |j|_tj|j|_ yru) rvrwrrrc_fcr;r activation_functionactrrrdropout)rzintermediate_sizerarr{s r(rwzImageGPTMLP.__init__s_ && ,i8 Y(9: &445zz&"4"45 r|rr~c|j|}|j|}|j|}|j|}|Sru)rrr;r)rzrs r(rzImageGPTMLP.forwards@ -0 /  M2  ]3 r|)rrrrwrYryrrrs@r(rrs#6U\\ellr|rc eZdZd fd Z ddej deedeej deej deej deej deed eed eej d e fd Z xZ S) ImageGPTBlockct||j}|j |jnd|z}t ||j |_t|||_t ||j |_ |jr/t|d||_ t ||j |_ t|||_y)NrsrT)rr)rvrwrrn_innerrqlayer_norm_epsilonln_1rr:ln_2add_cross_attentioncrossattention ln_cross_attnrmlp)rzrarrr inner_dimr{s r(rwzImageGPTBlock.__init__s (( &,nn&@FNNa+o %kv7P7PQ %f B %kv7P7PQ  % %"3Ft_h"iD !2;FD]D]!^D y&1r|rrrrrrrrrr~c |} |j|}|j||||||| } | d} | dd} | | z}|Yt|dstd|d|} |j |}|j |||||||| }|d} | | z}| |ddz} |} |j |}|j|}| |z}|f| zS)N)rrrrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rrrrrrr)rr:rrrrrr)rzrrrrrrrrrresidual attn_outputsroutputscross_attn_outputsfeed_forward_hidden_statess r(rzImageGPTBlock.forwards:! -0 yy !)/)! #1o qr"#h. ,4!12 =dVDZZ%H ..}=M!%!4!4%-#&;'="3-"5 " -Q/K${2M 212 66G  -0 %)XXm%<" #== '))r|rur) rrrrwrYryrr rrrrrs@r(r r s 2$'+15,08<9=$),115:*||:*UO:*!. :* ELL) :* ( 5 :*!) 6:*D>:*$D>:*!.:* :*r|r cFeZdZUeed<eZdZdZdZ dgZ fdZ dZ xZ S)ImageGPTPreTrainedModelrar. input_idsTr c$t||i|yru)rvrw)rzinputskwargsr{s r(rwz ImageGPTPreTrainedModel.__init__s &+F+r|c t|tjtfrl|jj j d|jj|j|jj jnt|tjry|jj j d|jj|jg|jj |jjn5t|tr%|jj jd|jD]m\}}d|vs d|vs|j j d|jjt!j"d|jj$zz oy)zInitialize the weights.g)rstdNrr;r1r>)rrLinearrr1r^normal_rainitializer_ranger3zero_ Embedding padding_idxrqfill_named_parametersmathrn_layer)rzmodulerhps r( _init_weightsz%ImageGPTPreTrainedModel._init_weightssQ fryy&1 2 MM   & &CT[[5R5R & S{{&   &&(  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>  1 2 MM   $ $S )..0 sGD!4H$4Cdkk.K.KdiiXY\`\g\g\o\oXoNp.pr sr|)rrrr__annotations__roload_tf_weightsbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrwr4rrs@r(r!r!s3 1O%!O&*#(),sr|r!c$eZdZdeffd ZdZdZdZe dde e jde e e e jde e jd e e jd e e jd e e jd e e jd e e jde e jde e de e de e de e de e jdedee eff dZxZS) ImageGPTModelrac vt|||j|_t j |j |j|_t j |j|j|_ t j|j|_ t jt|jDcgc]}t!||c}|_t%|j|j&|_d|_d|_d|_|j1ycc}w)NrrF)rvrwrrrrr,r_r5rr4r embd_pdropdrop ModuleListrangenum_hidden_layersr hrqrln_fmodel_parallel device_mapgradient_checkpointing post_init)rzrair{s r(rwzImageGPTModel.__init__s  ++<< 1 14>>B<< > >OJJv001 ERXRjRjLklq f Blm%dnn&:S:ST $&+#  ms D6c|jSrur5)rzs r(get_input_embeddingsz"ImageGPTModel.get_input_embeddings/s xxr|c||_yrurK)rznew_embeddingss r(set_input_embeddingsz"ImageGPTModel.set_input_embeddings2s !r|c|jD]-\}}|j|jj|/y)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrCr:r)rzheads_to_prunelayerrs r( _prune_headszImageGPTModel._prune_heads5s<+002 2LE5 FF5M   * *5 1 2r|r"past_key_valuesrtoken_type_ids position_idsr inputs_embedsrrrroutput_hidden_states return_dictrr%r~c | | n|jj} | | n|jj} | | n|jj} | | n|jj} | | t d|G|j |||j}|jd|d}|jd}n0|#|jdd}|jd}n t d| |jn |j}|jr%|jr| rtjdd} d}| rt)j*||d|zt(j,| }|j/d}|z|dkr t d |j|d}|ddddddf}|j1|j2 }d |z t)j4|j2j6z}|jj8rE|C|j\}}}||f}| t)j:||} |j=| } nd} |j?||jj@}||jC|}|jE|}||j1|jz}||jC|}||z}|jG|}||jdfz}| rdnd}| r|jj8rdnd}| rdnd}tI|jJD]L\} }!|jLr{t(jNjQ|j||j1|j}t|t(jRr|j1|j}| r||fz}|!||||| || | | | }"|"d}| r(||"dfz}|jj8r ||"dfz}|jLs|jTjWD]J\}#}$| |$dk(sdtY|#z|jZk7s+|j1dtY|#dzz}LO|j]|}|j|}| r||fz}|r|j_}| stad|||||fDStc|||||S)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTModel >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer+rz5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.Trz$batch_size has to be defined and > 0rr)rr%)rrrrrr>zcuda:c3$K|]}|| ywrur%)r&vs r(r)z(ImageGPTModel.forward..s=r*)last_hidden_staterUr attentionscross_attentions)2rarrYruse_return_dictr%warn_if_padding_and_no_attention_maskrrrVrrGtrainingrB warning_oncerr r from_legacy_cacheget_seq_lengthrYarangelong unsqueezetorrrrrinvert_attention_mask get_head_maskr1r5r4r? enumeraterCrEcuda set_deviceryrFrQstr last_devicerDto_legacy_cacherr)%rzr"rUrrVrWrrXrrrrrYrZrr% input_shape batch_sizerreturn_legacy_cache past_lengthencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeposition_embedsrtoken_type_embeds output_shapeall_self_attentionsall_cross_attentionsall_hidden_statesrIblockrrr]s% r(rzImageGPTModel.forward<s`2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T  & &4==##p" # Z?   \  #' 1CCOTO:I:Uo446[j  %+00[_EN   << [_{5RZ_ZdZdmstL'11!4L  %Q !GHH+00R@N ,AtT1,<=N,..TZZ.@N!N2ekk$**6M6Q6QQN ;; * */D/P=R=W=W=Y :  7$68O#P %-).4HQW)X&%)%?%?@V%W "%) " &&y$++2E2EF   HHY/M((<0%(:(:=;O;O(PP  % $ 8 ),==M -0 "m&8&8&<%>> $5b4%64;;;Z;Zr`d"6BD!$&&)" OHAu"" %%m&:&:;!-%3%6%6}7K7K%LNi6 ) ]-A-A BI#$58H$H!! %'=#"3- G$AJM &9WQZM&I#;;22+?71:-+O("" OO113ODAqAbEzgA&6$:J:J&J(5(8(83q1u:9M(N OA" OH -0 * **L9  1]4D D  -==?O':KM`bvw  9+++*1   r|)NNNNNNNNNNNNNN)rrrrrwrLrOrTrrrYryrrrrrrrrs@r(r<r<s~&"2-1@D1515/3,0048<9=$(,0/3&*15N ELL)N "%ell(;"<=N !. N !. N u||, N ELL)N   -N  ( 5N !) 6N D>N $D>N 'tnN d^N !.N !N " u?? @#N N r|r<z The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc&eZdZdgZdeffd Ze ddeejdee e ejdeejdeejdeejd eejd eejd eejd eejd eejdee dee dee dee deejde de e eff"dZxZS)ImageGPTForCausalImageModelingzlm_head.weightract||t||_t j |j |jdz d|_d|_ d|_ |jy)NrFr3) rvrwr<r.rr(r\r_r<rErFrHrzrar{s r(rwz'ImageGPTForCausalImageModeling.__init__s[  (0yy0A0AA0EER $ r|r"rUrrVrWrrXrrlabelsrrrYrZrr%r~c (||n|jj}|j||||||||| | | | ||}|d}|j|}d}| r|dddddfj }| dddfj }t }||j d|jd|j d}|s|f|ddz}||f|zS|St|||j|j|j|jS)a& input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling >>> import torch >>> import matplotlib.pyplot as plt >>> import numpy as np >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small") >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> model.to(device) # doctest: +IGNORE_RESULT >>> # unconditional generation of 8 images >>> batch_size = 4 >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1) # initialize with SOS token >>> context = context.to(device) >>> output = model.generate( ... input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40 ... ) >>> clusters = image_processor.clusters >>> height = image_processor.size["height"] >>> width = image_processor.size["width"] >>> samples = output[:, 1:].detach().cpu().numpy() >>> samples_img = [ ... np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples ... ] # convert color cluster tokens back to pixels >>> f, axes = plt.subplots(1, batch_size, dpi=300) >>> for img, ax in zip(samples_img, axes): # doctest: +IGNORE_RESULT ... ax.axis("off") ... ax.imshow(img) ```N) rUrrVrWrrXrrrrrYrZrr.r+r)losslogitsrUrr_r`) rarar.r<rrrrrrUrr_r`)rzr"rUrrVrWrrXrrrrrrYrZrr%transformer_outputsr lm_logitsr shift_logits shift_labelsloss_fctoutputs r(rz&ImageGPTForCausalImageModeling.forward"sRN&1%L!#qr'?557L')HL--b,2C2CB2GH,J[J[\^J_`D\$7$;;F)-)9TGf$ Ev E0/??-;;*550AA   r|)NNNNNNNNNNNNNNN)rrr_tied_weights_keysrrwrrrYryrrrrrrrrs@r(rrs++ ~ -1@D1515/3,0048<9=)-$(,0/3&*15!p ELL)p "%ell(;"<=p !. p !. p u||, p ELL)p   -p  ( 5p !) 6p &p D>p $D>p 'tnp d^p !.!p "#p $ u77 8%p p r|rz The ImageGPT Model transformer with an image classification head on top (linear layer). [`ImageGPTForImageClassification`] average-pools the hidden states in order to do the classification. c eZdZdeffd Ze ddeejdee e ejdeejdeejdeejdeejd eejd eejd ee d ee d ee dee de de e e ffdZxZS)ImageGPTForImageClassificationract|||j|_t||_t j |j|jd|_|jy)NFr) rvrw num_labelsr<r.rr(r\scorerHrs r(rwz'ImageGPTForImageClassification.__init__sR   ++(0YYv}}dooEJ  r|r"rUrrVrWrrXrrrrYrZr%r~c \| | n|jj} |j|||||||| | | |  }|d}|jd}|j |}d}||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&||j|j}n|||}n|jj dk(r=t}||jd |j |jd }n,|jj dk(rt}|||}| s|f|ddz}||f|zS|St!|||j"|j$|j& S) ay input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```N) rUrrVrWrrXrrrYrZrrr regressionsingle_label_classificationmulti_label_classificationr+)rrrUrr_)rarar.rr problem_typerrrYrhrUr rLrrrrrUrr_)rzr"rUrrVrWrrXrrrrYrZr%rrpooled_hidden_statesrrrrs r(rz&ImageGPTForImageClassification.forwardsd&1%f $D>f 'tnf d^f f  u66 7f f r|r)rrr<r!ro)5__doc__r0rDtypingrrrrYtorch.utils.checkpointrtorch.nnrrr activationsr cache_utilsr r generationrmodeling_layersrmodeling_outputsrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_imagegptr get_loggerrrBroModulerqrrr r!r<rr__all__r%r|r(rsO% '' AA!5)9 .YY 3   H %iX  i) i)X"))"J*.J*Z#so#s#sLp +p p f  %