rhsdZddlmZmZddlZddlZddlmZddlmZddl m Z m Z ddl m Z dd lmZdd lmZdd lmZmZdd lmZdd lmZmZmZddlmZer ddlmZddlm Z ejBe"Z#de$de$dejJfdZ&dejJdejJfdZ'dejJdejJdejJdejJfdZ(GddejRZ*GddejRZ+Gd d!eZ,eGd"d#eZ-eGd$d%e-Z.ed&'Gd(d)e-e Z/gd*Z0y)+zPyTorch CodeGen model.)OptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)auto_docstringis_torch_flex_attn_availablelogging) CodeGenConfig) BlockMask)make_flex_block_causal_masknum_posdimreturncddtjd|dtj|z zz }tjdtj|tjj |j }tj tj |tj|fdS) N?i'rdtypez i , j -> i jrr)torcharangeint64einsumfloatcatsincos)rrinv_freq sinusoid_inps /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/codegen/modeling_codegen.pycreate_sinusoidal_positionsr+0se QQekk JS PQRH<< WEKK0X0^0^0`bjkqqsL 99eii -uyy/FGQ OOxc|dddddddddf}|dddddddddf}tj| |fd}|jdS)Nrrr)r stackflatten)r-x1x2s r*rotate_every_twor57sS 1aCaC<B 1aADqD= B bS"I2&A 99R=r,tensorr&r'ctj|dddddddfdd}tj|dddddddfdd}||zt||zzS)Nrr)r repeat_interleaver5)r6r&r's r*apply_rotary_pos_embr9?s^  ! !#aD!m"4a ;C  ! !#aD!m"4a ;C SL-f5; <>"#;N$NOr,ct|jdk(r$|jdddddj}n\t|jdk(r#|jddddj}n!t dt|j|j dd ||zfz}|j |S) zM Merges attn_head_size dim and num_attn_heads dim into n_ctx rrrrz3Input tensor rank should be one of [4, 5], but is: Nr0)lenr`permute contiguousrOsizeview)rZr6rMattn_head_size new_shapes r* _merge_headszCodeGenAttention._merge_headsis v||  !^^Aq!Q2==?F  ! #^^Aq!Q/:: MMD$<$<<F !KK 9"Euc!!%)A)A4==Y_!`T%=%=t}}U[\!!%)A)A4==Y_!` aAq)..  ! !\%8%8 8-001D1DEO#2D  .;;vv||B'71'<"ES ?? &1a!24??!223EAq$//"334F!Q#4T__#445E1aDOO$556F(S9E(S9E))UFO4CIIufo26E&sC5C(S9Ekk!Q1% aAq)  !)-"0 L $**366-2E2E+Ft~~_klJC%)JJuc5.R[$\! \'' T5M5Mt}}] mmK0 ((5 L((r,N)NNNNNNFFN)rJ __module__ __qualname__r?rerpr}rr FloatTensorr LongTensorboolrtupleTensorr __classcell__rIs@r*r;r;Es6X< &$ )D'+6:3715$),159G) 1 12G)UOG)!!2!23 G) u//0 G) E--. G)D>G)$D>G)!!1!12G)  ellE%,,//0u||U5<<%8% c@Q:RRST V G)r,r;c\eZdZfdZdeej dej fdZxZS) CodeGenMLPct||j}tj|||_tj|||_t|j|_ tj|j|_ yr) r>r?n_embdrrUfc_infc_outractivation_functionactrArDdropout)rZintermediate_sizer[rLrIs r*r?zCodeGenMLP.__init__se MM YYy*;< ii 19= &445zz&"4"45 r,r~rc|j|}|j|}|j|}|j|}|Sr)rrrr)rZr~s r*rzCodeGenMLP.forwards@ =1 /  M2  ]3 r,) rJrrr?rr rrrrs@r*rrs,6Xe.?.?%@UEVEVr,rcVeZdZd fd Z ddeej deedeej deejdeej dee dee d eejd e e ejee eje ej d ffffd Z xZS) CodeGenBlockct||j |jnd|jz}t j |j|j |_t|||_ t|||_ y)Nrheps) r>r?n_innerrr LayerNormlayer_norm_epsilonln_1r;attnrmlp)rZr[rF inner_dimrIs r*r?zCodeGenBlock.__init__sc &,nn&@FNNa&--FW LLF4M4MN $VY7 i0r,r~rrxrryrrrr.c |} |j|}|j||||||||\} } |j|} | | z| z}|| fS)N)r~rrxrryrrr)rrr) rZr~rrxrryrrrresidual attn_outputsrzfeed_forward_hidden_statess r*rzCodeGenBlock.forwardss! -0 %)YY'!)%/)&/ & " l&*XXm%<"$'AAHL l**r,rr)rJrrr?rr rrrrrrrrrrs@r*rrs1'+6:3715$),159+ 1 12+UO+!!2!23 + u//0 + E--. +D>+$D>+!!1!12+ uU\\"HU5<<uGXGXZ]G]A^3^-_$`` a+r,rcFeZdZUeed<dZdZdgZdZdZ fdZ dZ xZ S)CodeGenPreTrainedModelr[ transformerTrpast_key_valuesc$t||i|yr)r>r?)rZinputskwargsrIs r*r?zCodeGenPreTrainedModel.__init__$s &+F+r,ct|tjfrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weights.)meanstdNr) isinstancerrUweightdatanormal_r[initializer_ranger=zero_ Embedding padding_idxrfill_)rZmodules r* _init_weightsz$CodeGenPreTrainedModel._init_weights's fryyl + MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r,) rJrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraphr?rrrs@r*rrs4 %&*#'("3!,*r,rc^eZdZfdZdZdZe ddeejdee e e e ejfdeejdeejdeejd eejd eejd eed eed eedeedeejde e effdZ dde ejdfdejdejde d ef dZedejdededej*dejdef dZxZS) CodeGenModelc t|||j|_|j|_t j |j|j|_t j|j|_ t jt|jDcgc]}t||c}|_t j |j|j"|_t'|j(|j*|j,z|_d|_|j1ycc}w)N)rFrF)r>r?rrL vocab_sizerrwterA embd_pdropdrop ModuleListrangen_layerrhrrln_fminrXn_ctxrMgradient_checkpointing post_init)rZr[irIs r*r?zCodeGenModel.__init__:s   ++<< 1 14>>BJJv001 5QWQ_Q_K`aa Vq AabLLV5N5NO f//A[A[1[\&+#  bs,Ec|jSrr)rZs r*get_input_embeddingsz!CodeGenModel.get_input_embeddingsJs xxr,c||_yrr)rZnew_embeddingss r*set_input_embeddingsz!CodeGenModel.set_input_embeddingsMs !r, input_idsrrxtoken_type_idsrry inputs_embedsrroutput_hidden_states return_dictrrc | | n|jj} | | n|jj} ||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||j|}t|tdtfs t d|r | t}|jd}| 9||j!nd}t#j$|||z|j&} || j)d}|j+||| || }|j-||jj.}|}|(|j1d |}|j|}||z}|j3|}d ||j5d f}| rd nd}| rd nd}t7|j8D]1\}}| r||fz}||||||||| | }|d}| s)||dfz}3|j;|}|j1|}| r||fz}| st=d ||||fDSt?|||| S)a inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzBThe `past_key_values` should be either a `Cache` object or `None`.rrrr/)rrxrryrrrc3&K|] }|| ywrr).0vs r* z'CodeGenModel.forward..sghgts)last_hidden_staterr~ attentions) r[rrruse_return_dictrOrtrainingrGrHrrtyperr r`get_seq_lengthr r!r unsqueeze_update_causal_mask get_head_maskrrmrrl enumeraterrrr )rZrrrxrrryrrrrrrr seq_lengthpast_seen_tokensr{r~token_type_embeds output_shapeall_self_attentionsall_hidden_statesrblockoutputss r*rzCodeGenModel.forwardPs.2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?ab b 0*nO"((+  !CRC^==?de "\\*:?L] &&y$++2E2EF %  %+00Z@N $ 8 ),==M -0 J (:(:2(>? $5b4"6BD!$&&) JHAu#$58H$H!**)#A,#"3- G$AJM &9WQZM&I## J& -0 %**<8  1]4D D )?T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[Kr,rrrrc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nrh) fill_valuerrr)diagonalrr/r)rr rrfullrtriur!r_expandcloner`rR masked_fill) rxrrrrrrr{r  mask_length padding_masks r*rzBCodeGenModel._prepare_4d_causal_attention_mask_with_cache_position s>  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r,) NNNNNNNNNNNN)F)rJrrr?rrrrr rrrrrrrr rr staticmethodintrrrrs@r*rr8s "15NR6:59371559$(,0/3&*59r E,,-r "%uU5<<5H/I(I"JKr !!2!23 r !!1!12 r u//0 r E--.r   1 12r D>r $D>r 'tnr d^r !!1!12r  u-- .r r v#( BellK78BllB B  B  BH4 444{{ 4  4  44r,rzM The CodeGen Model transformer with a language modeling head on top. ) custom_introc eZdZdgZfdZe ddeejdee e e e ejfdeejdeejdeejdeejd eejd eejd eed eed eedeedeejde e effdZxZS)CodeGenForCausalLMzlm_head.weightct||t||_t j |j |j|_|jyr) r>r?rrrrUrrlm_headr)rZr[rIs r*r?zCodeGenForCausalLM.__init__KsE  '/yy0A0AB  r,rrrxrrryrlabelsrrrrrrc $| | n|jj} |j|||||||| | | | |  }|d}|j|j t j }d}|`|j |j}|j||fd|jji|}|j |j}| s|f|ddz}||f|zS|St|||j|j|jS)aG inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) rrxrrryrrrrrrrrr)losslogitsrr~r)r[rrr1rRr rQr loss_functionrrrrr~r)rZrrrxrrryrr2rrrrrrtransformer_outputsr~ lm_logitsr4outputs r*rzCodeGenForCausalLM.forwardSsF8&1%J $D>J 'tnJ d^J !!1!12J u,, -!J J r,r/)r/rr)1__doc__typingrrr torch.utils.checkpointr activationsr cache_utilsrr generationr modeling_attn_mask_utilsr modeling_layersr modeling_outputsr rmodeling_utilsrutilsrrrconfiguration_codegenr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerrJrGr,rr+r5r9Moduler;rrrrr/__all__rr,r*rLsm" !.)>9O- 1 !;J   H %PP3P5<<P ==ELL=u||=X]XdXd= W)ryyW)v(#+-#+L*_**:G)GGT V /V  V r Kr,