rhDddlZddlZddlmZddlmZmZddlZddlm cm Z ddlm Z ddl m Z mZmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)e$rddl*m+Z+ddl,m-Z-ddl.m/Z/ne0Z-e%jbe2Z3GddejhjjZ6 dTdeejndee8fdZ9Gdde-Z:Gdde jvZ<Gdde jvZ=Gdd e jvZ>d!Z?dUd"Z@ dVd#d$d%ejnd&ejnd'ejnd(eejd)eBe8e8fd*e8d+e8d,eeCd-eeBejnejnfeBejnffd.ZDejfd#d$d%ejnd/e:dejnde8d)eBe8e8fd*e8d+e8d0ejd-eBejnfd1ZGd#d$d%ejnd&ejnd'ejnd(eejd)eBe8e8fd*e8d+e8d-eBejnfd2ZHeGeDeHd3ZIGd4d$e jvZJGd5d6eZKe#Gd7d8e!ZL dTd9ejnd&ejnd(eejnd:eejnd-eBejnejnejne8eejneejnff d;ZMd9ejnde8d-ejnf d?ZNe#Gd@dAeLZOGdBdCe jvZPe#dDEGdFdGeLZQe#dHEGdIdJeLZRe#dKEGdLdMeLZSe#GdNdOeLZTe#dPEGdQdReLZUgdSZVy)WN) nullcontext)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)auto_docstringis_flash_attn_2_availablelogging)is_triton_available)ModernBertConfig) flash_attn_varlen_qkvpacked_func)RotaryEmbedding) apply_rotaryc\eZdZe ddeej deefdZedZ y)ApplyRotaryEmbUnpadN cu_seqlens max_seqlenc |j}|j\}}}} |ddddfj|d| } t| ||d||dd|j |||||_|S)NrFT)seqlen_offsetsr"r# interleavedinplace) contiguousshapeviewrsave_for_backwardr#) ctxqkvcossinr"r# total_nnz_three_nheadsheaddimqks /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/modernbert/modeling_modernbert.pyforwardzApplyRotaryEmbUnpad.forward>snn.1ii+ 67GBQBZ__YG 4   !!  c3 3# c |j\}}}|j}|j\}}}}|ddddfj|d|} t | ||d||j ddd |ddddddfS)Nr%r&rFT)r'r"r#r(r) conjugate) saved_tensorsr*r+r,rr#) r.dor0r1r"r2r3r4r5dqks r7backwardzApplyRotaryEmbUnpad.backward]s"00S* ]]_.0hh+ 67GBQBinnYG4   !~~ 4tT455r9NN) __name__ __module__ __qualname__ staticmethodrtorchTensorintr8r?r9r7r!r!=sQ .2$(  U\\*  SM <66r9r!r"r#c4tj|||||S)a Arguments: qkv: (total_nnz, 3, nheads, headdim) - input tensor for packed QKV. cos, sin: (seqlen_rotary, rotary_dim / 2) interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). inplace: if True, apply rotary embedding in-place. seqlen_offsets: (batch_size,) or int. Each sequence in x is shifted by this amount. Most commonly used in inference when we have KV cache. cu_seqlens: (batch + 1,) or None max_seqlen: int Return: out: (total_nnz, dim) rotary_dim must be <= headdim Apply rotary embedding to the first rotary_dim of x. )r!apply)r/r0r1r"r#s r7apply_rotary_unpaddedrKts.  $ $S#sJ KKr9c "eZdZdZ d dededeedeejdeejf fd Z ddejd ejdeed e eje ejejfffd Zd efd ZxZS)!ModernBertUnpaddedRotaryEmbeddingzP The rotary position embeddings applied directly to unpadded sequences. dimbaser#devicedtypectt||||d||_||||j|||yyyy)a max_seqlen: if max_seqlen, device, and dtype are provided, we precompute the cos_sin_cache up to max_seqlen. If the max_seqlen, device, or dtype during training/inference differ, the cos_sin_cache will be recomputed during the forward pass. F)rNrOrPr(NrPrQ)super__init__r#_update_cos_sin_cache)selfrNrOr#rPrQ __class__s r7rUz*ModernBertUnpaddedRotaryEmbedding.__init__sS StFN$  !f&8U=N  & &z& & N>O&8 !r9r/r"returnc|(|j||j|jt||j|j ||}|S)z Apply rotary embedding *inplace* to qkv. qkv: (total_nnz, 3, nheads, headdim) cu_seqlens: (batch + 1,) cumulative sequence lengths max_seqlen: int max seq length in the batch rSr"r#)rVrPrQrK _cos_cached _sin_cached)rWr/r"r#s r7r8z)ModernBertUnpaddedRotaryEmbedding.forwardsS  !  & &z#**CII & V#       !!   r9cTd|jd|jd|jS)Nzdim=z, base=z , scale_base=)rNrO scale_baserWs r7 extra_reprz,ModernBertUnpaddedRotaryEmbedding.extra_reprs(dhhZwtyykt>OPPr9)g@NNNN)rArBrC__doc__rGfloatrrErPrQrUrFrtupler8strra __classcell__rXs@r7rMrMs$()-'+ O OOSM O & O  $ O.%)  \\LLSM  u||U5<<#=>> ? 2QCQr9rMceZdZdZdeffd ZejddejdejfdZ d de ejd e ejdejfd Z xZ S) ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcdt|||_tj|j |j |j|_tj|j |j|j|_ tj|j|_y)N) padding_idxepsbias)rTrUrkr Embedding vocab_size hidden_size pad_token_idtok_embeddings LayerNormnorm_eps norm_biasnormDropoutembedding_dropoutdroprWrkrXs r7rUzModernBertEmbeddings.__init__sw   ll6+<+P>P^d^q^qrLL!3!3vO_O_` JJv778 r9Tdynamic input_idsrYc`|j|j|j|Srb)r|ryru)rWrs r7compiled_embeddingsz(ModernBertEmbeddings.compiled_embeddingss%yy4#6#6y#ABCCr9 inputs_embedsc|"|j|j|}|S|jjr|j |n.|j|j|j |}|Srb)r|ryrkreference_compilerru)rWrr hidden_statess r7r8zModernBertEmbeddings.forwardsu  $ IIdii &>?M;;00((3YYtyy)<) *z&:M:Mt/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.rb) rArBrCrrUrEno_gradrr8rgrhs@r7rrs4///"U]]_ <