rht|ddlmZmZddlZddlmZddlmcmZddl Zddl m Z ddl m Z ddlmZddlmZmZmZmZmZmZmZmZd d lmZej6eZGd d ej<ZGd deZ ddZ!GddeZ"GddeZ#GddeZ$GddeZ%GddeZ&gdZ'y))CallableOptionalN)Cache)ALL_ATTENTION_FUNCTIONS)logging)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModelLlamaRotaryEmbeddingeager_attention_forward rotate_half) OlmoConfigcdeZdZdZdeddffd ZdejdejfdZxZ S) OlmoLayerNormz/LayerNorm but with no learnable weight or bias. hidden_sizereturnNc2t||f|_yN)super__init__normalized_shape)selfr __class__s x/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/olmo/modular_olmo.pyrzOlmoLayerNorm.__init__s !, hidden_statesc|j}tj|jtj |j dddj|S)N)dtypegh㈵>)eps)r#F layer_normtotorchfloat32r)rr! orig_dtypes rforwardzOlmoLayerNorm.forward"sO"(( ||M,,5==,A4CXCXZ^`djnorr   r ) __name__ __module__ __qualname____doc__intrr(Tensorr+ __classcell__rs@rrrs49/C/D/ U\\ ell r rceZdZfdZxZS)OlmoMLPcJt||tj|j|j d|_tj|j|j d|_tj|j |jd|_y)NF)bias) rrnnLinearrintermediate_size gate_projup_proj down_proj)rconfigrs rrzOlmoMLP.__init__*ss  4#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXr )r,r-r.rr2r3s@rr5r5)sYYr r5c |j|j}}|j|}|j|}||zt||zz}||zt||zz} |j|| j|fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r# unsqueezerr') qkcossin position_ids unsqueeze_dimq_typek_typeq_embedk_embeds rapply_rotary_pos_embrK1s|(WWaggFF -- &C -- &C3w;q>C/0G3w;q>C/0G ::f wzz&1 11r c eZdZ d dejdeejejfdeejdeedeejdeejeejeeejff dZ y) OlmoAttentionNr!position_embeddingsattention_maskpast_key_valuecache_positionrc |jdd}g|d|j}|j|} |j|} |j |} |j j | j|j j |j j | j|j j |j j | j|j j |j j | j|jdd} | j|jdd} | j|jdd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|j jdk7rt|j j}||| | | |f|j sdn |j"|j$d|\}}|j&g|dj)}|j+|}||fS) N)minmaxrr )rDrCrQeagerg)dropoutscaling)shapehead_dimq_projk_projv_projr>clip_qkvclamp_view transposerKupdate layer_idxr_attn_implementationrtrainingattention_dropoutrXreshape contiguouso_proj)rr!rNrOrPrQkwargs input_shape hidden_shape query_states key_states value_statesrCrD cache_kwargsattention_interface attn_output attn_weightss rr+zOlmoAttention.forwardNsB$))#2. 88b8$--8 {{=1 [[/ {{=1 ;;   +   T[[%9%9$9t{{?S?S  T   4;;#7#7"7T[[=Q=Q  R   T[[%9%9$9t{{?S?S  T#((6@@AF __\2<2)!. 2) ! 2) !!1!12 2) u||Xell3XeELL>Q5RR S2)r rMc(eZdZdedeffd ZxZS)OlmoDecoderLayerr>rcct|||t|j|_t|j|_t |||_y)N)r>rc)rrrrinput_layernormpost_attention_layernormrM self_attnrr>rcrs rrzOlmoDecoderLayer.__init__sF +,V-?-?@(5f6H6H(I%&f Jr )r,r-r.rr0rr2r3s@rrxrxsKzKcKKr rxceZdZdZy)OlmoRotaryEmbeddingc|jddddfjj|jdddj |j }|dddddfj}t |j jtr/|j jdk7r|j jnd}tj|d5|j|jzjdd}tj||fd }|j|jz}|j|jz} || fcdddS#1swYyxYw) NrrSrmpscpuF) device_typeenabledr )dim)inv_freqfloatexpandrYr'device isinstancetypestrr(autocastracatrCattention_scalingrD) rxrEinv_freq_expandedposition_ids_expandedrfreqsembrCrDs rr+zOlmoRotaryEmbedding.forwards0 MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C &,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C8    s BE22E;N)r,r-r.r+rvr rrrs r rc$eZdZdeffd ZxZS) OlmoModelr>c t||tjt |j Dcgc]}t ||c}|_t|j|_ ycc}wr) rrr8 ModuleListrangenum_hidden_layersrxlayersrrnormr}s rrzOlmoModel.__init__s[  mmBGH`H`Ba bY fi 0 b  "&"4"45  csA1)r,r-r.rrr2r3s@rrrs6z66r rc eZdZy)OlmoForCausalLMN)r,r-r.rvr rrrsr r)rrOlmoPreTrainedModel)Nr)(typingrrr(torch.nnr8torch.nn.functional functionalr%torch.utils.checkpoint cache_utilsrmodeling_utilsrutilsrllama.modeling_llamar r r r rrrrconfiguration_olmor get_loggerr,loggerModulerr5rKrMrxrrr__all__rvr rrs%  5   +   H %  BII  YhY283)N3)lK(K . 6 6 &  r