rh(DddlmZmZddlZddlmZddlmZddlmZddl m Z ddl m Z d d l mZmZmZmZd d lmZe j(eZGd dedZGddej0ZGddeZGddeZGddeZGddeZgdZy))Optional TypedDictN)nn)ACT2FN)Cache)Unpack)logging)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigceZdZUdZej ed<ej ed<eed<eed<ejed<y)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensor/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyrr%s7"###### __r#rF)totalc`eZdZdZdeffd ZdejdejfdZxZ S)GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configc`t||j|_|j|_t |j |_tj|j|jdzd|_ tj|j|jd|_ y)Nr F)bias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr( __class__s r$r,zGraniteMoeSharedMLP.__init__Gs  ,,!:: !2!23IIdoot7G7G!7KRWXYYt'7'7uUr# hidden_statesreturnc|j|}|jdd}|j|d|dz}|j|}|S)Nr )dimrr)r3chunkr1r4)r6r8chunked_hidden_statess r$forwardzGraniteMoeSharedMLP.forwardPs^))-8 - 3 3A2 3 >(=a(@ADYZ[D\\ **=9 r#) rrrrrr,rTensorr? __classcell__r7s@r$r'r'>s2V5VU\\ellr#r'c|eZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee d ee ejejfd e ede ejee ejejfffdZxZS)GraniteMoeSharedDecoderLayerr( layer_idxctt||||jdk(rd|_yt||_y)Nr)r+r,r/r' shared_mlpr6r(rEr7s r$r,z%GraniteMoeSharedDecoderLayer.__init__Ys3 +"("A"AQ"F$L_`fLgr#r8attention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingskwargsr9c l|} |j|}|jd|||||||| d| \}} | ||jzz}|} |j|}|j |\} }|j | }n| |j |z}~ | ||jzz}|f}|r|| fz }|r||fz }|S)aC Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )r8rIrJrKrLrMrNrPr")input_layernorm self_attnresidual_multiplierpost_attention_layernormblock_sparse_moerG)r6r8rIrJrKrLrMrNrOrPrQresidualself_attn_weightsmoe_hidden_states router_logitsoutputss r$r?z$GraniteMoeSharedDecoderLayer.forward]sL!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=43K3K#KK !55mD +/+@+@+O(= ?? "-M- 0NNM  =43K3K#KK  "  )+ +G   ' 'Gr#)NNNFFNFN)rrrrr r,rr@rrrbooltupler r FloatTensorr?rArBs@r$rDrDXs,h5h#h2637*.,1$)59/4KOO||O!.Ou//0 O ! O $D> OD>O!!1!12O'tnO&eELL%,,,F&GHO45O u  (51B1BEDUDU1U+V"WW XOr#rDceZdZUeed<dgZy)GraniteMoeSharedPreTrainedModelr(rDN)rrrrr_no_split_modulesr"r#r$raras ""78r#rac$eZdZdeffd ZxZS)GraniteMoeSharedModelr(c t||tjt |j Dcgc]}t ||c}|_ycc}wN)r+r,r ModuleListrangenum_hidden_layersrDlayersrHs r$r,zGraniteMoeSharedModel.__init__sE  mmNSTZTlTlNm n )&) < n  nsA)rrrrr,rArBs@r$rdrds 5  r#rdc*eZdZdgZdeffd ZxZS)GraniteMoeSharedForCausalLMzlm_head.weightr(cdt||t||_|j yrf)r+r,rdmodel post_initr5s r$r,z$GraniteMoeSharedForCausalLM.__init__s&  *62  r#)rrr_tied_weights_keysrr,rArBs@r$rlrls*+5r#rl)rlrdra)typingrrrr activationsr cache_utilsrprocessing_utilsr utilsr granitemoe.modeling_granitemoer r rrconfiguration_granitemoesharedr get_loggerrloggerrModuler'rDrardrl__all__r"r#r$r|s ' ! & C   H %)52"))4T#9Tn9&?9  O "7 fr#