rhrXddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZmZd d lmZmZmZmZd d lmZd dlmZd dlmZej8eZdZGddeZ Gdde Z!GddeZ"GddeZ#GddeZ$GddeZ%gdZ&y))OptionalUnionN)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack)TransformersKwargslogging) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-0414c eZdZy)Glm4MLPN__name__ __module__ __qualname__x/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/glm4/modular_glm4.pyrr%rrcdeZdZdedeffd Z ddejdeejdeejdee dee d eejd ee ejejfd e ed e ejee ejejfffd ZxZS)Glm4DecoderLayerconfig layer_idxct||j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ y)N)r"r#)eps) super__init__ hidden_size Glm4Attention self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfr"r# __class__s rr'zGlm4DecoderLayer.__init__*s !--&f J6?*6+=+=6CVCVW(3F4F4FFL_L_(`%(3F4F4FFL_L_(`%"-f.@.@fFYFY"Zr hidden_statesattention_mask position_idspast_key_value use_cachecache_positionposition_embeddingskwargsreturnc |} |j|}|jd|||||||d|\}} |j|}| |z}|} |j|}|j |}|j |}| |z}|S)N)r3r4r5r6r7r8r9r)r-r*r/r.r+r0) r1r3r4r5r6r7r8r9r:residual_s rforwardzGlm4DecoderLayer.forward5s!,,]; )4>>  ')%)) 3     q55mD  =0  55mD / // >  =0 r)NNNFNN)rrrrintr'torchTensorr LongTensorrbooltupler r FloatTensorr? __classcell__r2s@rr!r!)s [z [c [2637*.$)59KO!||!!.!u//0 ! ! ! D> !!!1!12!&eELL%,,,F&GH!-.! u  (51B1BEDUDU1U+V"WW X!rr!c eZdZy)r)Nrrrrr)r)Yrrr)c8eZdZdeedeeefffd ZxZ S)Glm4ForCausalLM super_kwargsr;c "t|di|S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```r)r&r?)r1rLr2s rr?zGlm4ForCausalLM.forward^s4w...r) rrrr r rrEr r?rGrHs@rrKrK]s0/12/ u,, -//rrKc eZdZy)Glm4ForSequenceClassificationNrrrrrOrO{rrrOc eZdZy)Glm4ForTokenClassificationNrrrrrQrQrrrQ)Glm4PreTrainedModel Glm4ModelrKrOrQ)'typingrrrA cache_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsr processing_utilsr utilsr r glm.modeling_glmrrrrphi3.modeling_phi3rconfiguration_glm4r modeling_glm4r get_loggerrlogger_CHECKPOINT_FOR_DOCrr!r)rKrOrQ__all__rrrrcs # B96&0tt(*&   H %+ g -1-` L /n/< $@  !:  r