rhv ddlZddlmZddlmZddlmZddlmZm Z e je Z erddl Z dZ d"deed ed d eed ed effdZ d"deed ed d eed ed effdZ d"deed ed d eed ed effdZ d#ded d d eed ed effdZ d#ded d d eed ed effdZ d#ded d d eed ed effdZeeeeeedZ d$dedededeedeef dZd#dedeefdZd#dedeefdZd#dedeefdZd#dedeefdZd#dedeefdZ d#dedeefd Z!eeeee e!dZ"d#dedeefd!Z#y)%Nwraps)Optional)PretrainedConfig)is_torch_availableloggingcBddtfd}|S)ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. ctj|dz}t|jdr|jj}n|jj }||kDrTt|ds)|j |j||dz\|_}|jd|jdy|jj||_ |jd|jdy) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r original_max_position_embeddings long_inv_freqseq_leninv_freqF persistentN) torchmaxhasattrconfigr max_position_embeddings rope_init_fnr register_bufferoriginal_inv_freqto)self position_idsdevicerr _s s/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_updatez6dynamic_rope_update..longrope_frequency_update+s))L)A- 4;; B C/3{{/[/[ ,/3{{/R/R , 5 541(,(9(9KK1QTU1U):)%"A  T-?-?E R&*%;%;%>%>v%FD "  T-C-CPU Vctj|dz}||jkDrA|j|j||\}|_|j d|d||_||jkrj|j|jkDrP|jj||_|j d|jd|j|_yyy)a dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rrrFrN) rrmax_seq_len_cachedrrattention_scalingroriginal_max_seq_lenrr)rrrrrs r dynamic_frequency_updatez5dynamic_rope_update..dynamic_frequency_update>s ))L)A- T,, ,/3/@/@f^e/@/f ,Hd,  X% H&-D # T.. .43J3JTMfMf3f&*%;%;%>%>v%FD "  T-C-CPU V&*&?&?D # 4g .r"cd|jvr|||jn$|jdk(r|||j|||S)Ndynamic)rlongrope) rope_typer)rxrr'r! rope_forwards r wrapperz$dynamic_rope_update..wrapperQsJ  & $T< I ^^z ) %dL JD!\22r"r)r-r.r'r!s` @@r dynamic_rope_updater/s/W&@& <33 Nr"rrz torch.devicerreturnz torch.Tensorcd|j}t|dr |jnd}t|ddxs|j|j z}t ||z}d}d|tjd|dtjj|tj|z zz }||fS) ax Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). partial_rotary_factor?head_dimNrdtyperr7) rope_thetarr2getattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrrbaser2r4dimattention_factorrs r _compute_default_rope_parametersrD\s$   Dr?rr@) rrrrAr2r4rBrrFrCrs r _compute_dynamic_ntk_parametersrOsJ&   D>6A:NTW[^ab[bTcd dDdu||AsAU[[ILLTZbgbmbmLnqttuvH % %%r"cB|j}t|dr |jnd}t|d|j|j z}t ||z}|jd}|jjd}|jjd} |jjd} d|jvr|jd} |j| z }n |j} dd } |)| r| rt| || | || z }n| |}|jjd xsd } |jjd xsd }dfd}d}|tjd|dj|tj|z z}d|z }d||zz }|jjdd}|| |||| |\}}d ||||dzj|tjz }|d |z z||zz}||fS)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r2r3r4rFrCmscalemscale_all_dimr rcJ|dkryd|ztj|zdzS)Nrr3g?)mathlog)scalerQs r get_mscalez,_compute_yarn_parameters..get_mscales( A:V|dhhuo-33r" beta_fast beta_slowc|tj||dztjzz zdtj|zz S)zPInverse dimension formula to find the dimension based on the number of rotationsr5)rTrUpi) num_rotationsrBrArs r find_correction_dimz5_compute_yarn_parameters..find_correction_dimsBdhh6-!:Kdgg:UVWW\]`d`h`him`n\noor"c||||}||||}|r,tj|x}}tj|}t|dt ||dz fS)z.Find dimension range bounds based on rotationsrr)rTfloorceilrmin) low_rothigh_rotrBrArtruncatelowhighr^s r find_correction_rangez7_compute_yarn_parameters..find_correction_rangesc!'36MN"8S$8OP  3 'C#99T?D3{CcAg...r"c||k(r|dz }tj|tj|z ||z z }tj|dd}|S)NgMbP?r6rr)rr>float32clamp)rbrrB linear_func ramp_funcs r linear_ramp_factorz4_compute_yarn_parameters..linear_ramp_factorsL #: 5LC||Cu}}=Cc R KK Q2 r"rr5r8reT)r)r9rr2r:r;r<r=rGgetrr@rr>r)rrrrAr2r4rBrFrCrQrRr rWrXrZrhrn pos_freqsinv_freq_extrapolationinv_freq_interpolationrerfrginv_freq_extrapolation_factorrr^s @r _compute_yarn_parametersrtsR$   DN *V-@-@@+1+>+>?a+b(//2RR+1+I+I(4  n$Z%?*VUcBd%de )&1 ##'' 4:I##'' 49Ip/aa03363UX[[\I 9_ FY$67""&&z48H%iCGgiqrIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r"c|j}t|dr |jnd}t|d|j|j z}t ||z}|jd}|jd}|jjd} |jjd} t|dr&|j} |j|jz } n |j} | I| dkrd} nAtjd tj| tj| z z} |r,|| kDr'tj|tj | } n&tj|tj | } tj"d |d tj$| j'|z } d| || zzz }|| fS) a~ Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r2r3r4 long_factor short_factorrFrCr rrJrr5)r9rr2r:r;r<r=rGror rrTsqrtrUrrNrjr>r?r@)rrrrAr2r4rBrvrwrFrCr ext_factorsinv_freq_shapers r _compute_longrope_parametersr{"s$   D))*LMO&8'*::$''kH$G[[+;!;X=NPXYN$w.@EUXgEghM]*n= 1, got rGrorrrrKr@rr)rrrGr+rrrFs r (_validate_linear_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M ))+,MM=kZ ( #F ~Z6&3,QRXQYZ[;Gr"c*|j}|jd|jdd}ddh}dh}t|j}t ||||||d}|t |t r|dkrtjd|yy)Nr+rrFr rr3rr)rrrGr+rrrrFs r )_validate_dynamic_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M78M ))+,MM=-]hi ( #F ~Z6&3,QRXQYZ[;Gr"c|j}|jd|jdd}ddh}hd}t|j}t ||||||d}|t |t r|dkrtjd||jd}|-t |t r|d krtjd ||jd } | (t | t stjd | |jd } | (t | t stjd| | xsd| xsdkrtjd| d| dyy)Nr+rrF>rQrerXrZrRrCr rr3rrCrL`rope_scaling`'s attention_factor field must be a float greater than 0, got rXz6`rope_scaling`'s beta_fast field must be a float, got rZz6`rope_scaling`'s beta_slow field must be a float, got rYrzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r) rrrGr+rrrrFrCrXrZs r _validate_yarn_parametersrs&&L  l.>.>vt.LMI (+MM ))+,MM=-]hi ( #F ~Z6&3,QRXQYZ[#''(:;#Z8H%-PTdghThZ[kZl m   -IZ 5%AOPY{[\  -IZ 5%AOPY{[\RIN+]^g]hi66?[@X Z ,r"c|j}|jd|jdd}hd}hd}t|j}t |||||t |dr |j nd}t|d|j|jz}t||z} |jd } t| ts*td | Drtjd | t!| | d zk7r'tjd | d zdt!| |jd} t| ts*td| Drtjd| t!| | d zk7r'tjd| d zdt!| t |drtj#dy|jd} | tjdn-t| t$r| dkrtjd| |jd} | /t| t$r| dkrtjd| yyy)Nr+r>r+rvrw>rFrCr rr2r3r4rwc3HK|]}t|ttfywNrKr=r@.0r,s r z0_validate_longrope_parameters..s1dRS*Qe 2M1d "zC`rope_scaling`'s short_factor field must be a list of numbers, got r5z5`rope_scaling`'s short_factor field must have length z, got rvc3HK|]}t|ttfywrrrs r rz0_validate_longrope_parameters..s0bQRAU|1L0brzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.rFz1Missing required keys in `rope_scaling`: 'factor'rrCgr)rGrorrrrr2r:r;r<r=rKlistallrrlen warning_oncer@)rrrGr+rrrr2r4rBrwrvrFrCs r _validate_longrope_parametersrsI&&L  l.>.>vt.LMI@MVM ))+,MM=-]hi NNN OFE*fsl NNUV\U]^ _'++,>?  '.6:JS:Pbcsbtu;Q (r"c|j}|jd|jdd}hd}t|j}t |||||d}|t |t r|dkrtjd||d}|d }|t |t stjd ||t |t stjd |||krtjd |d ||d} | t | tstjd| | |jk\r&tjd| d|jyy)Nr+r>rFr+r}r~r rrFr3rr}r~z<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rGrorrrrKr@rrr=r) rrrGr+rrrFr}r~r s r _validate_llama3_parametersr&s&&L  l.>.>vt.LMIvM ))+,MM=kZ ( #F ~Z6&3,QRXQYZ["#45O#$67j%&HUVeUfghz2BE'JVWgVhij?* q 5o5F H (44V'W$'/zBbdg7h ^/0 2 (6+I+II u/00MfNlNlMm o Jr"ct|dd}|y|jd|jdd}tj|}| |||ytj d|dy) zO Validate the RoPE config arguments, given a `PretrainedConfig` object rGNr+rrrzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')r:roROPE_VALIDATION_FUNCTIONSrr)rrrGr+ validation_fns r rope_config_validationrUsw6>48L  l.>.>vy.QRI-11)rs 1.   H %;~*.'+!& % && ^ $&c]& >5 ! &>*.'+!& % && ^ $&c]& >5 ! &>*.'+!*& % &*& ^ $*&c]*& >5 ! *&\PTZ& Z&&4Z&?G}Z& >5 !Z&|PT3& 3&&43&?G}3& >5 !3&nPT&, &,&4&,?G}&, >5 !&,Z05. $,( $(!% llllC= l # l:[.>[XVY][ \5E \T\]`Ta \ \6F \U]^aUb \$ &6$ Xc]$ N/*:/RU/d! (8! xPS}! L168 %-)  #3 (3- r"