rh% UddlmZddlZddlZddlZddlZddlmZmZddl Z ddl m Z ddl m Z ddl mZddlmZmZddlmZej(eZe jj/Zed rer dd lmZmZmZmZd2d Zd3d Zd4d5d Z e jBe jDe jFe jHe jJe jLe jNe jPe jRe jTe jVd Z,dZ- d6 d7dZ.dZ/ d8 d9dZ0GddZ1Gdde1Z2Gdde1Z3Gdde1Z4Gdde1Z5Gdde5Z6Gdd e1Z7Gd!d"e7Z8Gd#d$e1Z9Gd%d&e1Z:Gd'd(e1Z;Gd)d*eZ<e<Z=d*e>d+< d:d,Z? d;d-Z@ d2d.ZA d4d/ZBd=2.5`.RANK LOCAL_RANK WORLD_SIZEncclglooxcclhccl)cudacpuxpuhpurCCL_WORKER_COUNTrcclrz2.8T) accept_dev)backendrank world_sizezWe tried to initialize torch.distributed for you, but it failed. Make sure you init torch distributed in your script to use `tp_plan='auto'`.w)r OSErrortorch_C_get_acceleratortypegetattr distributedis_initializedintosenvirongetinit_process_group set_device Exceptioncurrent_devicedevicesysopendevnullstdoutstderrget_world_sizeinit_device_mesh)tp_plantp_size device_typer4r" local_rankr# backend_mapr!eindex tp_devicer6 device_map device_meshs |/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/integrations/tensor_parallel.pyinitialize_tensor_parallelismrH)s&  $U +KLL((++-22KUK0N    + + - rzz&)*DRZZ 56JRZZ 56J#)&PVWK!ook2Ge#BJJNN;Mq,Q(Re#,EeX\,]    0 0tXb 0 c$UK8Ne#))*5e!!#bjj&>"?@/:e/CN ) ) +E [%0I UQY"**c* "**c* J ,g%2C2C2R2R2TG##44Y^^gZPK j+w 66+ Z  s1C8I$$ I>- I99I>ct|tr8t|}||zdk(s Jd|d|||z}|Dcgc]}||z c}S||zdk(s Jd|||z}|g|zScc}w)a Convert block count or proportions to block sizes. This function accepts - The number of blocks (int), in which case the block size is total_size//blocks; or - A list of block sizes (list[int]). In the second case, if sum(blocks) < total_size, the ratios between the block sizes will be preserved. For instance, if blocks is [2, 1, 1] and total_size is 1024, the returned block sizes are [512, 256, 256]. rz Cannot split z in proportional blocks: zPrepacked is not divisible by ) isinstancelistsum) total_sizeblocks total_blocks part_sizeblock single_sizes rG_blocks_to_block_sizesrSas&$6{ L(A-lzlJcdjck/ll-,. /56e E!66F"a'R+I&)RR' F* }v%% 7s A&ctjdd|}||vr||Sd|vr1|jddd|vr|r||jdddSy)a Get the TP style for a parameter from the TP plan. The TP plan is a dictionary that maps parameter names to TP styles. The parameter name can be a generic name with wildcards (e.g. "*.weight") or a specific name (e.g. "layer_1.weight"). The `is_weight` is important because for weights, we want to support `.weights` and `.bias` cases seamlessly! but not parrent classes for `post_init` calls \d+*.rN)resubrsplit)parameter_namer= is_weightgeneric_param_names rG_get_parameter_tp_planr_{sr^<W$)** " "'9'@'@a'H'Kw'V[d)00a8;<< ) BOOLU8I8I16F16BF16I32F32F64I64F8_E4M3c |}|j|}|j}t|d}g} d} |D].} | |z} || z} |dz| z}| t| | z| |zz } | | z } 0|j }d}|dk(s|dk(r$|dj t j}d }|dk(r|| df}n6|dk(s|d k(r |d d | df}n!|dk(s|d k(r|d| f}ntd |d|r|S|j t|S)u When weights are packed (gate_up_proj), we need to make sure each shard gets its correct share. So if you have: gate_proj ( 16, 5120, 8190) and up_proj ( 16, 5120, 8190) packed as gate_up_proj ( 16, 5120, 2 * 8190) And you shard along the last dimension, you need to interleave the gate and up values: Now, if we shard along the last dimension across TP_size (Tensor Parallelism size), we must interleave the values from gate and up projections correctly. Let's take TP_size = 4 for an example: Packed tensor `gate_up_proj` --------------------------------------------------------------- [ G0 G1 G2 G3 | G4 G5 G6 G7 | ... | U0 U1 U2 U3 | U4 U5 U6 U7 | ... ] ↑─────────────↑ ↑─────────────↑ ↑─────────────↑ ↑─────────────↑ Gate Slice 0 Gate Slice 1 Up Slice 0 Up Slice 1 Explanation: - The first half of the tensor (left of the center) holds the gate_proj values. - The second half (right of the center) holds the up_proj values. - For TP=4, we divide each half into 4 slices. In this example, we show two slices for brevity. - Each shard receives one slice from the gate part and the corresponding slice from the up part. For instance: • Shard 0 gets: [ Gate Slice 0, Up Slice 0 ] = [ G0, G1, G2, G3, U0, U1, U2, U3 ] • Shard 1 gets: [ Gate Slice 1, Up Slice 1 ] = [ G4, G5, G6, G7, U4, U5, U6, U7 ] • … and so on. This ensures that each shard receives an equal portion of both gate and up projections, maintaining consistency across tensor parallelism. r)rMrNrrXFrkF8_E5M2.TNzUnsupported dim z", only dim 0, 1 or 2 are supported) shapesizerSrange get_dtypetor&float16 ValueErrorstr_to_torch_dtype)param empty_paramrFr"dimslice_rMr# block_sizestensors_slices block_offset block_sizeshard_block_sizestartstop slice_dtypecastedtensors rGget_packed_weightsrsZ>F""3'J!!#J(JqIKNL!# %3''q,,% u 4lT6IJJ " #""$KFi;)#; . ax+, SBY>3./ SBY^+,+C50RSTT  yy+K899r`c|dk7r td|dk\r|n||jz}|j|}||z}||z}|jd|}|j|dzd} |jg||||| } t |} t |dz} t t | j} | | | | c| | <| | <| j| }|j|}|S)as Reorders a tensor that was reconstructed from sharded packed weights into its canonical packed format. For example, if a weight was packed (e.g., gate_proj and up_proj) and then sharded, DTensor.full_tensor() might produce an interleaved layout like [G0, U0, G1, U1, ...] along the sharded dimension. This function reorders it to [G0, G1, ..., U0, U1, ...]. This is an inverse operation to get_packed_weights. Args: reconstructed_tensor: The tensor reconstructed from DTensor (e.g., via .full_tensor().contiguous()). sharded_dim: The dimension index in the reconstructed_tensor that was originally sharded. world_size: The tensor parallel world size. num_packed_projs: The number of projections that were packed together (e.g., 2 for gate_up_proj). Returns: The reordered tensor in canonical packed format. rzNum blocks different from 2 is not supported yet. This is most likely a bug in your implementation as we only pack gate and up projections together.rNrX) rvndimrpviewlenrKrrpermute reshape_as)packed_parameter sharded_dimr# num_blocksactual_sharded_dimtotal_size_on_sharded_dimoriginal_block_size_on_dimshard_chunk_size prefix_shape suffix_shape tensor_view axis_ws_abs axis_npp_abs permute_ordertensor_permutedfinal_ordered_tensors rGrepack_weightsrsO0Q c  )4q(8kL\LaLa>a 0 6 67I J!:j!H1Z?#))*=+=>L#))*KL>Y[hit[u;M+ l ;)k))=9O+556FG r`cr|j}|dkr||z}||k\rtd|d||j}ttj |}||k\rtd|d|t j|j||z }||z} t| |z|j|} tdg|z} | |j|krt| | | |<|t| St|j} d| |<tjt| tjS)a Generalized tensor sharding across a multi-dimensional device mesh. Extract only the fraction of the parameter owned by the given `rank` when the parameter would have gone sharding at provided `dim`. Extraction follows the pytorch `Shard` placement so that sharding and materializing back to full tensor follows `Shard` semantics. `Shard` follows torch.chunk style sharding of the tensor. We demonstrate some cases below on how sharding happens including some edge cases such as some ranks having an empty tensor as shard. Below implementation is robut to all these cases. Case (1) empty_param (16, 5120, 8190) dim 0 device_mesh.size() 4 rank 0 gets (4, 5120, 8190) (0 ... 4, 5120, 8190) rank 1 gets (4, 5120, 8190) (4 ... 8, 5120, 8190) rank 2 gets (4, 5120, 8190) (8 ... 12, 5120, 8190) rank 3 gets (4, 5120, 8190) (12 ... 16, 5120, 8190) Case (2) empty_param (16, 5120, 8190) dim 0 device_mesh.size() 14 rank 0 gets (2, 5120, 8190) (0 ... 2, 5120, 8190) rank 1 gets (2, 5120, 8190) (2 ... 4, 5120, 8190) rank 2 gets (2, 5120, 8190) (4 ... 6, 5120, 8190) rank 3 gets (2, 5120, 8190) (6 ... 8, 5120, 8190) rank 4 gets (2, 5120, 8190) (8 ... 10, 5120, 8190) rank 5 gets (2, 5120, 8190) (10 ... 12, 5120, 8190) rank 6 gets (2, 5120, 8190) (12 ... 14, 5120, 8190) rank 7 gets (2, 5120, 8190) (14 ... 16, 5120, 8190) rank 8 gets (0, 5120, 8190) rank 9 gets (0, 5120, 8190) rank 10 gets (0, 5120, 8190) rank 11 gets (0, 5120, 8190) rank 12 gets (0, 5120, 8190) rank 13 gets (0, 5120, 8190) Case (3) empty_param (16, 5120, 8190) dim 0 device_mesh.size() 3 rank 0 gets (6, 5120, 8190) (0 ... 6, 5120, 8190) rank 1 gets (6, 5120, 8190) (6 ... 12, 5120, 8190) rank 2 gets (4, 5120, 8190) (12 ... 16, 5120, 8190) In case (2), empty shards are returned with appropriate dimension to allow for operations to work smoothly. Args: param (torch.Tensor): The tensor to shard. empty_param (torch.Tensor): A tensor used for shape reference. device_mesh (torch.Tensor): Shape [d_0, ..., d_n] representing the mesh. rank (int): Global rank of the current process/device. dim (int): Dimension along which to shard the tensor. rzdim z* is out of bounds for tensor of dimension zRank z is out of bounds for mesh size N)dtype)rzrvrproperatormulmathceilminslicetuplerKr&emptyint64) rxryrFr"rz param_dim mesh_shaper# shard_sizerend slice_indices dimensionss rGget_tensor_shardr!s;h!I Qw#o i4u$NykZ[[""J j1J z5&FzlSTT;,,S1J>?J : E ej +"3"3C"8 9C4[MI-M {  %%"5#. cU=)**ekk"JJsO ;;uZ( <z#distribute_module..}s#vWbAcr`c||Srr)rroutputsrF output_fns rGrz#distribute_module..siPSU\^iFjr`)r_forward_pre_hooksregister_forward_pre_hookregister_forward_hook)modulerFrrs ```rGdistribute_modulerqsG 6 $ $%*    , ,-c d   ( ()j k Mr`cBeZdZdZdZedZedZdZddZ y) TensorParallelLayer9 General tensor parallel layer for transformers. Tcyrr input_layoutsdesired_input_layoutsrrrFs rG_prepare_input_fnz%TensorParallelLayer._prepare_input_fns[^r`cyrroutput_layoutsuse_local_outputrrrFs rG_prepare_output_fnz&TensorParallelLayer._prepare_output_fnsY\r`ctr)NotImplementedErrorselfrxry param_typeparam_casting_dtype to_contiguousr"rFs rGpartition_tensorz$TensorParallelLayer.partition_tensors!!r`c |jrat||t|j|j|j t|j |j|jyyr) use_dtensorrrrrrrrrrrrFs rGprepare_module_tpz%TensorParallelLayer.prepare_module_tpsZ    ..0B0BDD^D^_//1D1DdF[F[\   r`Nr nn.Modulereturnr) __name__ __module__ __qualname____doc__r staticmethodrrrrrr`rGrrs4K^^\\"r`rc`eZdZdZdddd d fdZedZedZd dZxZ S) GatherParallelzi Simple class used to define the hooks to add to a layer when we just want to gather the outputs NT)rrrct||xs tf|_||_tf|_||_yr)super__init__rrrrr)rrrr __class__s rGrzGatherParallel.__init__s> +:y{<,&/k^" 0r`c|j|_|r&t|dtr|dj }|SNr) get_groupexpert_parallel_grouprJr to_localrs rGrz GatherParallel._prepare_input_fns:$/$9$9$;! jG4AY'')F r`ct|tjr2tj|tj j d|Stj|dtj j d|S)NF)opasync_opr)rJr&Tensordist all_reduceReduceOpSUMrs rGrz!GatherParallel._prepare_output_fnsV gu|| , OOG (9(9E J OOGAJ4==+<+c 12 $$&E ((** r`c tt||t|jddt|jddyrrrs rGrz"IsolatedParallel.prepare_module_tprr`rr) rrrrrrrrrrr`rGrrs9  r`rcPeZdZdZdddfd ZedZedZdZxZ S)ReplicateParallelz This class is used to replicate computation in a TP layer (used in SP regions when we don't use sequence parallelism for example) T)rrct|tf|_tf|_tf|_||_||_yr)rrrrrrrr)rrrrs rGrzReplicateParallel.__init__sB 'k^({n&/k^" 0&r`cb|d}t|tstj|||d}|S)NrF run_check)rJr from_localrs rGrz#ReplicateParallel._prepare_input_fns4ay ,0"--lKbghLr`cJ|r t|tr|jS|Srrrs rGrz$ReplicateParallel._prepare_output_fns"%5*Wg:Vw!c\ccr`c|dj|}|r|j}tj||t gd}|S)N.Fr )rtrr r rrs rGrz"ReplicateParallel.partition_tensorsFc 12 $$&E""5+ }PUV r` rrrrrrrrrrrs@rGrrsC'+T'ddr`rc`eZdZdZddddd d fdZedZdZedZxZ S) ColwiseParallelrNTrrrrct||xs tf|_|xs t df|_tf|_||_||_yNro) rrrrrrrrrrrrrrrs rGrzColwiseParallel.__init__sR +:y{<-:r<&/k^" 0&r`c|d}t|tstj|||d}||k7r|j|d}|S)NrFr  placementsrrJr r  redistributers rGrz!ColwiseParallel._prepare_input_fn$sVay ,0"--lKbghL 1 1'44@U`e4fLr`c |dk(rt||||d}tdg} ntdg} t||||d}|j|}|r|j}|jr7t j ||| d|j|j}tj||jS)NbiasrornFr rpstride requires_grad) rrrtrrr r rqrr Parameteris_floating_point rrxryrrrr"rF parametershards rGrz ColwiseParallel.partition_tensor1s  ( [$PRSI2YKE2YKE( [$PRSILL!45 !,,.I   **;kFVFVFXalasasauI||IY5P5P5RSSr`c|j|k7r|j|d}|r t|tr|j S|S)NFr)rrrJr rrs rGrz"ColwiseParallel._prepare_output_fnEsG    /**nu*UG%5*Wg:Vw!c\ccr`r) rrrrrrrrrrrs@rGrrsi+/+/!% '( ') '  '  T(ddr`rceZdZdZy)PackedColwiseParallelct||||d}|j|}|r|j}|jr#t j ||t dgd}tj||jS)NrnFr r rrtrrr r rrr!r" rrxryrrrr"rFr$s rGrz&PackedColwiseParallel.partition_tensorOv'uk;bQ LL!45 !,,.I   **9kE"I;Z_`I||IY5P5P5RSSr`Nrrrrrr`rGr(r(N Tr`r(cheZdZdZddddd d fdZdZedZedZd d Z xZ S) RowwiseParallela Partition a compatible nn.Module in a row-wise fashion. Currently supports nn.Linear and nn.Embedding. Users can compose it with ColwiseParallel to achieve the sharding of more complicated modules. (i.e. MLP, Attention) Keyword Args: input_layouts (Placement, optional): The DTensor layout of input tensor for the nn.Module, this is used to annotate the input tensor to become a DTensor. If not specified, we assume the input tensor to be sharded on the last dimension. output_layouts (Placement, optional): The DTensor layout of the output for the nn.Module, this is used to ensure the output of the nn.Module with the user desired layout. If not specified, the output tensor is replicated. use_local_output (bool, optional): Whether to use local :class:`torch.Tensor` instead of :class:`DTensor` for the module output, default: True. Returns: A :class:`ParallelStyle` object that represents Rowwise sharding of the nn.Module. NTrct||xs tdf|_|xs t f|_||_||_yr)rrrrrrrrrs rGrzRowwiseParallel.__init__osF +8uRy:-<> 0&r`c z|dk7rt||||d}tdg} ntg} |dd}|j|}|r|j }|j r7t j||| d|j|j}tj||jS)NrroFrr) rrrrtrrr r rqrrr!r"r#s rGrz RowwiseParallel.partition_tensor}s  ( [$PRSI2YKE[MEaILL!45 !,,.I   **;kFVFVFXalasasauI||IY5P5P5RSSr`ct|dr2|j&|jj|_d|_|d}t |t st j |||d}||k7r|j|d}|S)NrrFr Tr)hasattrrr_biasrJr r rrs rGrz!RowwiseParallel._prepare_input_fns 3 CHH$8))+CICHay ,0"--lKbghL 1 1'44@U`d4eLr`c|j|k7r|j|d}|j}t|dr||jz }|S)NTrr5)rrrr4r5rs rGrz"RowwiseParallel._prepare_output_fnsR    /**nt*TG""$ 3 syy Gr`c d|_|jrt|tjrt df|_nbt|tjrtf|_n7t|tjrt df|_n tdt||t|j|j|j t|j|j |j"yy)NTrozBRowwiseParallel currently only support nn.Linear and nn.Embedding!)_distribute_module_appliedrrJrLinearrr Embeddingrr!rrrrrrrrrs rGrz!RowwiseParallel.prepare_module_tps,0)   &")),EJ2YL*FBLL1.7k^*FBLL1.3Bi\*)*noo ..0B0BDD^D^_//1D1DdF[F[\   r`rr) rrrrrrrrrrrrs@rGr0r0\sl*+/+/!% '( ') '  'T(    r`r0ceZdZdZy)PackedRowwiseParallelct||||d}|j|}|r|j}|jr#t j ||t dgd}tj||jS)NroFr rr*r+s rGrz&PackedRowwiseParallel.partition_tensorr,r`Nr-rr`rGr<r<r.r`r<cTeZdZdZddddd fdZedZedZdZxZ S) SequenceParallelai SequenceParallel replicates a compatible ``nn.Module`` parameters and runs the sharded computation with input sharded on the sequence dimension. This currently supports ``nn.LayerNorm``, ``nn.Dropout``, and the `RMSNorm python implementation `__ This style implements the operation that is described in the paper `Reducing Activation Recomputation in Large Transformer Models `__ If the input passed in to this ``nn.Module`` is a :class:`torch.Tensor`, it assumes that the input is already sharded on the sequence dimension and converts the input to a :class:`DTensor` sharded on the sequence dimension. If the input passed in to this ``nn.Module`` is already a :class:`DTensor` but is not sharded on the sequence dimension, it would redistribute the input to be sharded on the sequence dimension. The output of the ``nn.Module`` will be sharded on the sequence dimension. Keyword Args: sequence_dim (int, optional): The sequence dimension of the input tensor for the ``nn.Module``, this is used to annotate the input tensor to become a DTensor that is sharded on the sequence dimension, default: 1. use_local_output (bool, optional): Whether to use local :class:`torch.Tensor` instead of :class:`DTensor` for the module output, default: False. Returns: A :class:`ParallelStyle` object that represents Sequence Parallel of the ``nn.Module``. Example:: >>> # xdoctest: +SKIP(failing) >>> from torch.distributed.tensor.parallel import parallelize_module, SequenceParallel >>> from torch.distributed.device_mesh import init_device_mesh >>> ... >>> m = Model(...) # m is a nn.Module that contains a "norm" nn.LayerNorm submodule >>> tp_mesh = init_device_mesh("cuda", (8,)) >>> >>> # By default, the input of the "norm" will be converted to DTensor that shards on the sequence dim >>> # and the output of "norm" will return a sharded on sequence dimension :class:`DTensor`. >>> >>> sharded_mod = parallelize_module(m, tp_mesh, {"norm": SequenceParallel()}), >>> ... .. note:: SequenceParallel style assumes ones initialization if there are weights in the nn.Module (i.e. ``nn.LayerNorm`` or ``RMSNorm``, and they by default have ones initialization). If you have custom inits for the weights on those modules, you need to broadcast the weights before/after parallelizing to ensure that they are replicated. rXF) sequence_dimrrct|tf|_t df|_tf|_||_d|_t |f|_ ||_y)NrXT) rrrrrrrrrsequence_sharding)rr@rrrs rGrzSequenceParallel.__init__s\ 'k^&+Ah["({n 0"' "5!7 0r`c|d}t|tstj|||d}||k7r|j|d}|S)NrFr Trrrs rGrz"SequenceParallel._prepare_input_fnsRay ,0"--lKbghL 1 1'44@U`d4eLr`cZ|jtfd}|jS)NTr)rrrrs rGrz#SequenceParallel._prepare_output_fns1&&! ~' !!r`c|d}|j|}|r|j}|jr"tj||t gd}t j||jS)N.Fr r) rtrrr r rrr!r"r+s rGrz!SequenceParallel.partition_tensorsk#J LL!45 !,,.I   **9kIK=\abI||IY5P5P5RSSr`)r@r-rrrrs@rGr?r?sF*X/0%]b1"" Tr`r?c(eZdZdZfdZdZxZS)GroupedGemmParallelzb Applies Expert Parallelism to MoE experts by loading the correct experts on each device. c0t|d|_yNF)rrr)rrs rGrzGroupedGemmParallel.__init__*s  r`c|}|jd} | |jzdk7r td| d|jd| |jz} ||| z|dz| zj|}|r|j }|S)NrzAGlobal number of experts must be divisible by number of devices: z % z != 0rX)rprqrvrtr) rrxryrrrr"rFep_rankglobal_num_expertslocal_num_expertss rGrz$GroupedGemmParallel.partition_tensor.s(..q1  0 0 2 2a 7STfSggjkvk{k{k}j~DE /+2B2B2DDg 11Wq[DU4UVYYZmn $$&E r`)rrrrrrrrs@rGrGrG%s! r`rGcDeZdZdZdZedZedZdZddZ y) RouterParallelzQ Allows to reshape the router scores to support running expert parallel. c.||_||_d|_yrI)argskwargsr)rrQrRs rGrzRouterParallel.__init__As   r`cF|d}t|tr td|S)Nrz5RouterParallel does not support DTensor input for now)rJr rrs rGrz RouterParallel._prepare_input_fnFs'ay lG ,%&]^ ^r`c|j|j}}|j|z}|\}} |dd||z|dz|zf}| j| |z|k7d} | |z} || fS)aW Imagine if you had 4 tokens, top_k = 4, and 128experts. With EP = 8. Imagine router_indices being: [ 52, 42, 119, 67], [102, 89, 61, 40], [ 82, 103, 4, 34], [ 93, 23, 109, 11], then you can map which rank should be getting which values [3, 2, 7, 4], [6, 5, 3, 2], [5, 6, 0, 2], [5, 1, 6, 0], Thus for say rank 0, you fill with 0 the index tensor [ 0, 0, 0, 0], [ 0, 0, 0, 0], [ 0, 0, 4, 0], [ 0, 0, 0, 11], This works well. For another rank you need to make sure you round to num_local_expert because the next operation will one hot encode the router index vector. This allows us to know directly which local expert is hit. Similarly the scores are indexed with something created form router_indices. The kinda naive training loop that we use for device_map "auto" uses a similar logic. Here we are just making each rank believe that he is alone, and he computes his part of the hiddenstates. NrXr)get_local_rankrq num_experts masked_fill) rrrrrFrKep_sizenum_local_experts router_scoresrouter_indicess rGrz!RouterParallel._prepare_output_fnMsF'5579I9I9KOOw6(/% ~%a3D)DRS WhGh)h&hi '33^GX5X]d4dfgh'*;;n,,r`cR|dj|}|r|j}|Sr)rtrrs rGrzRouterParallel.partition_tensorxs+c 12 $$&E r`c tt||t|jddt|jddyrrrs rGrz RouterParallel.prepare_module_tps4   D**D$ 7 D++T4 8  r`Nr) rrrrrrrrrrrr`rGrOrO<s?!  (-(-T r`rOceZdZedrherfeeeeeeededee e de e e ed ZyiZy)ParallelInterfacer )r)rF)r) colwiserowwise colwise_rep rowwise_rep local_colwise local_rowwiselocalgatherlocal_packed_rowwisesequence_parallel replicate grouped_gemm ep_routerN)rrrr _torch_distributed_availablerr0rrrr<r?rrGrO_global_mappingrr`rGr_r_s& %U +0L'(&(*)+F*E,?,?%'$&$9e$L!1!3*,/1') "#r`r_ALL_PARALLEL_STYLEScFd|vr|jddn|\}}t||}|s|S|dvr|S|dk(r tdg}nF|dk(r|dk(r tg}n0tdg}n#|dk(r|dk(r tdg}n td g}t j ||d S) z Converts a local variant of weights to a DTensor with corresponding placements. Shouldn't be done ever except of before saving the model. rWrX)rhrerdrhrorerrdrnFr )r[r_rrr r )r$r\rFr=_rtp_stylers rGconvert_local_tensor_to_dtensorrss 69N5JN))#q1P^MAz%ng>H QQ))Bi[ _ $  #+J)J _ $  )J)J   ijE RRr`c|jD]B\}}t|tjs!t|tr2t ||||||<D|S)z Replaces all tensors that were sharded with `local_*` strategy with DTensor to make determining their proper size possible. )itemsrJr&rr rs) state_dictr=rFkeyvalues rG%replace_state_dict_local_with_dtensorrysV!&&(` U eU\\ *:eW3M=eS+W^_JsO` r`c /t} |j|_fd_yy#t$r}td|dd|Yd}~6d}~wwxYw)ai This function is called in `PretrainedModel.post_init()`. It is responsible of adding hooks to the modules of the `model`, based on the `PretrainedModel._tp_plan`. This is the place where we add the `pre_forward` and `post_forwards` hooks. These are defined for each `TensorParallelLayer` as `_prepare_input_fn` and `_prepare_output_fn`. NTrying to prepare 0, but it's not supported. Corresponding module: z Fix it's TP plan: c.jdS)Nz TP Plan: )__repr__)current_module_planrsrGrz5add_tensor_parallel_hooks_to_module..sV__%6$7{CVBW"Xr`)rorrprint _hf_tp_planr~) modelrr= layer_namerrFr\tp_layerrBs ` ` rG#add_tensor_parallel_hooks_to_modulers|&&':;   & &v{ ; 1X'#  $ZL0`ag`hh{|}{~   s5 AAAc $d|vr|jddn|\} } |jxsi} | jtt |ddxsi|j | } t |}t|| } tjdk(r7| tjd| dntjd| d| | " t| }|j||| ||||}n|ddj!|}t#|t$j&j(s/t$j&j)||j+ }t-| | ||S#t$r!}td |d | d d |Yd}~d}~wwxYw)a This function is called in `from_pretrained` when loading a model's checkpoints. It receives the pointer to the parameter (or the parameter itself) and takes care of "sharding". All process run this function, so they just load the partition of the tensor that they require. Main uses cases: - column / rowise parallelism, you just shard all the weights of the layer (weight and bias) - packed layers: you slice the weights, then shard like above - custom operation: - you want to add an all-gather at the end of a local layer. - you want to have a layer that is isolated from the rest of the world (because torch.DTensor does not work well with `.view` for instance) rWrX_tp_planNrzTensor sharding plan for z+ not found, using default 'replicate' plan.z: r{r|z" Fix it's TP plan, current layer: z : r)r[rupdater*r) get_submoduler-r_rget_rankloggerinfororrrrtrJr&rr!r"setattr)rrxryr\r is_contiguousr"rF set_param param_namerr= module_to_tpcurrent_shard_planrrBs rGshard_and_distribute_modulers ?B^>S^223:YgJ nn"G NN74; D9?R@&&z2L t9D/H }}!  % KK3JH--{J0C]TXZeEa /0 eUXX// 0""5 8U8U8W"X L*e, L#  $^$44deqdrsUV^U__bcdbef   s E%% F.F  Fcn|y|Dchc]}tjdd|}}t|}|}|D]}d|vr|jdddn|}tjdd|}||vr#|j ||j |\d|vr<|jdddx}|vr#|j ||j |t |dkDrtjd|t |dkDr(tjdd j|yycc}w) z Verify the TP plan of the model, log a warning if the layers that were not sharded and the rules that were not applied. NrUrVrWrXrz>The following TP rules were not applied on any of the layers: z'The following layers were not sharded: z, ) rYrZsetr[popdiscardrrwarningjoin) expected_keysr=rw generic_keysunsharded_layers unused_rulesrr^parent_param_names rGverify_tp_planrsJ 8EFBFF63,FLF<(L .1SjSZZQ'*c VVFC<  (   / 0  $ $S ) & &ASAZAZ[^`aAbcdAe,e,=jq+q   . /  $ $S )   <1WXdWefg q @K[A\@]^_!'GsD2c zd}t|ddxsij}t|jdj|_|jj |||_||_|gt|trtj|}|jr6d}t|jd|jj|_|jD]\}}t||t|ddx}rP|jj |jjD cic] \} } |d| | c} } t|ds~t|jd|i}|ik(rt|jdi}|jj |jjD cic] \} } |d| | c} } |jtd rt r|jj#D]} | t$vs t'd | d t$|j)D]O\}}t|d d s6ddlm} t/||jd }| |||jd||d|_Q|Scc} } wcc} } w)Nrbase_model_tp_plan_ep_planbase_model_ep_planr=rWconfig base_modelr z"Unsupported tensor parallel style z. Supported styles are _is_hookedFr)r)r\r=r])rrr=rrrFT)r*copyrrr_tp_size _device_meshrJdictr from_dictenable_expert_parallelnamed_childrenrur4r rmvaluesrorv named_modules)transformers.integrations.tensor_parallelrr_r) rdistributed_configrFr>_planr=namerplankvrs rGdistribute_modelr9s Euj$/52;;=GU\\+?@EEGEN NN'"EN$E% ($ /!2!CDrzv}}.BBG NN ! ! @Q@Q@S"T1dV1QC=!#3"T UV ~~!&?&FKg&&( wA++ #EaSH_`s_t!uvv w"//1 %LD&6<7i-T5>>ejk3!!NN!(, + !%F  % L3#U #Us J1 J7 r)rMr-rNzint | list[int]rz list[int])T)r\strr=dict[str, str]rz str | None)r) r torch.Tensorrr-r#r-rr-rrrr)r$rr\rr=rrr )rvdict[str, torch.Tensor]r=rrr)rz list[str]r=zdict[str, str] | None)E __future__rrrr.rY functoolsrrr&torch.distributedr+rrrutilsr r utils.genericr get_loggerrr is_availablermtorch.distributed.tensorr rrrrHrSr_ruint8int8int16rubfloat16int32float32float64r float8_e4m3fnrwrrrrrrrrrr(r0r<r?rGrOr_ro__annotations__rsryrrrrrr`rGrsf#  % +6,   H % %00==?U#(DMM57p&4& JJ ++ ** ;; == NN ;; == == ;;"" A:P > "> > >  >  > BM=d     $8' (' T! *! H + F;d);d| TO Td)dN TO TQT*QTh-.J (J Z(0*;)<&<SS-0SHVS S: '    Z^Y2qu0f`>*r`