rh?hddlmZddlmZddlmZmZddlZddlm cm Z ddl m cm cmZddlmZddlmZddlmZddlmZmZmZmZmZd ed eed eeeeffd Zd eded eed eeedfeedfffdZd ed edee ed eed eeedfeedfff dZ!dejDded eed ee ee effdZ#dejHded eed ejHfdZ%dejLjNdee(d efdZ)deded eed eedffdZ*d ejHfdZ+y)) defaultdict)Sequence)castOptionalN) ShapeType) DeviceMesh) DTensorSpec) _StridedShardPartial Placement ReplicateShard mesh_shape placementsreturnc tt|t|k(s$tdt|dt|dg}tt}t }t |D]R\}}t |tr!||jj||f8|j||ft |ts\|j|vrtd|d|d|d|j|vs|j|j|j|j}||}t|dkDs|j\} } | j|k(std | jd |d ||| z}|j| t|jft|dkDr{U|S) a$ Replace Strided Shards with regular shards in an adjusted order. Returns a list of (mesh_dim, placement) tuples where the list order is the sharding order. ex. [Shard(0), _StridedShard(0, split_factor=2), Shard(0)] -> [(0, Shard(0)), (2, Shard(0)), (1, Shard(0))] /Expected one placement per mesh dim, but found  placements and mesh dims.zTStrided sharding does not allow Shard() to appear after the strided part has ended. z at mesh dim z in z violates this assumption.rz@Can only convert _StridedShard to ordered Shard if split_factor(z) == aggregate mesh size ())len RuntimeErrorrlistset enumerate isinstancer dimappendrNotImplementedErroraddpop split_factor) rrordereddeferred_strided_placementsstrided_part_ended_for_dimmesh_dimpstrided_placementsaggregate_sizestrided_mesh_dimstrideds r/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/torch/distributed/tensor/_utils.py_explicit_order_placementsr-s z?c*o - Z))9#j/9J+ W  G"-d"3!$ ,I ! a ' ' . 5 5xm D NNHa= )!U#5566-778cxjPT%,&@B 5577.221559)D)H)H)O&%/%9N01A54F4J4J4L1('&33~E"."bcjcwcwbxy<mA&6- :R&R'4Q'7-'GM!$ HI 7';<i0AQR R3S4  &&r.r3ct|dk7r tdt||jk7r%tdt|d|jdt |dt r|St |dt r*tjt|}t|jDcgc]#}tj||j%}}tj||d}|dj }t|jDcgc] }||k7s | } }|D]A} tj"|| | | s td| j%} || |z }Ct|} || |dj <tj&| Std t)|dd cc}wcc}w) a  Compute the global size of a DTensor from the given local tensor shape, the mesh and placements. Different from `compute_global_tensor_info`, which assumes sharding is even, this util allgathers local shards' shapes from all ranks and thus can support uneven sharding. NOTE: Currently this function only supports 1D mesh. Args: shape (:class:`torch.Size`): Shape of the local tensor mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. placements (Sequence[:class:`Placement`]]): The attribute of the DTensor that describes its layout on the mesh topology. Return: tensor_shape: Shape of the global DTensor. z>compute_global_tensor_shape only supports 1 placement for now.rrrr)devicez?Non-sharded dimentions should have identical size across ranks.zPlacement type z not supported.)rrrMrrr rtorchrFrrNrI empty_likerYfuncolall_gather_inplacerequaltolistSizerO) r3r0rr>_gathered_shaped_tensorssharded_dim_sumrBd other_dims shape_tensorshape_tensor_listr/s r,compute_global_tensor_shaperhs. :!! L   :$))# Z))9$))K Q  *Q-+ JqM5 )ll4;/ 499;'#    [1C1C D# #  !!"9;GqM%% !&tyy!1DAQ)^aD D3 |D]}t|tjtfr|jcSt|t t fsHt|dkDsWt|dtjtfs{|djcStd|d)z Find the device mesh object from args. It returns None if no mesh is found. NOTE: we can optimize this search if needed rz+Cannot find device mesh from args for op : rH) rdtensorDTensorr device_meshrr<r ValueError)rirjargs r,try_find_mesh_from_argsrq8s& cGOO[9 :?? " sT5M *C1 3q6GOO[#ABq6%% %& B7)1M NNr. global_stridecpdgtzt|D]q\}}|jstt|j }t tD]*}||kDs|xx|j|zcc<,stfdt tDS)z Compute the stride of a local tensor shard, given the global stride of the DTensor. NOTE: Currently this function is assuming the DTensor is evenly shardable. rXc34K|]}||zyw)Nr8).0rUrrstride_divisorss r, z'compute_local_stride..^s%34 aOA..s) rrrKrrrrNrIr<)rrr0rmesh_idxr'rUjrvs` @r,compute_local_striderzMscC ..O ,> ! ::<UA""A3}-. > #mA&66#A&$))H*==& > > 8=c->P8Q r.ct|tjr|St|tr|g}n;t |dk(r"t|dt rt |d}n t |}tj|S)z Unify variable types of size argument to torch.Size Acceptable types include: int, Sequence[int], Tuple[int], Tuple[Sequence[int]], or torch.Size rXr)rrZr`intrrr)rI torch_sizes r,normalize_to_torch_sizer~cse$ # $V TaJtAw9$q'] $Z ::j !!r.), collectionsrcollections.abcrtypingrrrZ)torch.distributed._functional_collectives distributed_functional_collectivesr\torch.distributed.tensor._apirF_apirltorch._prims_commonrtorch.distributed.device_meshr&torch.distributed.tensor._dtensor_specr (torch.distributed.tensor.placement_typesr r r r rr<r|r-r5rr2TensorrVr`rh_ops OpOverloadobjectrqrzr~r8r.r,rs#$! :://)4>22'/ ':2 eCN#$2j++#-+;CI;N+ 5c?E#s(O +,+^D8D8D8DI&D8# D8  5c?E#s(O +, D8N8' LL8' *8'8@8K8' 49d3i  8'v; ::; '; 5=i5H;  ZZ; |O ZZ " "O*26*:OO*$.