^i; SSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK r SSK J r SSKJr SSKJr SSKJr SSKJr SSKJr "S S 5r"S S \5rS"S \ R.\R0\R2S\ R4\ R6\\4S\S\ R<\R0\R24SjjrSr "SS5r!S\S\S\ RD\#\ RH\44Sjr%SS\%4S \ R.\R0\R2S\S\&S\ RNSS4 Sjjr(\RRS#S\ R.\#S\ R<S4Sjj5r*"S S!5r+g)$N)Path)core)JobEnvironment)CommandFunctionDelayedSubmission)environment_variablescj^\rSrSrSrU4SjrS\RS\RS\4Sjr Sr U=r $) CheckpointableaDerived callable classes are requeued after timeout with their current state dumped at checkpoint. __call__ method must be implemented to make your class a callable. Note ---- The following implementation of the checkpoint method resubmits the full current state of the callable (self) with the initial argument. You may want to replace the method to curate the state (dump a neural network to a standard format and remove it from the state so that not to pickle it) and change/remove the initial parameters. cp>[TU]U5n[U5(dSURS35eU$)NzClass z^ is marked as Checkpointable but doesn't have a __call__ method. Please add a __call__ method.)super__new__callable__name__)clsargskwargsinstance __class__s M/mnt/rpi/tmp/demucs-venv-sys/lib/python3.13/site-packages/submitit/helpers.pyrCheckpointable.__new__,sM7?3'    A CLL>! A A rrreturnc [U/UQ70UD6$)z3Resubmits the same callable with the same argumentsr)selfrrs r checkpointCheckpointable.checkpoint3s!7777r) r __module__ __qualname____firstlineno____doc__rtpAnyr r__static_attributes__ __classcell__)rs@rr r s5 88"&&8=N88rr c\rSrSrSrSS\SS4SjjrS\RS\R4S \RS \RSS4S jr S\ 4S jr S\R\4S jrS\R \R4SjrSrg)FunctionSequence:aThis is for gathering several estimations into one function, which will return the sequence of outputs. Also this "function" is stateful, hence it can be stopped, and recovered, which is useful when job can be preempted. Usage ----- func = FunctionSequence() func.add(my_function1, arg1, kwarg1=value_kwarg1) func.add(my_function2, arg1, arg2) result1, result2 = func() Note ---- This function is checkpointable because: - it derives from Checkpointable - it keeps DelayedSubmission objects as attribute, which in turn store the results of the computation in memory once they are computed. So at checkpoint time, those results will be saved, and only the non-computed results will be computed once the job restarts. verboserNcXl/UlgN)r,delayed_functions)rr,s r__init__FunctionSequence.__init__Qs =?rfunc.rrcTURR[U/UQ70UD65 gr.)r/appendr )rr2rrs raddFunctionSequence.addUs& %%&7&Nt&Nv&NOrc,[UR5$r.)lenr/rs r__len__FunctionSequence.__len__Xs4))**rc,[UR5$r.)iterr/r9s r__iter__FunctionSequence.__iter__[sD**++rcUR(a5[SU55n[SUS[UR53SS9 URVs/sHo"R 5PM sn$s snf)Nc3@# UHoR5v M g7fr.)done).0fs r ,FunctionSequence.__call__..`s.AvvxxszStarting from /Tflush)r,sumprintr8r/result)rrBrDs r__call__FunctionSequence.__call__^sh <<...D N4&#d.D.D*E)FGt T $ 6 6 61HHJ 6   sA1)r/r,F)rr!r"r#r$boolr0r%Callabler&r5intr:Iteratorr r>ListrMr'r rrr*r*:s,@@@P CK0PP266PVZP++,"++&78, "''"&&/ rr*jobstimeoutpoll_frequencyrc## [R"5n[5nUb"[R"5U- U:a[e[U5H8upVXT;aM UR 5(dM#UR U5 Uv M: [ U5[ U5:Xag[R"U5 M7f)u Yields jobs as they complete (finished, failed or were cancelled). Raises a TimeoutError if the result isn’t available after timeout seconds. timeout can be an int or float. If timeout is not specified or None, there is no limit to the wait time. Parameters ---------- jobs: list Jobs instances timeout: int/float Maximum time (in sec) to wait for jobs completion poll_frequency: float Frequency in second at which we check job status. Yields ------ Job The next completed job N)timeset TimeoutError enumeraterBr5r8sleep)rUrVrWstart jobs_doneijobs r as_completedrbgs6 IIKE UI   499;#6#@ oFA~xxzz a  & y>SY &  >" s A0C6A Cc j[R"U40UD6RS5R5$)Nzutf-8) subprocess check_outputdecodestrip)str_argsrs rrun_cmdris,  " "8 6v 6 = =g F L L NNrc \rSrSrSrSS\S\R\S\S\R\ S\R\ 4 S jjr \ SS \S \4S jj5r SS jrSrSrg) RsyncSnapshotaTakes a snapshot of the git repository that the script lives in. This ensures that remote jobs always use the code from when they are scheduled and not the code from when they are launched / re-started. Parameters ---------- snapshot_dir: Path A path to where the snapshot should be created with_submodules: bool Whether or not submodules should be included in the snapshot exclude: Sequence[str] An optional list of patterns to exclude from the snapshot include: Sequence[str] A list of relative file names to include from the snapshot. Useful for .so or other build artifacts that are genarally not tracked by git. Note ---- - Only files that are checked in to the repository are included in the snapshot. If you have experimental code that you would like to include in the snapshot, you'll need to `git add` the file first for it to be included, or use `include` arg. N snapshot_dirroot_dirwith_submodulesexcludeincludecURSS9 [U5UlU=(d [/SQ5Ul[R "5UlX0lX@lXPl g)NTthrow)gitz rev-parsez--show-toplevel) availablerrmrirncwd original_dirrorprq)rrmrnrorprqs rr0RsyncSnapshot.__init__sO T" . TG,S$T  HHJ.  rrtrc`[R"S5(dU(a [S5egg)Nrsyncz-RsyncSnapshot requires rsync to be installed.FT)shutilwhich RuntimeErrorrss rrvRsyncSnapshot.availables&||G$$"#RSSrc [R"5Ul[UR5nUR (aSOSnUR R5(dUUR RRSSS9 [R"SSSSU3[UR 5/5 [R"5n[S US UR3USS 9 [!["R$R'S UR(555n[+URS SS9nUR,H n[/XeS9 M SSS5 [SSSURU[UR 5/U-5 SSS5 [0R2"UR 5 g!,(df  Nh=f!,(df  N@=f)Nz--recurse-submodulesz-sT)parentsexist_okruclonez --depth=2zfile://z git ls-files z | grep -v ^16 | cut -f2- > )rwshellc3*# UH nSU4v M g7f)z --excludeNr )rCpats rrE*RsyncSnapshot.__enter__..s8dWcPS+s9KWcsautf8)encoding)filer{z-az --files-from)rrwrxstrrnrormexistsparentmkdirrd check_calltempfileNamedTemporaryFilerinamelist itertoolschain from_iterablerpopenrqrKoschdir)rrnsubtfilerpoincs r __enter__RsyncSnapshot.__enter__si HHJt}}%(,(<(<$$  ''))    $ $ * *4$ * G  ! !5';'(@TVYZ^ZkZkVl"m n ( ( *e mC5(DUZZLQW_gk l9??888dW[WcWc8ddeGejj#71<= 30s) to avoid overloading squeuerzThere are no jobs to monitorNz, c3r# UH-n[UR5RSS5Sv M/ g7f)_rrN)rjob_idsplit)rCras rrEmonitor_jobs..s,%WRV3c#**o&;&;C&CA&FRVs57z Monitoring z jobs from job arrays z Tforce)moderrz%All jobs finished, jobs with indices z failedrHz Whole process is finished, took rz minutes)r8rKjoinrrZrYget_info collections defaultdictr\stateupperr5rBr]rR) rUrWrrr job_arraysrrr`rars r monitor_jobsrsz& #g%gg# YF { ,-6#%WRV%W"WXYJ Kx5j\ EF IIK  G  '  * ,,S1 oFA yy( ) - -a 0xxzz6"&&q)& $Jx$89 z&! "c$i / 9:L9MWU]a b ,jA >" " ,S$))+@U2UY[1[-\,]]e fgr extra_namesc#j# Sn[RVs0sHEnURS5(d X!;dX ;dM%U[RRU5_MG nnSv [RR U5 gs snf![RR U5 f=f7f)a`Removes slurm and submitit related environment variables so as to avoid interferences when submiting a new job from a job. Parameters ---------- extra_names: Sequence[str] Additional environment variables to hide inside the context, e.g. TRITON_CACHE_DIR and TORCHINDUCTOR_CACHE_DIR when using torch.compile. Note ---- A slurm job submitted from within a slurm job inherits some of its attributes, which may be confusing a cause weird gres errors (or pytorch distributed). Submitting within this context should prevent this. Usage ----- with submitit.helpers.clean_env(): executor.submit(...)  MASTER_ADDR MASTER_PORTRANK WORLD_SIZE LOCAL_RANKLOCAL_WORLD_SIZE)SLURM_SLURMD_SRUN_SBATCH_ SUBMITIT_N)renviron startswithpopupdate)r distrib_namesx cluster_envs r clean_envr's,kMA LLO P P! 2::>>! '  +& +&s-B3$B $B #B3&B*%B3!B00B3cJ\rSrSrS SjrS\4SjrS S\S\SS4SjjrS r g) TorchDistributedEnvironmentiMrNc[5UlURRSUlUR 5UlURR UlURRUl URRUl URRURR-Ul g)aOConstruct a class holding the parameters required to properly setup PyTorch distributed (with the default env:// initialization method). Examples -------- >>> dist_env = TorchDistributedEnvironment().export() >>> torch.distributed.init_process_group(backend="nccl") >>> print(f"master: {dist_env.master_addr}:{dist_env.master_port}") rN) r_job_env hostnames master_addr_get_master_port master_port global_rankrank num_tasks world_size local_rank num_nodeslocal_world_sizer9s rr0$TorchDistributedEnvironment.__init__Ns'( ==2215002MM-- --11--22 $ 7 74==;R;R RrcSup[RRS5nUc;[R"UR R 5nURX5$[U5nU$)N)i Ni`r) rrgetrandomRandomrrrandintrR)rMIN_MASTER_PORTMAX_MASTER_PORTmaster_port_strrngrs rr,TorchDistributedEnvironment._get_master_port`sY+9(**..7  "-- 4 45C;;@ @/* rset_cuda_visible_devices overwritecUR[UR5[UR5[UR5[UR 5[UR 5S.nU(d,UH&nU[R;dM[SUS35e U(a[UR 5US'[RRU5 U$)aRExport all the environment variables required to properly setup PyTorch distributed (with the default env:// initialization method) i.e. MASTER_ADDR, MASTER_PORT, RANK, WORLD_SIZE (to which LOCAL_RANK and LOCAL_WORLD_SIZE are added). Parameter ---------- set_cuda_visible_device: bool if True, updates CUDA_VISIBLE_DEVICES to use only the device matching the local rank. overwrite: bool if True, overwrites the environment variables if they exist; this can be useful when launching a job from another job. Returns -------- TorchDistributedEnvironment the current instance rz'Cannot export environment variables as z is already setCUDA_VISIBLE_DEVICES) rrrrrrrrrr~r)rrrenv_varskeys rexport"TorchDistributedEnvironment.exportms8 ++t//0 Ndoo.doo. #D$9$9 :  "**$&)PQTPUUd'eff  $/24??/CH+ , (# r)rrrrrrrr)TF) rr!r"r#r0rRrrPrr'r rrrrMsBS$ # *.+"&++ ' ++rr)N )r ),r contextlibrrrrr|rdrrYtypingr%pathlibrrcore.job_environmentr core.utilsrr r r r*rJobRrUnionrRfloatrSrbrirkDictrSetrrPrQrcontextmanagerrrr rrrs  0:>F88:* ~* ^26(# ++dhhtvv& '(# [[#u*- .(#(#[[$&&!" (#VOI$I$XV5V#VSUSZSZ[^`b`f`fgj`k[kSlV("9 0h ++dhhtvv& '0h0h0hKK 0h  0hf "'2;;s+"'R[[5F"'"'JKKr