Hi All, I Have A Question Regarding Clearml Task Initialization With Multithreading. I'M Using Python'S Joblib Library And The Parallel Class To Run An Experiment In Multiple Parallel Threads. The Experiment Runs To Completion But I Get Incomplete Mllogge

Answered

Hi all, I have a question regarding ClearML Task initialization with multithreading. I'm using Python's joblib library and the Parallel class to run an experiment in multiple parallel threads. The experiment runs to completion but I get incomplete MLLogger logs from the multithreaded run and also my child tasks are left in 'Draft' state while only the parent task runs and emits the incomplete logs. Was wondering if anyone had any suggestions or docs references for best practices in ClearML multithreaded logging with joblib and MLLogger? Thanks

  				
Posted 
	12 months ago

					More
				  		
  Report
		
					HungryTurtle13
				
					0

Votes Newest

Answers

Hi @<1619867994005966848:profile|HungryTurtle13>

I'm using Python's joblib library and the Parallel class to run an experiment in multiple parallel threads.

I believe joblib creates subprocesses not threads, but yes you are correct,
Basically once Task.init is called, every forked/spawned process will be automatically logged to the main process Task (you can, and probably should call either Task.init or Task.current_task() from the forked processes, but this is just a detial)
The main limitation we cannot "patch" joblib before it spwans the multiple processes so every process knows that it has a "parent" Task to report to. Does that make sense ?

  				
Posted 
	12 months ago

					More
				  		
  Report
		
					AgitatedDove14
				
					0
					 × 1

Write your answer

678 Views

1 Answer

12 months ago