Sure @<1523720500038078464:profile|MotionlessSeagull22> DM me 🙂

Hi @<1523701205467926528:profile|AgitatedDove14> , I wanted to ask you something. Is it possible that we can talk over voice somewhere so that I can explain my problem better?

If you one each "main" process as a single experiment, just don't call Task.init in the scheduler

Create one experiment (I guess in the scheduler)
task = Task.init('test', 'one big experiment')
Then make sure the the scheduler creates the "main" process as subprocess, basically the default behavior)
Then the sub process can call Task.init and it will get the scheduler Task (i.e. it will not create a new task). Just make sure they all call Task init with the same task name and the same project name.

like if u see in above image my project name is abcd18 and under that there are experiments Experiment1, Experiment2 etc.

i mean all 100 experiments in one project

no i want all of them in the same experiment

And you want all of them to log into the same experiment ? or do you want an experiment per 60sec (i.e. like the scheduler)

like in the above picture

each subprocess logs one experiment as task

yes

Just so I understand,
scheduler executes main every 60sec
main spins X sub-processes
Each subprocess needs to report scalars ?

and then log using logger

then my combined function create a sub task using Task.create(task_name=exp_name)

main will initialize parent task and then my multiprocessing occurs which call combined function with parameters as project_name and exp_name

my scheduler will be running every 60 seconds and calling main function

what changes should I make here?

def combined(path,exp_name,project_name):
temp = Task.create(task_name="exp_name")

logger = temp.current_logger()
logger.report_scalar()

def main():
task=Task.init(project_name="test")
[pool.apply_async(combined, args = (row['Path'], row['exp_name'], row['project_name'])) for index,row in temp_df.iterrows()]

scheduler = BlockingScheduler()
scheduler.add_job(main, 'interval', seconds=60, max_instances=3)
scheduler.start()

No. since you are using Pool. there is no need to call task init again. Just call it once before you create the Pool, then when you want to use it, just do task = Task.current_task()

Is this u meant?

then if there are 10 experiments then I have to call Task.create() for those 10 experiments

I have to create a main task for example named as main

Oh I got it.

so, if I call Task.init() before that line there is no need of calling Task.init() on line number 92

It will not create another 100 tasks, they will all use the main Task. Think of it as they "inherit" it from the main process. If the main process never created a task (i.e. no call to Tasl.init) then they will create their own tasks (i.e. each one will create its own task and you will end up with 100 tasks)

then if there are 100 experiments how it will create 100 tasks?

so , it will create a task when i will run it first time

Before this line, call Task.init

and that function creates Task and log them