If you one each "main" process as a single experiment, just don't call Task.init in the scheduler

Create one experiment (I guess in the scheduler)
task = Task.init('test', 'one big experiment')
Then make sure the the scheduler creates the "main" process as subprocess, basically the default behavior)
Then the sub process can call Task.init and it will get the scheduler Task (i.e. it will not create a new task). Just make sure they all call Task init with the same task name and the same project name.

but this gives the results in the same graph

so what I have done is rather than reading sequentially I am reading those experiments through multiprocessing and for each experiment I am creating new task with specified project_name and task_name

so, if I call Task.init() before that line there is no need of calling Task.init() on line number 92

Before this line, call Task.init

then if there are 10 experiments then I have to call Task.create() for those 10 experiments

Just so I understand,
scheduler executes main every 60sec
main spins X sub-processes
Each subprocess needs to report scalars ?

This code gives me the graph that I displayed above

I have 100 experiments and I have to log them and update those experiments every 5 minutes

no i want all of them in the same experiment

This code will give you one graph titled "loss" with two series: (1) trains (2) loss

main will initialize parent task and then my multiprocessing occurs which call combined function with parameters as project_name and exp_name

You can do:
task = Task.get_task(task_id='uuid_of_experiment')
task.get_logger().report_scalar(...)

Now the only question is who will create the initial Task, so that the others can report to it. Do you have like a "master" process ?

It will not create another 100 tasks, they will all use the main Task. Think of it as they "inherit" it from the main process. If the main process never created a task (i.e. no call to Tasl.init) then they will create their own tasks (i.e. each one will create its own task and you will end up with 100 tasks)

Like here in the sidebar I am getting three different plots named as loss, train_loss and test_loss

I have one more question?

then if there are 100 experiments how it will create 100 tasks?

I have to create a main task for example named as main

Sure

make sense ?

Can my request be made as new feature so that we can tag same type of graphs under one main tag

Sure, open a Git Issue :)

and under that there will be three graphs with title as train test and loss

So you want these two on two different graphs ?

Can my request be made as new feature so that we can tag same type of graphs under one main tag

And you want all of them to log into the same experiment ? or do you want an experiment per 60sec (i.e. like the scheduler)

now after 1st iteration is completed then after 5 minutes my script runs automatically and then again it logs into trains server

Just call the Task.init before you create the subprocess, that's it 🙂 they will all automatically log to the same Task. You can also call the Task.init again from within the subprocess task, it will not create a new experiment but use the main process experiment.

my scheduler will be running every 60 seconds and calling main function