I get gaps in the graphs.
For example, the first time I run, I create a task and run a loop:

Hi SourOx12
Is this related to this one?
https://github.com/allegroai/clearml/issues/496

Hmm, it seems as if the task.set_initial_iteration(0) is ignored...
What's the clearml version you are using ?
Is it the same one you have on the local machine ?

When I work through Colab, when I continue experimenting, I get gaps in the graphs.
For example, the first time I run, I create a task and run a loop:
for i in range(1,100): clearml.Logger.current_logger().report_scalar("test", "loss", iteration=i, value=i)

Then, on the second run, I continue the task via continue_last_task and reuse_last_task_id and write task.set_initial_iteration(0). Then I start the cycle:
for i in range(100,200):
clearml.Logger.current_logger().report_scalar("test", "loss", iteration=i, value=i)
And then on the graphs I get a gap.

Yey! okay let me make sure we add this feature to the Task.init arguments so one can control it from code 🙂

I'm so happy to see that this problem has been finally solved!

I can't think of any actual difference in flow ...
Can you try the following?
task._setup_reporter() task.set_initial_iteration(0)

AgitatedDove14
If I use this method, then new scalar values stop being added to the graph

Okay I think I know what's going on (there is a race that for some reason on CoLab acts differently).
As a quick hack you can do the following:
Task._report_subprocess_enabled = False task = Task.init(...) task.set_initial_iteration(0)

Can you give a small snippet to play with? Just to understand, when you run on local machine everything works fine? What do you do with Google Colab?

Hi, AgitatedDove14
Yes, that sounds like my problem. But I do not know how it can help me:(

AgitatedDove14
Yes, I have problems with continuing experiments in colab. I do everything the same as on my computer, but in the case of colab, I have gaps in the charts.

But I do not know how it can help me:(

In your code itself after the Task.init call add:
task.set_initial_iteration(0)See reply here:
https://github.com/allegroai/clearml/issues/496#issuecomment-980037382

AgitatedDove14
I upload to colab via “pip install clearml”, and therefore it is probably the most up-to-date there. The version on my computer and colab is 1.1.4

And it works correctly when running on my computer, and if I use colab, then for some reason it has no effect.

I think I'm lost on this one, when running in colab, is this continuing a previous experiment ?

AgitatedDove14 Of course, I added it when restoring the experiment. And it works correctly when running on my computer, and if I use colab, then for some reason it has no effect.

Full code https://colab.research.google.com/drive/1yePGFL4qWxhIOEzwVAOLv_B7rhEB5K_2?usp=sharing

AgitatedDove14 Hooray, it helped! Thank you very much!!!!