AgitatedDove14 I see that the default sample_frequency_per_sec=2. , but in the UI, I see that there isn’t such resolution (ie. it logs every ~120 iterations, corresponding to ~30 secs.) What is the difference with report_frequency_sec=30. ?

To help you debugging this: in the /dashboard endpoint, all projects were still there, but empty (no experiment inside). No experiments archived as well.

Looking forward to seeing the clearml-deploy 🤩 you guys rock 🚀

So it seems like it doesn't copy /root/clearml.conf and it doesn't pass the environment variables (CLEARML_API_HOST, CLEARML_API_ACCESS_KEY, CLEARML_API_SECRET_KEY)

Ho wow! is it possible to not specify a remote task? (If i am working with Task.set_offline(True))

Now I am trying to restart the cluster with docker-compose and specifying the last volume, how can I do that?

Looks like its a hurray then 😄 🎉 🍾

So it can be that when restarting the docker-compose, it used another volume, hence the loss of data

Nevertheless there might still be some value in that, because it would allow to reduce the starting time by removing the initial setup of the agent + downloading of the data to the instance - but not as much as I described initially, if instances stopped are bound to the same capacity limitations as new instances launched

nvm, bug might be from my side. I will open an issue if I find any easy reproducible example

So either I specify in the clearml-agent agent.python_binary: python3.8 as you suggested, or I enforce the task locally to run with python3.8 using task.data.script.binary

These images are actually stored there and I can access them via the url shared above (the one written in the pop up message saying that these files could not be deleted)

Usually one or two tags, indeed, task ids are not so convenient, but only because they are not displayed in the page, so I have to go back to another page to check the ID of each experiment. Maybe just showing the ID of each experiment in the SCALAR page would already be great, wdyt?

CostlyOstrich36 , this also happens with clearml-agent 1.1.1 on a aws instance…

Alright SuccessfulKoala55 I was able to make it work by downgrading clearml-agent to 0.17.2

Ok so it seems that the single quote is the reason, using double quotes works

Thanks SuccessfulKoala55 😁

I am doing so

There’s a reason for the ES index max size

Does ClearML enforce a max index size? what typically happens when that limit is reached?

Just tried, still the same issue

What I mean is that I don't need to have cudatoolkit installed in the current conda env, right?

Why?

The only thing that changed is the new auth.fixed_users.pass_hashed field, that I don’t have in my config file

The main issue is the task_logger.report_scalar() not reporting the scalars

AgitatedDove14 I finally solved it: The problem was --network='host' should be --network=host

And so in the UI, in workers&queues tab, I see randomly one of the two experiments for the worker that is running both experiments

CostlyOstrich36 I updated both agents to 1.1.2 and still go the same problem unfortunately. Since I can download the full log file from the Web UI, I guess the agents are reporting correctly?
Could it be that the elasticsearch does not return all the requested logs when it is queried from the WebUI to display it in the console?
Now that I think about it, I remember that on the changelog of the clearml-server 1.2.0 the following is listed:
` Fix UI Workers & Queues and Experiment Table pages ...

I can live with the current setup for now

Hi AgitatedDove14 , sorry somehow this message got lost 😄
clearml version is the latest at the time, 1.7.1 Yes, I always see the "model uploaded completed" for such stuck tasks I am using python 3.8.10

AgitatedDove14 If I call explicitly task.get_logger().report_scalar("test", str(parse_args.local_rank), 1., 0) , this will log as expected one value per process, so reporting works