Hi @<1546303254386708480:profile|DisgustedBear75> , what do you mean?
You get different results or your experiment fails?
Running in venv mode can be more prone to failure if you're running between different operating systems & python versions.
The default behavior of ClearML when running locally is to detect the packages used in the code execution (You can also provide specific packages manually or override auto detection entirely) and log them in the backend.
When a worker in a virtual...
Hi @<1546303254386708480:profile|DisgustedBear75> , there are a few reasons remote execution can fail. Can you please describe what you were trying to do and please add logs?
Also, you need to restart the agent between changes in the config
If you mean to fetch the notebook via code you can see this example here:
None
What do you mean exactly by run it as notebook? Do you mean you want an interactive session to work on a jupyter notebook?
Hi @<1785479228557365248:profile|BewilderedDove91> , I think this is the env variable you're looking for - CLEARML_AGENT_FORCE_CODE_DIR
VexedCat68 , do you mean does it track which version was fetched or does it track everytime a version is fetched?
I'm not sure what you mean by leaderboard, but you can add custom metrics to the smart dashboard and sort by that if this is what you're looking for
AgitatedDove41 , What version of ClearML are you using?
Was the artifact very large per chance or is there any chance you were having network issues at the time?
So when you do torch.save() it doesn't save the model?
I don't think you can currently assign cpu cores to the agents. They just use the resources they have in cpu mode
From the looks of it, it's failing to recreate the environment - something about numpy. Are you trying to run on two different OS's or different pythons? My best suggestion would be to try running inside docker
I think you also might find this video useful:
None
Can you share the entire log of the run?
CluelessElephant89 , I think the RAM requirements for elastic might be 2GB, you can try the following hack so it maybe will work.
In the machine that it's running on there should be a docker-compose.yml file (I'm guessing at home directory).
For the following https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml#L41 you can try changing it to ES_JAVA_OPTS: -Xms1g -Xmx1g and this might limit the elastic memory to 1 gb, however please note this might ...
RipeAnt6 , you have to manage your storage on the NAS yourself. We delete data only on the fileserver.
However, you could try mounting the NAS to the fileserver docker as a volume and then deletion should also handle files on the NAS 🙂
Hii @<1608271575964979200:profile|GiddyRaccoon10> , ClearGPT is a separate enterprise product 🙂
Can you provide a snippet to try and reproduce?
Hi @<1739093605621960704:profile|LovelySparrow29> , do you see any errors in mongo or elastic containers?
WackyRabbit7 I don't believe there is currently a 'children' section for a task. You could try managing the children to access them later.
One option is add_pipeline_tags(True) this should mark all the child tasks with a tag of the parent task
Hi @<1540142651142049792:profile|BurlyHorse22> , it looks like an error in your code that is bringing the traceback. What is happening during the traceback?
Hi @<1547028074090991616:profile|ShaggySwan64> , You can try this. However, Elastic takes space according to the amount of metrics you're saving. Clearing some older experiments would free up space. What do you think?
Can you add a full log of an experiment?
Here:
https://clear.ml/docs/latest/docs/configs/clearml_conf#agent-section
What you're looking for is this:sdk.development.default_output_uri
Also configure your api.files_server in ~/clearml.conf to point to your s3 bucket as well 🙂
Hi @<1547028131527790592:profile|PleasantOtter67> , nothing out of the box. You can however quite easily extract all that information and inject it into a csv programmatically.
I think the bigger question is how would you break it down? Each experiment has several nested properties.
Do try with the port through
Hi @<1695969549783928832:profile|ObedientTurkey46> , this is supported in the Scale/Enterprise licensees of ClearML (external IdP support). API access is always done using credentials.
Then it's the community server, that is not an enterprise version. In the PRO version only AWS/GCP autoscalers are available.
In your ~/clearml.conf you can specify the following to force the model to upload with the following setting:sdk.development.default_output_uri