And you're sure that clearml.conf
points to the correct server with the right credentials?
Hi @<1534496192929468416:profile|EagerGiraffe33> , what if you try to put a specific version of pytorch you've tested on your remote environment in the requirements section of the cloned task?
You can add it to your pip configuration so it will always be taken into account
Or should I set agent.google.storage {}?
Did you follow the instructions in the docs?
Hi @<1546303277010784256:profile|LivelyBadger26> , can you provide a snippet that reproduces this?
Can you also add a full log of the run that was showing the git pass in the startup print?
Hi @<1523711002288328704:profile|YummyLion54> , it hasn't been added to the PRO yet 🙂
I would guess sosudo docker logs --follow trains-webserver
Hi @<1523703397830627328:profile|CrookedMonkey33> , not sure I follow. Can you please elaborate more on the specific use case?
Currently you can add plots to the preview section of a dataset
Hi @<1756488209237282816:profile|IdealCamel64> , I think ClearML would be perfect for that. You can also enable users to have their own remote sessions directly to the GPUs (inside a container even). I'd check out ClearML's orchestration layer + remote sessions:
None
None
Regarding what @<1576381444509405184:profile|ManiacalLizard2> said, he's wrong I'm afraid. ClearML can run ...
Hi @<1759749707573235712:profile|PungentMouse21> , you should be able to access machine logs from the autoscaler, this should give you a place to search
Hi @<1603198163143888896:profile|LonelyKangaroo55> , you can see the commit ID in the execution tab of the experiment.
You added two logs, one with docker the other without. Each stopped on a different step. Is that consistent? What OS is the agent running on? Also what is the command you're using to run the agent?
OddShrimp85 , Hi 🙂
I'm afraid that the only way to load contents of setup A into setup B is to perform a data merge.
This process basically requires merging the databases (mongodb, elasticsearch, files etc.). I think it's something that can be done in the paid version as a service but not in the open one.
JitteryCoyote63 , can you try upgrading to the latest versions? Is there any reason why you stayed on older versions?
CluelessElephant89 , I think the RAM requirements for elastic might be 2GB, you can try the following hack so it maybe will work.
In the machine that it's running on there should be a docker-compose.yml
file (I'm guessing at home directory).
For the following https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml#L41 you can try changing it to ES_JAVA_OPTS: -Xms1g -Xmx1g
and this might limit the elastic memory to 1 gb, however please note this might ...
Hi FierceRabbit20 , I don't think there is such an option out of the box but you can simply add it to your startup of the machine or create a Cron job
Also please see the other questions I asked previously
Hi @<1874989039501709312:profile|LividDragonfly0> , podman is supported out of the box and is transparent, you can just run the agent with --docker
command.
Hi DepressedFox45 ,
For the agent you'll need to run clearml-agent init
Hi @<1789465500154073088:profile|ScaryShrimp8> , can you provide the full log of the run? How did you set up & run the agent? What version and what OS are you on?
Are you running the agent on the same machine as the server?
Hi RoundMosquito25 , yes you can. You can first filter by enqueued experiments and also in the top left you have a small checkbox. If you click it, you will have an option to select all experiments and not only the ones that are loaded
Hi GrittyHawk31 ,
To avoid reusing tasks you can set the following configuration in ~/clearml.conf
sdk.development.task_reuse_time_window_in_hours: 0
In any case - models shouldn't be replaced but just new ones will be created in the system
I think this is covered in the enterprise version
Hi @<1840924589736071168:profile|EmbarrassedCrab49> , the AI Gateway is part of only the Enterprise version, its not available in the OS
Hi @<1858681577442119680:profile|NonchalantCoral99> , I would suggest opening a new workspace with PRO, tying it to a generic email ( devop@company.com for example) so when people shift, you won't have a problem with emails.
I think this is what you're looking for, tell me if it helps 🙂
Hi @<1570220852421595136:profile|TeenyHedgehog42> , are you using the latest version of clearml-agent
? Can you provide a stand alone code snippet that reproduces this behavior for you?