Also tried task.get_logger().report_text(str(task.data.hyperparams))
-> AttributeError: 'Task' object has no attribute 'hyperparams'

So I created a symlink in /opt/train/data -> /data

I got some progress TimelyPenguin76 , Now the task runs and I get the error from docker:
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

I still don't see why you would change the type of the cloned Task, I'm assuming the original Task had the correct type, no?

Because it is easier for me that I create a training task out of the controller task by cloning it (so that parameters are prefilled and I can set the parent task id)

Please wait a few mins - This example is not valid, I will share a new one soon

CostlyOstrich36 , this also happens with clearml-agent 1.1.1 on a aws instance…

torch==1.7.1 git+ .

AgitatedDove14 yes 🙂

I have two controller tasks running in parallel in the trains-agent services queue

agent.package_manager.type = pip ... Using base prefix '/home/machine1/miniconda3/envs/py36' New python executable in /home/machine1/.trains/venvs-builds/3.6/bin/python3.6 Also creating executable in /home/machine1/.trains/venvs-builds/3.6/bin/python Installing setuptools, pip, wheel...

awesome, thank you 👍

Yes AgitatedDove14 🙂

yes

Thanks! Corrected both, now its building

AgitatedDove14 I see https://github.com/allegroai/clearml-session/blob/main/clearml_session/interactive_session_task.py#L21= that a key pair is hardcoded in the repo. Is it being used to ssh to the instance?

The weird thing is that the second experiment started immediatly, correctly in a docker container, but failed with User aborted: stopping task (3) at some point (while installing the packages). The error message is suprizing since I did not do anything. And then all following experiments are queued to services queue and stuck there.

Yes this is correct. I am trying to create a minimal reproducable example

I am sorry to give infos that are not very precise, but it’s the best I can do - Is this bug happening only to me?

AgitatedDove14 one last question: how can I enforce a specific wheel to be installed?

(I use trains-agent 0.16.1 and trains 0.16.2)

Yes, I stayed with an older version for a compatibility reason I cannot remember now 😄 - just tested with 1.1.2 and it’s the same
I tried specifying the bucket directly in my clearml.conf, same problem. I guess clearml just reads from the env vars first

Hi TimelyPenguin76 , any chance this was fixed? 🙂

Thanks!

Sure 🙂 Opened https://github.com/allegroai/clearml/issues/568

Hi TimelyPenguin76 , any chance this was fixed already? 🙂

This is the mapping of the faulty index:
` {
"events-plot-d1bd92a3b039400cbafc60a7a5b1e52b_new" : {
"mappings" : {
"dynamic" : "strict",
"properties" : {
"@timestamp" : {
"type" : "date"
},
"iter" : {
"type" : "long"
},
"metric" : {
"type" : "keyword"
},
"plot_data" : {
"type" : "binary"
},
"plot_len" : {
"type" : "long"
},
"plot_str" : {
...

(docker was install with sudo snap install docker )

in an ec2 machine

This works well when I run the agent in virtualenv mode (remove --docker )