Reputation
Badges 1
25 × Eureka!BTW, we figure out thatΒ Β
'
Β is belong the echo
yep, when seeing the full command it is apparent
EnviousStarfish54 you can also run the docker-compose on one of the machines on your local LAN. but then you will not be able to access it from home π
Hi AbruptWorm50
I am currently using the repo cache,
What do you mean by "using the repo cache" ? This is transparent, the agent does that, users should not access that folder?
I also looked at the log you send, why do you think it is re-downloading the repo?
Hi, I would like to understand how I can set the pip cache location for my agent,
ClumsyElephant70 by default the pip cache (and all other cache folders) are mounted back into the host itself ~/.clearml/
I'm assuming the idea is shared cache, if this is the case, do:docker_pip_cache = ~/my_shared_nfs/pip-cache
https://github.com/allegroai/clearml-agent/blob/e3e6a1dda81bee2dd20a64d09746568e415f1823/docs/clearml.conf#L139
are you using matplotlib ? could it be the binding check if matplotlib exists ? could it be you are running it with DEBUG on (i.e. global log level debug) ?
Usually in the /tmp folder under a temp filename (it is generated automatically when spinned)
In case of the services, this will be inside the docker itself
DilapidatedDucks58
is there any way to post Slack alerts for the frozen experiments?
The latest RC should solve the PyTorch data loader, do you want to test it?pip install clearml==0.17.5rc2
Maybe different API version...
What's the trains-server version?
DefeatedOstrich93 can you verify lightning actually only stored once ?
Hi RoundSeahorse20
Try the following , let me know if it worked.clear_logger = logging.getLogger('clearml.metrics') clear_logger.setLevel(logging.ERROR)
Basically try with the latest RC π
pip install trains 0.15.2rc0
JitteryCoyote63 while it's running, could you give me a few details on the setup, maybe I can reproduce it.
Is it using pytorch distributed ?
Are all models uploaded to S3 ?
etc.
now that I know I could use the right click I'll use it like in google drive etc.
That was the initial thought, but I think right clicking on a web page is not you "go to action", especially for Mac ppl ...
UnsightlyShark53 See if this one solves the problem :)
BTW: the reasoning for the message is that when running the task with "trains-agent" if the parsing of the argparser happens before the the Task is initialized, the patching code doesn't know if it supposed to override the values. But this scenario was fixed a long time ago, and I think the error was mistakenly left behind...
AstonishingSeaturtle47 How would the code run without the sub-modules? And what is the problem we are trying to solve? (Because unfortunately there is no switch to disable it)
neat! please update on your progress, maybe we should add an upgrade section once you have the details worked out
Yes, I think you are correct, verified on Firefox & Chrome. I'll make sure to pass it along.
Thanks SteadyFox10 !
btw:
If you need to access it, just bash into the running dockerdocker exec -it <container_name> /bin/bash
JitteryCoyote63 with pleasure π
BTW: the Ignite TrainsLogger will be fixed soon (I think it's on a branch already by SuccessfulKoala55 ) to fix the bug ElegantKangaroo44 found. should be RC next week
What will I do to fix my problem?
What is the problem? we just proved the upload speed is just fine?
Please hit Ctrl-F5 refresh the entire page, see if it is till empty....
Hi CooperativeFox72 trains 0.16 is out, did it solve this issue? (btw: you can upgrade trains to 0.16 without upgrading the trains-server)