Hi JitteryCoyote63 ,
If you run in docker mode it will execute the shell script. I think it should be printed in the logs on startup.
Can you please add a larger chunk of the autoscaler log?
Hi PunyWoodpecker71 , what are you trying to do?
Do you have K8s?
let me know if it changes anything. Of course rerun the agent afterwards
Hi @<1556812506238816256:profile|LargeCormorant97> , I think you would need to go deeper and investigate each docker's environment and see what is run inside each container and what is the entrypoint since there are several containers each in charge of something else.
Is there a specific reason you need to deploy it without docker?
You can read up on the caching options in your ~/clearml.conf
You can make virtualenv creation a bit faster
Thanks for the info! This happened when you had 2 spot instances running something, correct?
Do you know what was the state of the experiments at the time?
Or are you just trying to run clearml-agent?
Hi @<1649221402894536704:profile|AdventurousBee56> , I'm not sure I understand. Can you add the full log and explain step by step what's happening?
Hi DrabCockroach54 , what version of ClearML are you using? What OS is this running on?
Hi @<1566596960691949568:profile|UpsetWalrus59> , looks right to me 🙂
Does the Autoscaler try to spin new instances?
AbruptWorm50 , you can send me. Also can you please answer the following two questions?When were they registered? Were you able to view them before?Also, you mention plots but in the screenshot you show debug samples. Can I assume you're talking about debug samples?
HappyDove3 Hi 🙂
Well, since all the data is logged you can simply use the API to retrieve it and create the tables yourself quite easily!
Hi @<1585441130525233152:profile|TrickyGoose45> , you mean that the mongodb you set up is not part of the docker compose?
Can you please elaborate on what you're trying to do and what is failing?
What happens if you delete ~/.clearml It's clearml's cache folder
Hi VivaciousBadger56 , This is a good question. In my opinion it's best to start with viewing videos from ClearML's YouTube video. This one is especially useful:
https://www.youtube.com/watch?v=quSGXvuK1IM
As regards to which steps to take, I think the following should cover most bases:
Experiment tracking & management - See that you can see all of the expected outputs in the ClearML webUI Remote experiment execution - Try and execute an experiment remotely using the agent. Change some c...
CumbersomeParrot30 , try setting the following environment variable to true:CLEARML_SKIP_PIP_VENV_INSTALL
Hi ExcitedSeaurchin87 ,
How are you trying to run the agents? Also, are you trying to run multiple agents on the same GPU?
I'm just not sure what error you're getting
Didn't have a chance to try and reproduce it, will try soon 🙂
In case of non mutable copy you get a direct link to the cache folder, you should not change the content. In case of mutable it's a local copy for you to change or play around with, you can make any changes you want there 🙂
No, but I think it would make sense to actually share reports outside of your workspace, similar to experiments. I'd suggest opening a GitHub feature request
Hi @<1810483998732849152:profile|NonsensicalDuck81> , as long as you have docker installed, yes 🙂
DepressedChimpanzee34 , this has been reported and should be solved in one of the following versions 🙂
