It can be changed with this env var for the apiserver:
CLEARML__hosts__elastic__events__args__timeout=<new number>
Though the better handling could be either increase the elasticsearch capacity (memory and cpu) or decrease the load (send events in smaller batches)
@<1554638179657584640:profile|GlamorousParrot83> , can you add also the full log?
Does it enqueue the task? From what you posted it should simply create a task and then enqueue it without any further action
Can you provide a code snippet that makes agent hang?
Hi!
What version of ClearML-Agent are you using?
Also from within the docker, what do you get when you run the following commands:which python
which python3
which python3.7
which python3.8
Also please note that your path is wrong
What do you mean by verify? How are you currently running your HPO?
SparklingElephant70 , a full log would be the best. It can be downloaded from the webapp 🙂
You can contact the sales team via the contact form 🙂
Hi @<1523722618576834560:profile|ShaggyElk85> , please see here: None
I think these are the ones you're looking for:
CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL
Regarding your second question, I think this is possible only when talking about submodules
How many tasks do you figure this thing is iterating through the project?
After you store the model in ClearML server accessing it later becomes almost trivial 🙂
Hi SmugSnake6 , looks like network issue. Did you make any changes to your network/server recently
TenseOstrich47 , what do you mean exactly? Every task you run ends on 'aborted' status?
DistinctShark58 , Hi 🙂
You can change all the default ports. This can be done through the docker compose for example.
Hi, SmugTurtle78 ,
Can you please try with "Resource": "*"
?
Also these are the settings that I use, Some might be redundant so consults with your devops guys 🙂
` {
"Sid": "EC2InstanceManagement",
"Effect": "Allow",
"Action": [
"ec2:AttachClassicLinkVpc",
"ec2:CancelSpotInstanceRequests",
"ec2:CreateFleet",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:Desc...
Hi ShallowCormorant89 ,
When does 1. happen? Can you add the full log?
Regarding 2, can you please elaborate? What is your pipeline doing and what sort of configurations would you want to add? On the pipeline controller level or steps?
Hi @<1539417873305309184:profile|DangerousMole43> , I think you can do it if add some code to the pipeline controller to extract the console logs from a failed step
DepressedChimpanzee34 , what is the url like?
The url should be something like https://<WEBSITE>.<DOMAIN>/v2.14/debug/ping
Hi MagnificentWorm7 , what version of ClearML server are you running?
Also, I think that maybe there is a bug with the CPU mode: I tried to run tests with instance without GPU , marked the option "Run in CPU mode (no gpus)" and I saw on the experiment logs that its trying to run the docker with "--gpus all" option and failed right after the execution.
Which instance type did you use?
HungryArcticwolf62 , I couldn't find something relevant 😞
AgitatedDove14 , wdyt?
Hi ElegantCoyote26 ,
It doesn't seem that using port 8080 is mandatory and you can simply change it when you run ClearML-Serving - i.e docker run -v ~/clearml.conf:/root/clearml.conf -p 8085:8085
My guess is that the example uses port 8080 because usually the ClearML backend and the Serving would run on different machines
There is an options for a configurations vault in the Scale/Enteprises licenses - basically applying global settings without having to edit clearml.conf
BoredPigeon26 , when you copy and paste the link provided to you by the UI into the browser, can you see the image?
Hi 🙂
Please try specifying the file itself explicitly
Hi UnevenDolphin73 ,
I don't think that ClearML exposes anything into env variables unless done so explicitly.
If I'm understanding correctly you're hoping for a scenario where ClearML would expose some contents of sdk.aws.s3
for example so you could use it later in boto3, correct?
If that's the case why not use env vars to begin with?
I think so, yes. You need a machine with a GPU - this is assuming I'm correct about the n1-standard-1
machine