Reputation
Badges 1
38 × Eureka!AgitatedDove14
Are you saying the second time this line is missing?
Yes.
Can you send the full Task log?
I will send the log in direct messages.
when I restart the agent, it works fine, but on the second launch docker does not mount the ssh keys folder:'-v', '/tmp/clearml_agent.ssh.rbw8o0t7:/root/.ssh',
I don’t understand why. AgitatedDove14 JitteryCoyote63 could you explain the logic behind that? CLEARML_AGENT_DISABLE_SSH_MOUNT variable is not set.
So it fails with this log message:
` ...
Using cached repository in "/root/.clearml/vcs-cache/<MY_REPO>.git.893c8c47c9813c27eb1fe8d0aeb77a11/<MY_REPO>.git"
fatal: Could not read f...
AgitatedDove14 sorry, no, in fact my configuration looks like:
` ...
agent.git_user=""
agent.git_pass=""
agent.git_host=""
agent.package_manager.extra_index_url= [
]
agent {
worker_id: ""
worker_name: ""
force_git_ssh_protocol: true
... `
AgitatedDove14 we can read sys/fs/cgroup/memory/memory.limit_in_bytes to get the limit
https://faun.pub/understanding-docker-container-memory-limit-behavior-41add155236c
Hi CostlyOstrich36
How are you mounting the credentials?
Is this also mounted into the docker itself?
as I wrote above, it is mounted automatically:'-v', '/tmp/clearml_agent.ssh.kqzj9sky:/root/.ssh
What version of
ClearML-Agent
are you using?
1.3.0
task log
` task ca3ab0ce39aa436f9e656fff378a2c25 pulled from c39519fcfb3f4353808fd266d6100795 by worker v012-0:gpuGPU-0929fd0f-eff1-91f1-854e-9874599660c3
2022-12-12 16:32:21
Current configuration (clearml_agent v1.5.1, location: /tmp/.clearml_agent.guezjnez.cfg):
api.version = 1.5
api.verify_certificate = true
api.default_version = 1.5
api.http.max_req_size = 15728640
api.http.retries.total = 240
api.http.retries.connect = 240
api.http.retries.read = 240
api.http.retri...
can’t find the desired method, I thought that tasks.get_configurations is exactly what I need, but it is not
AgitatedDove14
Specifically
/tmp/clearml_agent.ssh.rbw8o0t7
is the copy of the .ssh that the agent created, and now it is mounting it into the container
but why is it mounted only once? second and following containers do not mount the folder
@<1523701181375844352:profile|ExasperatedCrocodile76> hi, try to pass “--network=host” to --docker_args
example:
clearml-task --project project --name name --script run.py --queue queue --requirements requirements.txt --docker python:3.7.13-bullseye --docker_args "--cpus=8 --memory=16g --network=host"
AgitatedDove14 no, it’s not a request.
I have custom python class, that uses a lot of models from frameworks that supported by ClearML already. I want to enable auto reporting for all models by using command clearml_task.connect(my_custom_class_instance)
, but it doesn’t work the way I need it to — there is the only one loss curve, because because this graph is redrawn every time a new instance starts training.
Is there any way to reporting all instances inside my custom class without ...
AgitatedDove14 done) btw, could you show me the place in the code where scalars are written? I want to make a hotfix
RoundMosquito25 hi, any updates?
AgitatedDove14 , do you know the answer?