BoredHedgehog47 you need to make sure "<path here>/train.py" also calls Task.init (again no need to worry about calling it twice with different project/name)
The Task.init call will make sure the auto-connect works.
BTW: if you do os.fork , then there is no need for the Task.init, the main difference is that POpen starts a whole new process, and we need to make sure the newly created process is auto-connected as well (i.e. calling Task.init)
Very lacking wrt to how things interact with one another
If I'm reading it correctly, what you are saying is that some of the "big picture" / holistic approach on how different parts interact with one another is missing, is that correct?
I think ClearML would benefit itself a lot if it adopted a documentation structure similar to numpy ecosystem
Interesting thought, what exactly would you suggest we "borrow" in terms of approach?
Specifically for this one, this is the auto generated docstring from the actual code, so PR to the
https://github.com/allegroai/clearml/blob/e53a76b713910adaf87578c69e86f8154d4ab4c1/clearml/logger.py#L152
You are correct, the agent will clone the git and install the requirements, as written in the task installed packages section. Regrading the git branch, notice it will pull the specific commit id as stated in the execution section, and it will apply any uncommitted changes. You can edit the execution section and change the commit to the latest in a specific version (you should probably also clear the uncommitted changes of you do that)
Hi TightElk12
would like to understand the limitations ofΒ
Task.current_task()
Basically this will always get you an instance of the current Task. This will work from sub-processes as well as the main process. Is there a specific scenario you have in mind, or a challenge with the use case ?
could one also limit the number of CPU cores available?
If you are running in docker mode you can add:--cpus=<value>
see ref here: https://docs.docker.com/config/containers/resource_constraints/
Just add it to extra_docker_arguments
:
https://github.com/allegroai/clearml-agent/blob/2cb452b1c21191f17635bcb6222fa8bfd82afe29/docs/clearml.conf#L142
So good news (1) Dashboard is being worked on as we speak. (2) we released clearml-serving doing exactly that, the next release of clearml-serving will include integration with kfserving (under the hood) essentially managing the serving endpoints on top of the k8s cluster , wdyt?
Requested version: 2.28, Used version 1.0" for some reason
This is fine that means there is no change in that API
In the documentation it warns about
.close()
"Only call Task.close if you are certain the Task is not needed."
Maybe this is not clear enough, this means you do not need to automatically Add/Log/Track things into the Task in the current process.
This does Not mean you cannot access the Task or its artifacts
Mark closed means to externally (i..e not from the process that crated the Task, maybe even from a different machine) close and mark the task as completed (this...
Thank you @<1523701949617147904:profile|PricklyRaven28> !!!
Let me see if we can reproduce and how to solve it
Would it also be possible to query based on
multiple
user properties
multiple key/value I think are currently not that easy to query,
but multiple tags are quite easy to do
tags=["__$all", "tag1", "tag2],
Hi MinuteGiraffe30
Thank you so much for your awesome product!
π !
s address 10.68.167.10. I am able to send requests from all other virtual machines on the server to the address 10.68.167.10:8008. However, when I try to do this from my own computer connected to the corporate network via VPN, it fails to connect to 8008.
I'm assuming there is a firewall on the VPN connection itself (i.e. the VPN gateway) that blocks 8008 port, as you already tried curl to 8008 is...
Wonβt they be printed out as well in the Web UI?
They would in the log, but it will not be stored back on the Task (the idea is these are "agent specific" additions no need for them to go with the Task)
So Iβve tried the approach and it does work,
ScantChimpanzee51 What do you mean it does not work? what exatcly are you trying with task.connect and does not work?
Is there a way to inject environment variables into a Task or into its container?
Yes you can with:
` task.s...
SubstantialElk6 when you say "Triton does not support deployment strategies" what exactly do you mean?
BTW: updated documentation already up here:
https://clear.ml/docs/latest/docs/clearml_serving/clearml_serving
If you need to change the values:config_obj.set(...)
You might want to edit the object on a copy, not the original π
I also found that you should have a deterministic ordering
before
you apply a fixed seed
Not sure I follow ?
Just making sure, the machine that you were running the "trains-init" on can access the API server ?
And after having called
Task.init()
the second time, the automatic logging of resources and tensorboard plots works as well. I would recommend adding explanation to the docs for
Oh yeah! you always need to call Task.init first, Task,current_task should be called from anywhere you like but after the Task.init was called.
Can you share the log?
Ohh... I would not delete them then ... π
Maybe kind of heuristics (files created a week ago can be deleted?!)
In your code, can you print the following:import os print(os.environ.keys())
There should be a few keys the Pycharm plugin is sending from the local machine, pointing to the git repo
I see, when you run it manually (i.e. not via an agent) what do you have under the configuration tab in the UI (meaning do you see both argparser arguments there)?
Hi DrabCockroach54
This seems like a pip issue trying to install from source, try upgrading the pip version and before installing numpy, it should solve it π€
Thanks GreasyPenguin66
How about:!curl
BTW, no need to rebuild the docker, next time you can always do !apt update && apt install -y <package here>
π
yey π notice that when executed by the agent the call execute_remotely
is skipped, and so does the If statement I added (since running_locally will return False when the process is executed by the agent)
RobustSnake79 this one seems like scalar type graph + summary table, correct?
BTW: I'm not sure how to include the "Recommendation" part π
not sure what is the "right way" π
But I do pkill -f "trains-agent --gpus 0"
This will kill a process that started "trains-agent --gpus 0" Notice it matches the cmd pattern so it has to match the way you executed the agent. You can check it with ps -Af | grep trains-agent
Is task.parent something that could help?
Exactly π something like:# my step is running here the_pipeline_task = Task.get_task(task_id=task.parent)
Is there a helper function option at all that means you can flush the clearml-agent working space automatically, or by command?
Every Task execution the agent clears the venv (packages are cached locally, but the actual venv is cleared). If you want you can turn on the venv cache, but there is no need to manually clear the agent's cache.