So far, i modified the code to set DOCKER_ROOT_CONF_FILE to what i want !!!
Interesting, do you think a PR is a good next step ? how one would configure it?

MysteriousBee56 when you run the trains-agent with --foreground , before it starts the docker it print the full command line, could you send it please?
I can't figure out where the extra ' came from...
Also could you send the trains.conf file?
(feel free to redact and confidential information)

Sadly, I think we need to add another option like task_init_kwargs to the component decorator.
what do you think would make sense ?

but perhaps it is worth adding to the docs page a hint to avoid using the CLEARML_TASK_ID env variable, perhaps I am not the only one to ever try it

Good idea, any thoughts on where ? I cannot find a trivial place to put these things

MuddySquid7
are you saying that for some reason the models pick the artifacts ? Is that reproducible ? (they are two different things)
Can you see the df.pkl on the Models section of the Task (in the UI) ?

This is because we have a pub-sub architecture that we already use, it can handle retries, etc. also we will likely want multiple systems to react to notifications in the pub sub system. We already have a lot of setup for this.

How would you integrate with your current system? you have a restapi or similar to trigger event ?

but I was hoping ClearML had a straightforward way to somehow represent ALL ClearML events as JSON so we could land them in our system.

Not sure I'm followi...

Fix in the next version 🙂

can i run a random task from a queue? like this 

clearml-agent execute --id <TASK_ID>

  or

ChubbyLouse32 This will just work out of the box 🙂
No need to enqueue the Task, just reset it (in the UI)

Bugs, definitely GitHub, this is the easiest to track.
Documentation, if these are small issues, Slack is fine, otherwise, GitHub issue.
Regrading the documentation, we are working on another iteration of improvement, but if you find inaccuracies/broken links please report 🙂

https://allegro.ai/clearml/docs/docs/faq/faq.html?highlight=clearml_config_file#clearml-configuration

Hi @<1610083503607648256:profile|DiminutiveToad80>
You mean the pipeline logic? It should autodetect the imports of the logic function (like any Task.init call)
You can however call Task.force_requirements_env_freeze and pass a local requiremenst.txt
Make sure to call it before create the Pipeline object
None

Hi RotundHedgehog76

we have issues with

clearml-agent

when using standalone mode. ...

What is the use case for standalone mode? is this venv or docker mode?

Hi WittyOwl57
I think what happens is it auto-logs the joblib load/save calls, these calls track models used/created by the code, and attach them to the model repository representing these model.
I'm assuming there are multiple load/save , and there are multiple model instances pointing to the same local file "file:///tmp/..." . The earning basically says it is re-registering existing models.
Make sense ?

DeliciousBluewhale87 Yes I think so, do notice that you might end up with maximum of 12 pods.
You can also do the following with max 10 nodes: (notice --queue can always get a list of nodes it will pull based on the order of the queues)
python k8s_glue_example.py --queue high_priority_q low_priority_q --ports-mode --num-of-services 10

I'm running hyper parameter optimzation on LSF cluster where every task is an LSF job running without clearml-agent

WOW this is so cool! 🎊

Hi @<1569858449813016576:profile|JumpyRaven4>
What's the clearml-serving version you are running ?

This happens even though all the pods are healthy and the endpoints are processing correctly.

The serving pods are supposed to ping "I'm alive" and that should verify the serving control plan is alive.
Could it be no requests are being served ?

Does that work?

Hi @<1523701066867150848:profile|JitteryCoyote63>

Setting to redis from version 6.2 to 6.2.11 fixed it but I have new issues now

Was the docker tag incorrect in the docker compose ?

GreasyPenguin14 you mean the artifacts/models ?

Task.get_projects() 🙂

im not running in docker mode though

hmmm that might be the first issue. it cannot skip venv creation, it can however use a pre-existing venv (but it will change it every time it installs a missing package)
so setting CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1 in non docker mode has no affect

there is almost zero overhead if your docker container alreadyt has everything (including the agent) preinstalled and you set it with CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1
it then should basically just run the code.

Oh what if the script is in the container already?

Hmm, the idea of clearml is that the container is a "base environment" and code is "injected", this makes sure it is easy to reuse it.
The easiest way is to add an "entry point" scripts that just calls the existing script inside the container.
You can have this python initial script on your local machine then when you call clearml-task it will upload the local "entry point" script directly to the Task, and then on the remote machin...

Nice workaround!
RoughTiger69 how do I reproduce this behavior? (I'm still unsure on why exactly the clearml binding broke it, and would like to fix that)
(can you also provide the crash trace, maybe that could help as well)

Hi EnchantingOstrich20
You how doe s clearml get it there?
In runtime it analyzes the code you are running looking for imports then checks the version you have actively used (i.e. active venv / python) and lists it there.
You can also override those in code, or edit them after you clone the ask and before you enqueue it for remote execution

GiddyTurkey39 what do you have in the Task itself
(i.e. git repo uncommitted changes installed packages)

VictoriousPenguin97 I'm not sure there is an easy solution, basically you have to edit both MongoDB (artifacts) and Elastic (think debug samples) 😞

DeliciousBluewhale87 not on the opensource, for some reason it is not passed 😞
Could you explain the use case ?

I prepared my own image and want use this venv

No worries, it creates a "transparent" venv, it uses everything from the docker (the penalty of create a new venv is negligible 🙂 , you end up with the exact same set of packages)

Ohh... I would not delete them then ... 😞
Maybe kind of heuristics (files created a week ago can be deleted?!)