sure

Hi TrickyFox41

is there a way to cache the docker containers used by the agents

You mean for the apt get install part? or the venv?
(the apt packages themselves are cached on the host machine)
for the venv I would recommend turning on cache here:
https://github.com/allegroai/clearml-agent/blob/76c533a2e8e8e3403bfd25c94ba8000ae98857c1/docs/clearml.conf#L131

If you are using user/pass for the git (i.e. not ssh key) we are not passing it to the pip install (and come to think about it, we probably should?!)

Hi PanickyMoth78
You mean like another Task? or maybe Slack message?

https://stackoverflow.com/questions/67419800/error-cannot-schedule-new-futures-after-interpreter-shutdown-with-boto3-while

CheekyFox58 what do you have in the plots Tab?

ThickDove42 you need the latest cleaml-agent RC for the docker setup script (next version due next week)
pip install clearml-agent==0.17.3rc0

Hmm and how would you imagine a transparent integration here (the example looks like a lot of boilerplate code...)

I can verify the behavior, I think it has to do with the way the subparser was setup.
This was the only way for me to get it to run:
script.py test blah1 blah2 blah3 42When I passed specific arguments (for example --steps) it ignored them...

Hi HelpfulDeer76

I mean that the task was being monitored on the demo ClearML server created by Allegro

Yes that is consistent with what I would expect to have happened
Basically if you are running it as k8s job, you can just configure the following environment variables:
CLEARML_WEB_HOST: CLEARML_API_HOST: CLEARML_FILES_HOST: CLEARML_API_ACCESS_KEY: <clearml access> CLEARML_API_SECRET_KEY: <clearml secret>

Are you asking regrading the k8s integration ?
(This is not a must, you can run the clearml-agent bare-metal on any OS)

Nice catch AverageBee39 🙂

actually the issue is that the packages are not being detected 😞
what happens if you do the following?
Task.add_requirements("tensorflow") task = Task.init(...)

Hi CleanPigeon16
You need to pass the private repository docker credentials to the aws instance, I would use the custom bash script option of the aws autoscaler to create the docker credentials file.

The way ClearML thinks about it is the execution graph would be something like:
script_1 -> script_2 -> script_3 ->

Where each script would have in/out, so that you can trace the usage.

Trying to combine the two into a single "execution" graph might not represent the orchestration process.

That said visualizing them could be done.
I mean in theory there is no reason why we could add those "datasets" as other types of building blocks, for visualization purposes only

(Of course this would o...

Hi FiercePenguin76
Artifacts are as you mentioned, you can create as many as you like but at the end , there is no "versioning" on top , it can be easily used this way with name+counter.
Contrary to that, Models do offer to create multiple entries with the same name and version is implied by order. Wdyt?

My pleasure 🙂
I do not think it should change anything, just pull the latest agent and reinstall
pip3 install -U clearml-agent

Is there any better way to avoid the upload of some artifacts of pipeline steps?

How would you pass "huge datasets (some GBs)" between different machines without storing it somewhere?
(btw, I would also turn on component caching so if this is the same code with the same arguments the pipeline step is reused instead of reexecuted all over again)

LOL RotundHedgehog76 🙂
Is this okay?
https://avatars.githubusercontent.com/u/38647316

This is sitting on top of the serving engine itself, acting a s a control plane.
Integration with GKE is being worked on (basically KFServing as the serving engine)

Thanks SubstantialElk6 !
I believe an initial a fix was pushed 😉 A full one (merging Task --env with k8s template) will be added soon

make sure the API port is 8008 and the web 8080

I still have name

my_name

, but the project name

my_project/.datasets/my_name

rather than

my_project/.datasets

Yes, this is the expected behavior

And I don't see any new projects / subprojects where that dataset creation Task is stored

They are marked "hidden" hence by default you cannot see them in the UI (so they will only appear in the Dataset page),
you can turn the UI hidden flag by going to your settings page and selecting "Con...

SharpDove45 FYI:
if you set the environment variable CLEARML_NO_DEFAULT_SERVER=1 , it will make sure never to default to the demo server

Alright I have a followup question then: I used the param --user-folder “~/projects/my-project”, but any change I do is not reflected in this folder. I guess I am in the docker space, but this folder is not linked to my the folder on the machine. Is it possible to do so?

Yes you must make sure the docker can mount a persistent folder for you to work on.
Let me check what's the easiest way to do that

Will do! 🙂

Hi @<1541954607595393024:profile|BattyCrocodile47>

Can you help me make the case for ClearML pipelines/tasks vs Metaflow?

Based on my understanding

• Metaflow cannot have custom containers per step (at least I could not find where to push them)
• DAG only execution. I.e. you cannot have logic driven flows
• cannot connect git repositories to different component in the pipeline
• Visualization of results / artifacts is rather limited
• Only Kubernetes is supported as underlying prov...

You added it to the deprecated function 🙂
https://github.com/allegroai/clearml/blob/8496b99888deda45955bdd95f22dc929a88ec67b/clearml/backend_config/config.py#L400
See this line
https://github.com/allegroai/clearml/blob/8496b99888deda45955bdd95f22dc929a88ec67b/clearml/backend_config/config.py#L338

Hi @<1619867971730018304:profile|WhimsicalGorilla67>
No 😞 only the "admin" (owner) of the workspace has access to it

I was not able to reproduce with the example code 😞
https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_from_decorator.py