So what is the mechanism that you "automagically" pick things up (for information, I don't think this is relevant to our usecase)

If you use joblib.dump (which is like pickle but safer/faster) it will be auto logged
https://github.com/allegroai/clearml/blob/4945182fa449f8de58f2fc6d380918075eec5bcf/examples/frameworks/scikit-learn/sklearn_joblib_example.py#L28

Thanks @<1523701713440083968:profile|PanickyMoth78> for pining, let me check if I can find something in the commit log, I think there was a fix there...

let me know if it works 🙂

is it a good "I see" ? 🙂

. I am not sure this is related to the fact the model is not correctly converted to TorchScript

Because Triton Only supports TorchScript (Not torch models) 🙂

I think the limit is a few GB, I'm not sure, I'll have to check
And yes the oldest experiments will be deleted first (with the exception of published experiments, they will be deleted last)

I can then programmatically choose which file to import with importlib. Is there a way to tell clearml programmatically to analyze the files, so it can built up the requirements correctly?

Sadly no 😞
It analyzes the running code, then if it decides it is not a self contained script it will analyze the entire repo ...

I just saw that 

Task.create

 takes

Task.create is Not Task.init. It is meant to allow you to create new Tasks (think Jobs) from ...

Are you doing from keras import ... or from tensorflow.keras import ?

Are the key/secret the one you see in the UI?

Exactly, that’s my problem: I want to remove it to make sure it is reinstalled (because the version can change)

JitteryCoyote63 yes, this is definitely a pip bug... can you test with the latest pip version, maybe it was fixed? (i.e. git+https:// link)

PompousBeetle71 so basically exclude parameters that are considered "local" only, so that other people will not accidentally use them?

If we have the time maybe we could PR a fix?!

Could you post what you see under "installed packages" in the UI ?

PompousBeetle71 , what you are saying that for some reason the --gpus all will not configure the Nvidia drivers to use all the gpus, when running bare metal (i.e no docker). Did I understand you correctly ?

I think the real issue is that I am not able to specify a platform for the model,

None
there is no need to specify it, remove it from the config.pbtxt - the clearml-serving will automatically add the background

Glad to hear 🙂

error in my-package setup command:

Okay this seems like an error in the setup.py you have in the "mypackage" folder

Try:
task.update_requirements('\n'.join([".", ])) 

JitteryCoyote63 see here https://stackoverflow.com/questions/55385900/pip3-setup-py-install-requires-pep-508-git-url-for-private-repo bottom line, you have to add package@ before the link, but if you do that and the package is already installed it will not install using the git repo, this is an issue with pip. I think that since the agent installs everything from scratch it should work for you. Wdyt?

Hi GrittyKangaroo27
Maybe check the TriggerScheduler , and have a function trigger something on k8s every time you "publish" a model?
https://github.com/allegroai/clearml/blob/master/examples/scheduler/trigger_example.py

IrritableGiraffe81 could it be the pipeline component is not importing pandas inside the function? Notice that a function decorated with pipeline component become a stand-alone, this means that if you need pandas you need to import inside the function. The same goes for all the rest of the packages used.
When you are running with run_loclly or debug_pipeline you are using your local env , as opposed to the actual pipeline where a new env is created inside the repo.
Can you send the Entire p...

Hi JuicyDog96
The easiest way at the moment (apologies for still lack of RestAPI documentation, it is coming:)
Is actually the code (full docstring doc)
https://github.com/allegroai/trains/tree/master/trains/backend_api/services/v2_8
You can access it all with an easy Pythonic interface, for example:
from trains.backend_api.session.client import APIClient client = APIClient() tasks = client.tasks.get_all()

how come the previous gitdiff passed ?

SoggyBeetle95 the question is, where does clearml stores these arguments, and the answer is on the Task object (from there the agent will take them and apply to the docker execution). Now since all users see all the tasks, they also see these arguments. Wdyt?

Honestly, this is all related to issue #340.

makes total sense.
But actually this id different from #340. The feature is to store the Data on the Task, this means each Task in your "pipeline" will be upload a new copy of the data. No?

I'd suggest some 

task.detach()

 method for remote execution maybe

That is a good idea, in theory it can also be used in local execution

Hi RoughTiger69

One quirk I found was that even with this flag on, the agent decides to install whatever is in the requirements.txt

Whats the clearml-agent you are using?

I just noticed that even when I clear the list of installed packages in the UI, upon startup, clearml agent still picks up the requirements.txt (after checking out the code) and tries to install it.

It can also just skip the entire Python installation with:
CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1

Hi RoughTiger69
Is the pipeline in question based on decorators or is it based on existing Tasks?

Could be nice to write some automation

@<1541954607595393024:profile|BattyCrocodile47> not restarting the docker, restarting the Docker service (on Mac it's an app, I think there is an option on the Docker app to do that)

well at this point I'm not sure it is still essential, we have 3 run-modes offline, local-server, cloud-sever and this option made it work for all of them.. can be that it is not required anymore and its just legacy..

LOL, sure if you have so many setups, that makes sense 🙂

this is strange.. you ran it with the dataclass config I added?

Yes but I had to remove the:
from config_files import cfgand instead used:
` @hydra.main(config_path="config_files", config_name="confi...