Hi VexedCat68
Are we talking youtubes ? docs? courses ?

that's the downside

Are you sure you added the pytorch channel in clearml.conf ?
https://github.com/allegroai/clearml-agent/blob/822984301889327ae1a703ffdc56470ad006a951/docs/clearml.conf#L64

Hi RoundMosquito25

however they are not visible either in:

But can you see them in the UI?

After testing the code again, I see the task parameter dictionary has been removed properly

Great!

However, I still have the same problem with duplicate tasks, as you can see in the image.

Any chance the pipeline script Itself is running from the agent (as opposed to running the pipeline code locally, then the pipelines are executed on the agent)?

So what will you query ?

If this doesn't help.
Go to your ~/clearml.conf file, at the bottom of the file you can add agent.python_binary and change it to to the location of python3.6 (you can run which python3.6 to get the full path):
agent.python_binary: /full/path/to/python3.6

Hi ExcitedCat13
Sure, download the plugin from the git repo (Install instructions in the repo).
Regarding remote debugging, are referring to ssh ?
The plugin itself is designed to make sure that when you work on a remote machine with pycharm clearml will log the local git repo and changes (as the .git folder is not synced to the remote machine)

Hi ScaryKoala63
Which versions are you using (clearml / lightning) ?

What is the recommended way of providing S3 credentials to cleanup task?

cleaml.conf or OS environment (AWS_ACCESS_KEY_ID ...)

Hi @<1541954607595393024:profile|BattyCrocodile47>

Has anyone used ClearML for this use case?

you mean as experiment management / model registry / data? I think this is the bread&butter of clearml 🙂
regrading the other options ion the list, I think most of them are alternatives to metaflow, not covering the parts you mentioned, no?

Yes it does. I'm assuming each job is launched using a multiprocessing.Pool (which translates into a sub process). Let me see if I can reproduce this behavior.

None
This example ?

Hi ScaryLeopard77
I think the error message you are getting is actually "passed" from Triton. Basically someone needs to tell it what the Model in/out look like (matrix size/type) this is essentially the content of the "config.pbtxt" , and this has to be set when spinning the model endpoint. does that make sense to you?

SubstantialElk6
Regrading cloning the executed Task:
In the pip requirements syntax, "@" is a hint that tells pip where to find the package if it is not preinstalled.
Usually when you find the @ /tmp/folder It means the packages was preinstalled (usually pre installed in the docker).
What is the exact scenario that caused it to appear (this was always the case, before v1 as well).
For example zipp package is installed from pypi be default and not from local temp file.
Your fix b...

Regarding the agent - No particular reason. Can you point me on how to do it?

This is a good place to start
https://clear.ml/docs/latest/docs/getting_started/mlops/mlops_first_steps

We need the automagic... 

This is one of the great benefits of using clearml 

😄
Sure, try this one:
Task.debug_simulate_remote_task('reused_task_id') task = Task.init(...)Notice it will take the arguments form the cleaml-task itself (e.g. override argparse arguments with what ...

Sorry ScaryLeopard77 I missed the reply,

the tutorial in the readme of clearml-serving repo doesn't mention it though. Where should I set it?

oh dear ... you are right (I think it was there in previous versions)
clearml-serving --helphttps://github.com/allegroai/clearml-serving/blob/ce6ec847b1e01c6f5bf35d638e6ceb8148db8a7a/clearml_serving/main.py#L142
This is the equivalent of what is created here in the example:
https://github.com/allegroai/clearml-serving/blob/ce6ec847b...

(BTW: any reason not to use the agent?)

Run ifconfig

correct

Getting the last checkpoint can be done via.
Task.get_task(task_id='aabbcc').models['output'][-1]

FierceRabbit20 it seems the Pipeline Task that was created is missing the "installed requirements" section. How are you creating the actual pipeline Task? is this from code?

Hi RoughTiger69

but still get the semantics of knowing when an (external) file changed?

How would you know it changed?
This implies you have a way to verify hash, which means you download the data , no?

Hi @<1541954607595393024:profile|BattyCrocodile47>

Can you trigger a pre-existing Pipeline via the ClearML REST API?

Yes

'd want to have a Lambda function trigger the Pipeline for a batch without needing to have all the Pipeline code in the lambda function.

Easiest is to use clearml SDK, which basically is clone / enqueue (notice that pipeline is also a kind of a Task). See here: [None](https://github.com/allegroai/clearml/blob/3ca6900c583af7bec18792a4a92592b94ae80cac/example...

SmugOx94

after having installed 

numpy==1.16

 in the first case or 

numpy==1.19

 in the second case. Is it correct?

Correct

the reason is simply that I'd like to setup an MLOps system where

I see the rational here (obviously one would have to maintain their requirements.txt)
The current way trains-agent works is that if there is a list of "installed packages" it will use it, and if it is empty it will default to the requirements.txt
We cou...

Hi @<1541954607595393024:profile|BattyCrocodile47>

I

do

have the SSH key placed at

/root/.ssh/id_rsa

on the machine,

Notice that the .ssh folder is mounted from the host (EC2 / GCP) into the container,

'-v', '/tmp/clearml_agent.ssh.cbvchse1:/.ssh'

This is odd, why is it mounting it to /.ssh and not /root/.ssh ?

Hi @<1566959357147484160:profile|LazyCat94>
So it seems the arg parser is detecting the configuration YAML
The first thing I would suggest is changing it to a relative path (so that when launched on remote machines it will find the YAML file)

Regardless how are you launching the HPO ? are you spinning a new agent ?
(as background, argparser arguments are injected in realtime by the agent or the HPO when running as subprocesses)

Okay good news, there is a fix, bad news, sync to GitHub will only be tomorrow

Hi AbruptWorm50

I was wondering if it possible to specify 'patience' of pruning algorithm?

Any of the kwargs passed to **optimizer_kwargs will be directly passed to the optuna obejct
https://github.com/allegroai/clearml/blob/2e050cf913e10d4281d0d2e270eea1c7717a19c3/clearml/automation/optimization.py#L1096

It should allow you to control the parameters, no?

Regrading the callback, what exactly do you think to put there?
Is the callback this enough?
https://github.com/allegro...

Okay, let's take a step back and I'll explain how things work.
When running the code (initially) and calling Task.init
A new experiment is created on the server, it automatically stores the git repo link, commit ID, and the local uncommitted changes . these are all stored on the experiment in the server.
Now assume the trains-agent is running on a different machine (which is always the case even if it is actually on the same machine).
The trains-agent will create a new virtual-environmen...