Hi CostlyElephant1
What do you mean by "delete raw data"? Data is always fetched to cached folders and clearml takes care of cache cleanup
That said notice that get mutable copy is a target you specify, in this case you should definetly delete after usage. Wdyt ?

Ohh yes, if the execution script is not on git and git exists, it will not add it (it will add it if it is in a tracked file via the uncommitted changes section)
ZanyPig66 in order to expand the support to your case. Can you explain exactly which files are on git and which are not?

VictoriousPenguin97 I'm not sure there is an easy solution, basically you have to edit both MongoDB (artifacts) and Elastic (think debug samples) 😞

That experiment says it's completed, does it mean that the autoscaler is running or not?

Not running, it will be "running" if actually being executed

My bad you have to pass it to the container itself:
https://github.com/allegroai/clearml-agent/blob/a5a797ec5e5e3e90b115213c0411a516cab60e83/docs/clearml.conf#L149
extra_docker_arguments: ["-e", "CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1"]

1. Yes the challenge is mostly around defining the interface. Regarding packaging, I'm thinking a similar approach to the pipeline decorator, wdyt?
2. Clearml agents will be running on k8s, but the main caveat is that I cannot think of a way to help with the deployment, at the end it will be kubectl that users will have to call in order to spin the containers with the agents, maybe a simple CLI to do that for you?

in the UI the installed packages will be determined through the code via the imports as usual ...

This is only in a case where a user manually executed their code (i.e. without trains-agent), then in the UI after they clone the experiment, they can click on the "Clear" button (hover over the "installed packages" to see it) and remove all the automatically detected packages. This will results in the trains-agent using the "requirements.txt".

Thanks MuddyCrab47 !!!
I found it!
It turns out the artifact upload will always upload from stream (aka no multi-upload). I will make sure we fix it in the next RC (I think the plan is to have it out this weekend)

Hi CluelessFlamingo93
I think the latest clearml-agent 1.5.1 fixed that issue (this is basically pip trying to "protect" you from mismatch packages)
can you upgrade your clearml-agent and test?
pip3 install clearml-agent==1.5.1

Yep, you are right

Hi LovelyHamster1 ,
you mean totally ignore the "installed packages" section, and only use the requirements.txt ?

it seems like each task is setup to run on a single pod/node based on the attributes like

gpu memory

,

os

,

num of cores,

worker

BoredHedgehog47 of course you can scale on multiple node.
The way to do that is to create a k8s Yaml with replicas, each pod is actually running the exact same code with the exact same setup, notice that inside the code itself the DL frameworks need to be able to communicate with one another and b...

Hi SteadyFox10 , this one will get all the last metric scalars
train_logger.get_last_scalar_metrics()

The difference is whether you are only supplying a "minutes" or you are also passing hour/day etc.
See the examples:
Every 15 minutes
add_task(task_id='1235', queue='default', minute=15)Every hour on minute 20 of the hour (i.e. 00:20, 01:20 ...)
add_task(task_id='1235', queue='default', hour=1, minute=20)

Hi TenderCoyote78

I'm trying to clearml-agent in my dockerfile,

I'm not sure I'm following, Are you traying to create a docker container containing the agent inside? for what purpose ?
(notice that the agent can spin any off the shelf container, there is no need to add the agent into the container it will take of itself when it is running it)

Specifically to your docker file:

RUN curl -sSL

| sh

No need for this line

COPY clearml.conf ~/clearml.conf

Try the ab...

This is because we have a pub-sub architecture that we already use, it can handle retries, etc. also we will likely want multiple systems to react to notifications in the pub sub system. We already have a lot of setup for this.

How would you integrate with your current system? you have a restapi or similar to trigger event ?

but I was hoping ClearML had a straightforward way to somehow represent ALL ClearML events as JSON so we could land them in our system.

Not sure I'm followi...

from what I gather there is a lightly documented concept

Yes ... 😞 the reason for it is that actually one could do:
` @PipelineDecorator.pipeline(...)
def pipeline(i):
....

if name == 'main':
pipeline(0)
pipeline(1)
pipeline(2) `Basically rerunning the pipeline 3 times
This support was added as some users found a use case for it, but I think this would be a rare one

Hi @<1529271085315395584:profile|AmusedCat74>

ClearML Scheduler where it doesn't reuse the task

What do you mean by doesn't reuse the Task, do you mean you want each time the scheduler is launched to basically overwrite the previous run ?

Hi CleanWhale17 , at least for the moment, the code although open ( https://github.com/allegroai/trains-web ) has no external theme/customization interface.
That said we do have some thoughts on it.., What did you have in mind ?

I just think that the create function should expect

dataset_name

to be None in the case of

use_current_task=True

(or allow the dataset name to differ from the task name)

I think you are correct, at least we should output a warning that it is ignored ... I'll make sure we do 🙂

Is there a way to connect to the task without initiating a new one without overriding the execution?

You can, but not with automagic, you can manually send metrics/logs...
Does that help? or do we need the automagic?

ReassuredTiger98

Can you explain what you meant by 

entropy point file?

There is no need to specify entry point file.
It is automatically detected when you run the Code manually on your machine.
My assumption was that the file "src/run_task.py" (based on your log) is just a test file, and hence was not added top the repository. So the agent failed to actually restore it from the git (files that are not added are not considered part of the git diff, this is usually git behavio...

Hi LackadaisicalOtter14

However, whenever we spin up a session, 

 always gets run and overwrites our configs

what do you mean by that?
The what config are being overwritten? (generally speaking, it just add the OS environment it needs to for the setup process)

Hi CheekyFox58
If you are running the HPO+training on your own machine, it should work just fine in the Free tier

The HPO with the UI and everything, is designed to run the actual training on remote machines, and I think this makes it a Pro feature.

StaleMole4 you are printing the values before Task.init had the chance to populate it.
Basically try moving the print after closing the Task (closing the tasks waits for the async update)
Make sense ?

Also I would suggest using Task.execute_remotely
https://clear.ml/docs/latest/docs/references/sdk/task#execute_remotely

Hi @<1619505588100665344:profile|GrievingHare27>

My understanding is that initiating a task with

Task.init()

captures the code for the entire notebook. I'm facing difficulties when attempting to build a final training pipeline (in a separate notebook) that uses only certain functions from the other notebooks/tasks as pipeline steps.

Well this is is kind of the limit of working with jupyter notebooks, referencing code from one to another is not really feasible (of co...

5 seconds will be a sleep between two consecutive pulls where there are no jobs to process, why would you increase it to a higher pull freq ?

Hi ThankfulOwl72 checkout TrainsJob object. It should essentially do what you need:
https://github.com/allegroai/trains/blob/master/trains/automation/job.py#L14

Hi SkinnyPanda43

Let's say that I install the shared libs with pip in editable mode on my development evironment, how does the clearml-agent will handle those libraries if I submit a job

So installing packages from local folders with "-e" is in general ill-advised.
But using a full git path should work out of the box. for example if you install pip install https://github.com/user/repo/repo.git then the agent will be able to install it on the remote machine. The main challenge...