Sounds good to me. DepressedChimpanzee34 any chance you can add a github feature request, so we do not forget to add it?

VexedCat68 yes 🙂 you can also pass the parent folder and it will zip the entire subfolders into a single artifact

I think I know what the prob is

DilapidatedDucks58 long story short:
if you do:
` from clearml import StorageManager
from clearml.storage.helper import StorageHelper

StorageHelper.get(" ", retries=5) `It should make sure that all the other s3:// links of this bucket will use the same original configuration (i.e. retries)

If this workaround works let's make sure we add it into the conf file, wdyt ?

OddAlligator72 what you are saying is, take the repository / packages from the runtime, aka the python code calling the "Task.create(start_task_func)" ?
Is that correct ?
BTW: notice that the execution itself will be launched on other remote machines, not on this local machine

None

sorry typo client.task. should be client.tasks.

let's call it an applicative project which has experiments and an abstract/parent project, or some other name that group applicative projects.

That was my way of thinking, the guys argued it will soon "deteriorate" into the first option :)

neat! please update on your progress, maybe we should add an upgrade section once you have the details worked out

Hi PunyGoose16 ,
I think the website is probably the easiest 🙂
https://clear.ml/contact-us/
I think they get back to quite quickly

How are you setting it up?

I see...
Current (and this will change soon) the entire delta is stored in a single file, so there is no real way to download a "subset" of the data, only a parent version 😞

Lets say that this small dataset has a ID ....

Yes this would be exactly the way to do so:

` param ={'dataset': small_train_dataset_id_here}
task.connect(param)

dataset_folder = Dataset.get(param['dataset']).get_local_copy()
... Locally it will use the small_train_dataset_id_here ` , then whe...

Hi @<1524560082761682944:profile|MammothParrot39>
The traditional solution is git submodules, basically main repo links to other repos. This way the agent can fully reproduce the full env.
Another option is to install the second repo as Python package with link to the repo and commit
And a third option is having the second repo as part of the docker.
Regrading env variables, you can add '-e env=val' as part if the docker arts section
Wdyt?

Won't it be too harsh to have system wide restriction like that ?

. Can I get gpu usage over time frame via API also?

task.get_reported_scalarsBut this will get you All the scalars, I think the next version of the server supports asking a specific one as well.
How are you implementing the alert monitoring?
Is is a stateless process starting every X min, or is it a state-full process running and monitoring ?

👍

YEYYYYYYyyyyyyyyyyyyyyyyyy

Maybe add http://requirements.tt to your repo?

Hi SparklingElephant70

Anyone know how to solve?
I tired git push before,

Can you send the entire log? Could it be that the requested commit ID does not exist on the remote git (for example force push deleted it) ?

YEY

I have to admit mounting it to a different drive is a good reason to bring this feature back, the reasoning was it means the agent needs to make sure it manages them (e.g. multiple agents running on the same machine)

but not as a component (using the decorator)

Hmm yes, I think that component calling component as an external component is not supported yet
(basically the difference is , is it actually running as a function, or running on a different machine as another pipeline component)

I noticed that when a pipeline step returns an instance of a class, it tries to pickle.

Yes this is how the serialization works, when we pass data from one node to another (by design it supports multiple mach...

Hi @<1610083503607648256:profile|DiminutiveToad80>
I think we will need more context for the log...
but I think there is something wrong with the GCP resource configuration of your autoscaler
Can you send the full autoscaler log and the configuration ?

JitteryCoyote63 it should just "freeze" after a while as it will constantly try to resend logs. Basically you should be fine 🙂
(If for some reason something crashed, please let me know so we can fix it)

🤞

Hi @<1523703472304689152:profile|UpsetTurkey67>

I circumvented the problem by putting timestamp in task name, but I don't think this is necessary.

Just pass reuse_last_task_id=False to Task.init, it will never try to reuse them 🙂
None

Okay, let me check soemthing

1st: is it possible to make a pipeline component call another pipeline component (as a substep)

Should work as long as they are in the same file, you can however launch and wait any Task (see pipelines from tasks)

2nd: I am trying to call a function defined in the same script, but unable to import it. I passing the repo parameter to the component decorator, but no change, it always comes back with "No module named <module>" after my

from module import function

c...

Sure, to get what exactly ?

Something is off here ... Can you try to run the TB examples and the artifacts example and see if they work?
https://github.com/allegroai/clearml/blob/master/examples/frameworks/tensorflow/tensorflow_mnist.py
https://github.com/allegroai/clearml/blob/master/examples/reporting/artifacts.py