seems to run properly now

Are you saying the problem disappeared ?

Hi ElegantCoyote26

is there a way to get a Task's docker container id/name?

you mean like Task.get_task("task_id_here").get_base_docker() ?

ow a Task's results page also has a plot for this, but I guess it's at the machine level and not the task level?

This is actually on the container level, meaning checked from inside the container. It should be what you are looking for

Hmm this is odd, could you provide the pipeline code maybe?

currently I'm doing it by fetching the latest dataset, incrementing the version and creating a new dataset version

This seems like a very good approach, how would you improve ?

If it cannot find the Task ID I'm guessing it is trying to connect to the demo server and not your server (i.e. configuration is missing)

So “wait” is a better metaphore for me

So I would do something like (I might have a few typos but that's the gist):

def post_execute_callback_example(a_pipeline, a_node):
    # type (PipelineController, PipelineController.Node) -> None
    print('Completed Task id={}'.format(a_node.executed))
    # wait until model is tagged, then pass it as argument
    while True:
        found = Moodel.query_models(...) # model filter here, inlucing tag and project
        if found:
         ...

Sounds good to me. DepressedChimpanzee34 any chance you can add a github feature request, so we do not forget to add it?

VexedCat68 yes 🙂 you can also pass the parent folder and it will zip the entire subfolders into a single artifact

I think I know what the prob is

DilapidatedDucks58 long story short:
if you do:
` from clearml import StorageManager
from clearml.storage.helper import StorageHelper

StorageHelper.get(" ", retries=5) `It should make sure that all the other s3:// links of this bucket will use the same original configuration (i.e. retries)

If this workaround works let's make sure we add it into the conf file, wdyt ?

OddAlligator72 what you are saying is, take the repository / packages from the runtime, aka the python code calling the "Task.create(start_task_func)" ?
Is that correct ?
BTW: notice that the execution itself will be launched on other remote machines, not on this local machine

None

sorry typo client.task. should be client.tasks.

neat! please update on your progress, maybe we should add an upgrade section once you have the details worked out

Hi PunyGoose16 ,
I think the website is probably the easiest 🙂
https://clear.ml/contact-us/
I think they get back to quite quickly

How are you setting it up?

I see...
Current (and this will change soon) the entire delta is stored in a single file, so there is no real way to download a "subset" of the data, only a parent version 😞

Lets say that this small dataset has a ID ....

Yes this would be exactly the way to do so:

` param ={'dataset': small_train_dataset_id_here}
task.connect(param)

dataset_folder = Dataset.get(param['dataset']).get_local_copy()
... Locally it will use the small_train_dataset_id_here ` , then whe...

Hi @<1524560082761682944:profile|MammothParrot39>
The traditional solution is git submodules, basically main repo links to other repos. This way the agent can fully reproduce the full env.
Another option is to install the second repo as Python package with link to the repo and commit
And a third option is having the second repo as part of the docker.
Regrading env variables, you can add '-e env=val' as part if the docker arts section
Wdyt?

Won't it be too harsh to have system wide restriction like that ?

. Can I get gpu usage over time frame via API also?

task.get_reported_scalarsBut this will get you All the scalars, I think the next version of the server supports asking a specific one as well.
How are you implementing the alert monitoring?
Is is a stateless process starting every X min, or is it a state-full process running and monitoring ?

👍

YEYYYYYYyyyyyyyyyyyyyyyyyy

Maybe add http://requirements.tt to your repo?

Hi SparklingElephant70

Anyone know how to solve?
I tired git push before,

Can you send the entire log? Could it be that the requested commit ID does not exist on the remote git (for example force push deleted it) ?

YEY

I have to admit mounting it to a different drive is a good reason to bring this feature back, the reasoning was it means the agent needs to make sure it manages them (e.g. multiple agents running on the same machine)

but not as a component (using the decorator)

Hmm yes, I think that component calling component as an external component is not supported yet
(basically the difference is , is it actually running as a function, or running on a different machine as another pipeline component)

I noticed that when a pipeline step returns an instance of a class, it tries to pickle.

Yes this is how the serialization works, when we pass data from one node to another (by design it supports multiple mach...

Hi @<1610083503607648256:profile|DiminutiveToad80>
I think we will need more context for the log...
but I think there is something wrong with the GCP resource configuration of your autoscaler
Can you send the full autoscaler log and the configuration ?

JitteryCoyote63 it should just "freeze" after a while as it will constantly try to resend logs. Basically you should be fine 🙂
(If for some reason something crashed, please let me know so we can fix it)

Hi @<1523703472304689152:profile|UpsetTurkey67>

I circumvented the problem by putting timestamp in task name, but I don't think this is necessary.

Just pass reuse_last_task_id=False to Task.init, it will never try to reuse them 🙂
None