Does that make sense CostlyOstrich36 ? Any thoughts on how to treat this? For the time being I'm also perfectly happy to include something specific to extra_clearml_conf , but I'm not sure how to set the sdk.aws.s3.credentials to be a list of dictionaries as needed

Yes, for example.

Oh and clearml-agent==1.1.2

Sorry, been away for a while!
I have no additional information, since it was a bug in my model that I have since eliminated...
Maybe it was just a matplotlib error and can be dropped for now. I'll let you know if it pops up again!

Trying now with 1.4.1, but I believe the changes you're referring to SuccessfulKoala55 were also introduced in 1.4.0, right?

Basically when there are occasionally extreme values (i.e. most values fall in [0, 50] range, and one value suddenly falls in 50e+12 range), the plotting library (matplotlib or ClearML, unsure) hangs for a really long time

Hm, this didn't happen until now; I'd be happy to try again with a new version, but something with 1.4.0 broke our StorageManager, so we reverted to 1.3.2

A couple of minutes or so

AFAICS it's quite trivial implementation at the moment, and would otherwise require parsing the text file to find some references, right?
https://github.com/allegroai/clearml/blob/18c7dc70cefdd4ad739be3799bb3d284883f28b2/clearml/task.py#L1592

Erm, one sec, let me find that thread...

Still happening with 1.4.1 SuccessfulKoala55

https://clearml.slack.com/archives/CTK20V944/p1652350709961419

Based on where it hangs, I think this has to do with reporting the scatter plots.

Yes it is

I'll reply there 👌

Not sure if @<1523701087100473344:profile|SuccessfulKoala55> or @<1523701827080556544:profile|JuicyFox94> maybe knows?

I was thinking of using the --volume settings in clearml.conf to mount the relevant directories for each user (so it's somewhat customizable). Would that work?

It would be amazing if one can specify specific local dependencies for remote execution, and those would be uploaded to the file server and downloaded before the code starts executing

Thanks for investigating this SweetBadger76 !

Any updates on this? We can't do anything with our K8s since this 404...

Thanks for the reply @<1523701827080556544:profile|JuicyFox94> ! I'll debug more and let you know

And last but not least, for dictionary for example, it would be really cool if one could do:
my_config = task.connect_configuration(my_config, name=name) my_other_config = task.connect_configuration(my_other_config, name=other_name) my_other_config['bar'] = my_config # Creates the link automatically between the dictionaries

Hey @<1523701205467926528:profile|AgitatedDove14> , thanks for the reply!

We would like to avoid dockerizing all our repositories. And for the time being we have not used the decorators, but we can do that too.
The pipeline is instead built dynamically at the moment.

The issue is that the components do not have their dependency. For example:

def step_one(...):
    from internal.repo import private
    # do stuff

When step_one is added as a component to the pipeline, it does ...

Okay trying again without detached

These are per-user. Essentially we log user DB access as well (for various backtracking afterwards), so it's beneficial for us to pass the user DB secrets to the task and not have it configured once on the agent.

I believe that happens natively thanks to pyhocon? No idea why it fails on mac

True, and we plan to migrate to pipelines once we have some time for it :) but anyway that condition is flawed I believe

Actually, it appears some elements (scalars, plots, etc) have no migrated by moving mongodb data.
Where are these stored? Any idea @<1523701827080556544:profile|JuicyFox94> ?

Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC

How or why is this the issue? I great something is getting lost in translation :D
On the local machine, we have all the packages needed. The code gets sent for remote execution, and all the local packages are frozen correctly with pip.
The pipeline controller task is then generated and executed remotely, and it has all the relevant packages.
Each component it launches, however, is missing the internal packages available earlier :(

I'm trying to build an easy SDK that would fit DS work and fit the concept of clearml pipelines.
In doing so, I'm planning to define various Step classes, that the user can then experiment with, providing Steps as input to other steps, etc.

Then I'd like for the user to be able to run any such step, either locally or remotely. Locally is trivial. Remotely is the issue. I understand I'll need to upload additional data to the remote instance, and pull a specific artifact back to the notebo...