I thought so too - so I added flush calls just in case, but nothing's changed.
This is somewhat weird since it always happens in the above scenario (Ray + ClearML), and always in the last task/job from Ray

Thought it might be via docker, thanks!

ClearML 1.1.4, Matplotlib 3.3.0 (it's not the latest as we have some backward compatibility issues)

For now this is okay - no data lost, really - but I'd like to make sure we're not missing any steps in the next upgrade

Note that it would succeed if e.g. run with pytest -s

But since this has come up a lot recently, any updates on #340? 😍

proj_suffix = "" i = 2 while Task.get_project_id(f"{proj_name}{proj_suffix}") is not None: tasks = Task.get_tasks(project_name=f"{proj_name}{proj_suffix}") if not [task for task in tasks if not task.get_archived()]: # Empty project, we can use this one... break proj_suffix = f"_{i}" i += 1

And task = Task.init(project_name=conf.get("project_name"), ...) is basically a no-op in remote execution so it does not matter if conf is empty, right?

Maybe they shouldn't be placed under /tmp if they're mission critical, but rather the clearml cache folder? 🤔

Woot! What about clearml-agent 1.2.0? 🙂

Another example - trying to validate dataset interactions ends with

` else:
self._created_task = True
dataset_project, parent_project = self._build_hidden_project_name(dataset_project, dataset_name)
task = Task.create(
project_name=dataset_project, task_name=dataset_name, task_type=Task.TaskTypes.data_processing)
if bool(Session.check_min_api_server_version(Dataset.__min_api_version)):
get_or_create_proje...

Right and then for text (file path) use some regex or similar for extraction, and for dictionary simply parse the values?

I would expect the service to actually implicitly inject it to new instances prior to applying the user's extra configuration 🤔

Indeed with ~ the .root call ends with an empty string, so it has a bit of different flow

Maybe https://github.com/awslabs/amazon-ecr-credential-helper ?

I think -

• Creating a pipeline from tasks is useful when you already ran some of these tasks in a given format, and you want to replicate the exact behaviour (ignoring any new code changes for example), while potentially changing some parameters.
• From decorators - when the pipeline logic is very straightforward and you'd like to mostly leverage pipelines for parallel execution of computation graphs
• From functions - as I described earlier :)

So caching results for steps with the same arguments is trivial. Ultimately I would say you can combine the task-based pipeline with a function-based pipeline to achieve such dynamic control as you specified in the first two scenarios.

About the third scenario I'm not sure. If the configuration has changed, shouldn't the relevant steps (the ones where the configuration changed and their dependent steps) be rerun?

At any case, I think if you stay away from the decorators, at the cost of a bi...

We have a mini default config (if you remember from a previous discussion we had) that actually uses the second form you suggested.
I wrote a small "fixup" script that combines this default with the one generated by clearml-init , and it simply does:
def_config = ConfigFactory.parse_file(DEF_CLEARML_CONF, resolve=False) new_config = ConfigFactory.parse_file(new_config_file, resolve=False) updated_new_config = ConfigTree.merge_configs(new_config, def_config)

• in the second scenario, I might have not changed the results of the step, but my refactoring changed the speed considerably and this is something I measure.
• in the third scenario, I might have not changed the results of the step and my refactoring just cleaned the code, but besides that, nothing substantially was changed. Thus I do not want a rerun.Well, I would say then that in the second scenario it’s just rerunning the pipeline, and in the third it’s not running it at all 😄
  (I ...

I don't think there's a PR issue for that yet, at least I haven't created one.

I could have a look at this and maybe make a PR.
Not sure what would the recommended flow be like though 🤔

TimelyPenguin76 CostlyOstrich36 It seems a lot of manual configurations is required to get the EC2 instances up and running.
Would it not make sense to update the autoscaler (and example script) so that the config.yaml that's used for the autoscaler service is implicitly copied to the EC2 services, and then any extra_clearml_conf are used/overwritten?

Why not give ClearML read-only access credentials to the repository?

Ubuntu 18.04, latest clearml version

You mean the host is considered the bucket, as I wrote in my earlier message as the root cause?

Seems up and running

It's pulled from the remote repository, my best guess is that the uncommitted changes apply only after the environment is set up?

@<1537605940121964544:profile|EnthusiasticShrimp49> It’ll take me still some time to find the MVC that generated this, but I do have the ClearML experiment page for it. I was running the thing from ipython , and was trying to create a task from a function:

Alternatively, it would be good to specify both some requirements and auto-detect 🤔

@<1523701205467926528:profile|AgitatedDove14> this

I will TIAS, but maybe worthwhile to also mention if it has to be the absolute path or if relative path is fine too!