Yeah I figured (2) would be the way to go actually 😄

Say I have Task A that works with some dataset (which is not hard-coded, but perhaps e.g. self-defined by the task itself).
I'd now like to clone Task A and modify some stuff, but still use the same dataset (no need to recreate it, but since it's not hard-coded, I have to maintain a reference somewhere to the dataset ID).

Since the Dataset SDK offers use_current_task , I would have also expected there to be something like dataset.link(task) or task.register_dataset(ds) ...

The odd thing is that it was already defined, and then when I clicked an S3 link, it asked me to fill it in again, adding a duplicate credentials row

After setting the sdk.development.default_output_uri in the configs, my code kinda looks like:
` task = Task.init(project_name=..., task_name=..., tags=...)
logger = task.get_logger()

report with logger freely `

Basically you have the details from the Dataset page, why should it be mixed with the others ?

Because maybe it contains code and logs on how to prepare the dataset. Or maybe the user just wants increased visibility for the dataset itself in the tasks view.

why would you need the Dataset Task itself is the main question?

For the same reason as above. Visibility and ease of access. Coupling relevant tasks and dataset in the same project makes it easier to understand that they're...

AgitatedDove14
I'll make a PR for it now, but the long story is that you have the full log, but the virtualenv version is not logged anywhere (the usual output from virtualenv just says which Python version is used, etc).

We can change the project name’s of course, if there’s a suggestion/guide that will make them see past the namespace…

I’d like to refrain from manually specifying the dependencies, since it adds a lot of overhead to extend

I've tried also e.g. setting gent.package_manager.priority_packages = ["poetry"] , and/or agent.package_manager.poetry_version = ">1.2.0" , and other flags, but these affect only the main /clearml_agent_venv environment, and not the one actually generated by the clearml-agent when executing the task

I'm not too worried about the dataset appearing (or not) in the Datasets tab. I would like it (the original task ) to to not disappear from the original project I assigned it to

Thanks!

I'm not entirely sure I understand the flow but I'll give it a go. I have two final questions:
This seems to only work for a single file (weights_path implies a single file, not multiple ones). Is that the case? Why do you see this as preferred to the dataset method we have now? 🤔

It's not exactly "debugging", but rather a description of the generated model/framework (generated with pygraphviz).

I guess it does not do so for all settings, but only those that come from Session()

Opened a matching feature request issue for this -> https://github.com/allegroai/clearml/issues/418

I'm not sure how the decorators achieve that; from the available examples and trials I've done, it seems that:

• Components anyway need to be available when you define the pipeline controller/decorator, i.e. same codebase
• The component code still needs to be self-composed (or, function component can also be quite complex)
• Decorators do not allow any dynamic build, because you must know how the component are connected at decoration time
  With that said, it could be that the provided example...

Also, creating from functions allows dynamic pipeline creation without requiring the tasks to pre-exist in ClearML, which is IMO the strongest point to make about it

Yes, using this extra_clearml_conf parameter you can add configuration

This is again exposing the environment variables on the WebUI for everyone to see.
The idea was to specify just the names of the environment variables, and that those would be exposed automatically to the EC2 instance, without specifying what values they should have (the value is taken from the agent running the scaler)

Yes, I’ve found that too (as mentioned, I’m familiar with the repository). My issue is still that there is documentation as to what this actually offers.
Is this simply a helm chart to run an agent on a single pod? Does it scale in any way? Basically - is it a simple agent (similiar to on-premise agents, running in the background, but here on K8s), or is it a more advanced one that offers scaling features? What is it intended for, and how does it work?

The official documentation are very spa...

Note that it would succeed if e.g. run with pytest -s

i.e.
ERROR Fetching experiments failed. Reason: Backend timeout (600s)
ERROR Fetching experiments failed. Reason: Invalid project ID

Much much appreciated 🙏

1.3.2

With clearml-agent 1.5.1

(in the current version, that is, we’d very much like to use them obviously :D)

The new task is not running inside a new subprocess. Our platform trains several models, and we'd like each of them to be tracked in their own Task . When running locally, this is "out of the box", as we can init and close before and after each model.
When running remotely, one cannot close the main task (since it is what orchestrates everything), and so this workaround was needed.

Worked like a charm

I believe that a Pipeline should have the system tags ( pipeline , maybe hidden ), even if it created in a running Task .

Looks good! Why is it using an OutputModel and an InputModel?

Ultimately we're trying to avoid docker in AWS autoscaler (virtualization on top of virtualization seems redundant), and instead we maintain an AMI for a faster boot sequence.
We had no issues when we used pip , but now when trying to work with poetry all these issues came up.
The way I understand poetry to work, is that it is expected there is one system-wide installation that is used for virtual environment creation and manipulation. So at least it may be desired that the ...