I believe that a Pipeline should have the system tags ( pipeline , maybe hidden ), even if it created in a running Task .

Happens pretty much consistently across all our projects -
Have a project with over 15 tasks (i.e. one that needs the Load More button) Click Load More, select a task that's not in the first 15 Let the page "rest" for a while (a couple of hours) Flip back to the page - the task is still active, but you cannot see it in the task list and there is no more Load More button

I have selected the correct project indeed

TimelyPenguin76 I added pip install --update clearml-agent to the extra_vm_bash_script for the autoscaler, that should at least guarantee the latest clearml agent is used on the instance, right?

Yes exactly that AgitatedDove14
Testing our logic maps correctly, etc for everything related to ClearML

Is Task.create the way to go here? 🤔

I'll give it a shot. Honestly, the SDK documentation for both InputModel and OutputModel is (sorry) horrible ...

Can't wait for the documentation revamping.

I've been answering there as well 🤕

https://github.com/allegroai/clearml/pull/533 🙂

Yes, you're correct, I misread the exception.
Maybe it hasn't completed uploading? At least for Datasets one needs to explicitly wait IIRC

Still crashing, I think that may not be the correct virtual environment to edit 🤔
It's the one created later down the line

It does not 🙂
We started discussing it here - https://clearml.slack.com/archives/CTK20V944/p1640955599257500?thread_ts=1640867211.238900&cid=CTK20V944
You suggested this solution - https://clearml.slack.com/archives/CTK20V944/p1640973263261400?thread_ts=1640867211.238900&cid=CTK20V944
And I eventually found this solution to work - https://clearml.slack.com/archives/CTK20V944/p1641034236266500?thread_ts=1640867211.238900&cid=CTK20V944

AgitatedDove14 I will try! I remember there were some issues with it, where I had to resort to this method first, but maybe things have changed since :)

No need for S3 or other resources?

Okay trying again without detached

So the ..data referenced in the example above are part of the git repository?
What about setting the working_directory to the user working directory using Task.init or Task.create ?

Seemed to work fine again in detached mode, what went wrong there :shocked_face_with_exploding_head:

Sorry, found it on my end!

If I add the bucket to that (so CLEARML_FILES_HOST= s3://minio_ip:9000/minio/bucket ), I then get the following error instead --

2021-12-21 22:14:55,518 - clearml.storage - ERROR - Failed uploading: SSL validation failed for ... [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1076)

Does that make sense SmugDolphin23 ?

It also happens when use_current_task=False though. So the current best approach would be to not combine the task and the dataset?

Say I upload each of these yamls as a configuration object (as with the above). Once I try to load bar.yaml remotely it will crash, since foo.yaml is missing (and is instead a clearml configuration object).
Does that make sense?

I have no tried to download the entire file, but yes

Yeah I figured (2) would be the way to go actually 😄

I'll have a look at 1.1.6 then!
And that sounds great - environment variables should be supported everywhere in the config, or then the docs should probably mention where they are and are not supported 🙂

I'll be happy to test it out if there's any commit available?

The thing I don't understand is how come this DOES work on our linux setups 🤔

I'll have yet another look at both the latest agent RC and at the docker-compose, thanks!

There was no "default" services agent btw, just the queue, I had to launch an agent myself (not sure if it's relevant)

I guess following the example https://github.com/allegroai/clearml/blob/master/examples/advanced/execute_remotely_example.py , it's not clear to me how the server has access to the data loaders location when it hits execute_remotely

We're using the example autoscaler, nothing modified

If everything is managed with a git repo, does this also mean PRs will have a messy metadata file attached to them?

I've tried also e.g. setting gent.package_manager.priority_packages = ["poetry"] , and/or agent.package_manager.poetry_version = ">1.2.0" , and other flags, but these affect only the main /clearml_agent_venv environment, and not the one actually generated by the clearml-agent when executing the task