Is it currently broken? 🤔

AgitatedDove14 the issue was that we'd like the remote task to be able to spawn new tasks, which it cannot do if I use Task.init before override_current_task_id(None) .

When would this callback be called? I'm not sure I understand the usecase.

I've also followed https://clearml.slack.com/archives/CTK20V944/p1628333126247800 but it did not help

... and any way to define the VPC is missing too 🤔

Ah, you meant “free python code” in that sense. Sure, I see that. The repo arguments also exist for functions though.

Sorry for hijacking your thread @<1523704157695905792:profile|VivaciousBadger56>

Setting the endpoint will not be the only thing missing though, so unfortunately that's insufficient 😞

There's code that strips the type hints from the component function, just think it should be applied to the helper functions too :)

Opened this - https://github.com/allegroai/clearml/issues/594

Feels like we've been over this 😄 Has there been new developments perhaps?
It's essentially that this - https://clear.ml/docs/latest/docs/guides/advanced/multiple_tasks_single_process cannot work in a remote execution.

No that does not seem to work, I get

task.execute_remotely(queue_name="default")
2024-01-24 11:28:23,894 - clearml - WARNING - Calling task.execute_remotely is only supported on main Task (created with Task.init)
Defaulting to self.enqueue(queue_name=default)

Any follow-up thoughts, @<1523701070390366208:profile|CostlyOstrich36> , or maybe @<1523701087100473344:profile|SuccessfulKoala55> ? 🤔

Of course Im using report_table in the above; it seems the support for Pandas DataFrame does not include support for MultiIndex other than by concatenating the indices together

That's fine (as in, it works), but it looks a bit weird and defies the purpose of a MultiIndex 🤔 Was wondering if there are plans to add better support for it

I'd like to set up both with and without GPUs. I can use any region, preferably some EU one.

So the pipeline runs successfully, I can find all the different tasks, but I cannot see them in the Pipelines tab…

The title is specified in the plot (see the example, even if small).
I'm just creating a figure normally with matplotlib and save it to disk.

nevermind! Found and answered (solution in the issue linked above)

Also full disclosure - I'm not part of the ClearML team and have only recently started using pipelines myself, so all of the above is just learnings from my own trials 😅

Great, thanks! Any idea about environment variables and/or other files (CSV)? I suppose I could use the task.upload_artifact for the CSVs. but I'm still unsure about the environment variables

No task, no dataset, just an empty container with no reference to the task it's attached.

It seems to me that it should not move the task if use_current_task=True ?

Sure CostlyOstrich36 , sorry it took me so long to reply. I minimized the window a bit here so everything will fill in nicely. Worth mentioning this happens on all pages of course, but I went to the profile page so you can also see the clearml server version.

Different AMI image/installing older Python instances that don't enforce this...
For future reference though, the environment variable should be PIP_USE_PEP517=false

Thanks for your help SuccessfulKoala55 ! Appreciate the patience 🙏

Thanks AgitatedDove14 , I'll first have to prove viability with the free version :)

StorageManager.download_folder(remote_url=' s3://some_ip:9000/clearml/my_folder_of_interest ', local_folder='./') yields a new folder structure, ./clearml/my_folder_of_interest , rather than just ./my_folder_of_interest

I narrowed the bug down to the "fix" in 1.8.1, see my other post

It is installed on the pipeline creating the machine.
I have no idea why it did not automatically detect it 😞

I know, that should indeed be the default behaviour, but at least from my tests the use of --python ... was consistent, whereas for some reason this old virtualenv decided to use python2.7 otherwise 🤨

SuccessfulKoala55 TimelyPenguin76
After looking into it, I think it's because our AMI does not have docker, and that the default instance suggested by ClearML auto scaler example is outdated

Sorry AgitatedDove14 , forgot to get back to this.
I've been trying to convince my team to drop poetry 😄

Yes, thanks AgitatedDove14 ! It's just that the configuration object passed onwards was a bit confusing.
Is there a planned documentation overhaul? 🤔

We have an internal mono-repo and some of the packages are required - they’re all available correctly for the controller, only some are required for the individual tasks, but the “magic” doesn’t happen 😞
That is, the controller does not identify them as a requirement, so they’re not installed in the tasks environment.