Hi ShallowGoldfish8 ,

You can get specific chunks/files using the part argument:
https://clear.ml/docs/latest/docs/references/sdk/dataset#get_local_copy

You can create a queue through the UI. You can go into Workers & Queues tab -> Queues -> "New Queue"
You can also create new queues using the API as well
https://clear.ml/docs/latest/docs/references/api/queues#post-queuescreate

Hi GrievingDeer61 , you need to create the queue yourself or change the queue that is being used to something you created 🙂

Hi TartBear70 ,

Did you run the experiment locally first? What versions of clearml/clearml-agent are you using?

Hi DistressedKoala73 ,

What version of ClearML are you using? Are you using a remote interpreter? You can also connect it manually with https://clear.ml/docs/latest/docs/references/sdk/task#set_script

ResponsiveHedgehong88 , do you have an option to log into the machine and see the state or if there were any errors? Is there any chance it's running out of memory? The agent also keeps a local log, can you take a look there to see if there is any discrepancy?

Aw you deleted your response fast

Yeah I misread the part where it's not in ps aux ^^

If the process is dead it will be removed from the UI after some time

You can contact the sales team via the contact form 🙂

Hi AverageRabbit65 , can you elaborate on what you're trying to do?
ClearML-Agent will automatically create a venv and install everything

I'm sorry. I think I wrote something wrong. I'll elaborate:
The SDK detects all the packages that are used during the run - The Agent will install a venv with those packages.
I think there is also an option to specify a requirements file directly in the agent.

Is there a reason you want to install packages from a requirements file instead of just using the automatic detection + agent?

the question how does ClearML know to create env and what files does it copy to the task

Either automatically detecting the packages in requirements.txt OR using the packages listed in the task itself

The one sitting in the repository

Have you run experiments with the SDK? i.e added Task.init()

And the pipeline runs with agents or locally?

Hi @<1547028031053238272:profile|MassiveGoldfish6> , are you self hosted or on the community server? What project is this, a pipelines/dataset project or just some regular project?

Hi WhoppingMole85 , you can actually do that with the logger.

Something along the lines of:
Dataset.get_logger().report_table(title="Data Sample", series="First Ten Rows", table_plot=data1[:10])

Does this help?

RoughTiger69 , do you have a rough estimate on the size that breaks it?

SoggyBeetle95 , in the ClearML UI you should see a small notification at the top when there is a new version available

Just making sure we cover all bases - you changed updated the optimized to use a base task with _allow_omegaconf_edit_ : True

Hi IrritableJellyfish76 , it looks like you need to create the services queue in the system. You can do it directly through the UI by going to Workers & Queues -> Queues -> New Queue

You can use https://clear.ml/docs/latest/docs/references/sdk/automation_controller_pipelinecontroller#start_locally
to run the pipeline locally

The ValueError is happening because there is no queue called services it appears

Hi GrittyHawk31 , can you elaborate on what you mean by metadata? Regarding models you can achieve this by defining the following in Task.init(output_uri="<S3_BUCKET>")

like some details about attributes, dataset size, formats.

Can you elaborate on how exactly you'd be saving this data?

here when we define output_uri in task_init in which format the model would be saved?

It depends on the framework I guess 🙂

Hi FranticLobster32 , what version of ClearML, of Agent & Hydra are you using?

I mean the version of the SDK

In the task hyper parameters section you have a section called Hydra. In that section there should be a configuration called _allow_omegaconf_edit_ , what is it set to?

Please try setting it to True, that should fix it

Aight. Thanks for the information. I'll take a look and see if it reproduces for me as well 🙂