why is pushing into the services queue required ...

The services queue is usually connected with an agent running in "services mode" which means this agent is executing multiple tasks in parallel (as opposed to regular agent that only launches one Task at a time, the assumption is that "service" Tasks are usually not heavy on cpu/ram so multiple instances make sense)

It seems like the web server doesn’t log the call to AWS, I just see this:

This points to the browser actually sending the AWS delete command. Let me check with FE tomorrow

What do you have in the artifacts of this task id: 4a80b274007d4e969b71dd03c69d504c

The system denies my deletion requiest since it deems the venv-builds dir as in use

Sorry, yes you have to take down the agent when you delete the cache 🙂

is that what you want to happen ?

is it a shared network mount ? could you just delete the entire ~/.clearml on the host machine ?

The reasoning is that most likely simultaneous processes will fail on GPU due to memory limit

ThickDove42 sorry, it took some time 🙂
import json from trains.backend_api.session.client import APIClient client = APIClient() events = client.events.get_task_plots(task='task_id_here') table = json.loads(events.plots[0]['plot_str']) print('column order', table['data'][0]['cells']['values'])Not the most comfortable way, but at least it is there

Nice SubstantialElk6 !
BTW: you can configure your cleaml client to store the changes from the latest Pushed commit (and not the default which is latest local commit)
see store_code_diff_from_remote: in clearml.conf:
https://github.com/allegroai/clearml/blob/9b962bae4b1ccc448e1807e1688fe193454c1da1/docs/clearml.conf#L150

Hi SubstantialElk6
try:
--docker "<image_name> --privileged"Notice the quotes

👍

Hi WackyRabbit7
I believe this is fixed in clearml-server 1.1 (this is a plotly color issue), releasing later today or tomorrow 🙂

I'm trying to achieve a workflow similar to the one

You mean running everything on a single machine (manually)?

. Would you have any suggestions about where I could look to debug? Maybe the docker logs of the web server?

Let me check, we had the same issue reported today, Let me double check with front-end people and get back to you

Yes you can 🙂 (though not on the open-source version)

Hi GrievingTurkey78

How can I check the server dashboard to make sure everything is working? I have tried to access the external ip through https but the browser is not able to connect.

What do you mean by the server dashboard ?

regrading (2) see here: https://allegro.ai/docs/faq/faq/#web-auth

on the host machine or inside the containers that are spinning on the host machine ?

but when I run the same task again it does not map the keys.. (edited)

SparklingElephant70 what do you mean by "map the keys" ?

Thank you GreasyPenguin14 , I think you are correct, in offline mode it should not check the "demo server" configuration (as it will not try to connect to a server anyhow).
Could you open a github issue? so this issue is addressed quickly

it does appear on the task in the UI, just somehow not repopulated in the remote run if it’s not a part of the default empty dict…

Hmm that is the odd thing... what's the missing field ? Could it be that it is failing to Cast to a specific type because the default value is missing?
(also, is issue present in the latest clearml RC? It seems like a task.connect issue)

Thanks GreasyPenguin14 !

Long story short, not any longer (in previous versions of k8s it was possible, but after the runtime container change it is not supported)

no available 😞

After you call task.set_initial_iteration(0) what do you get with task.get_initial_iteration() , is it 0 ?

Hi GiddyTurkey39
us the config file connect to the Task via Task.connect_configuration ?

Can you please tell me how to return the folder where the script should run?

add it to the python path

PYTHONPATH="/src/project"

Can I change the parameters before executing the draft task

Yes you can, after you clone the experiment everything becomes editable, so you can edit the config in the UI.
For example, let's assume I have config.yml, and in my code I do:
my_file = task.connect_configuration('config.yml') with open(my_file, 'rt') as f: ...Then after I clone it in the UI and edit the configuration, when it will be executed remotely,
my_file will contain the content of the configuration as s...

@<1657918706052763648:profile|SillyRobin38> out of curiosity did you compare performance of tensorrt-llm vs vllm ?
(the jury is still out on that, just wondered if you had a chance)

LOL yes 🙂
just make sure it won't be part of the uncommitted changes of the AWS autoscaler 😉

poetry

 stores git related data in ... you get an internal package we have with its version, but no git reference, i.e. 

internal_module==1.2.3

 instead of 

internal_module @H4dr1en

This seems like a bug with poetry (and I think I have run into this one), worth reporting it, no?