Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
GrievingKoala83
Moderator
19 Questions, 42 Answers
  Active since 11 June 2023
  Last activity 4 days ago

Reputation

0

Badges 1

40 × Eureka!
0 Votes
5 Answers
367 Views
0 Votes 5 Answers 367 Views
Hi everyone! I'm trying to use task.launch_multi_node(nodes, devices=gpus, hide_children=True) in conjunction with pytorch-ligtning. I am using the latest ve...
3 months ago
0 Votes
5 Answers
73 Views
0 Votes 5 Answers 73 Views
Hello! Can you help me with Model Endpoints tab - how to connect it to existing clearml-serving instance?
11 days ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
one year ago
0 Votes
2 Answers
328 Views
0 Votes 2 Answers 328 Views
hello everyone! is it possible to transfer data (datasets, models) from one ClearML instance to another? How can I do this?
5 months ago
0 Votes
1 Answers
119 Views
0 Votes 1 Answers 119 Views
Hello everyone! I need to run the pipeline on schedule. Are there any restrictions on running pipelines through the scheduler? Would hidden project for pipe ...
one month ago
0 Votes
1 Answers
718 Views
0 Votes 1 Answers 718 Views
Hello! Is there a way to launch clearml apps (for example clearml schedulers) via API or code with status tracking on ClearML application tab? If we run clea...
8 months ago
0 Votes
28 Answers
796 Views
0 Votes 28 Answers 796 Views
Hi! I'm running launch_multi_mode with pytorch-lightning task.execute_remotely(queue_name='my-queue' config = task.launch_multi_node(args.nodes)) ddp = DDPSt...
9 months ago
0 Votes
3 Answers
803 Views
0 Votes 3 Answers 803 Views
Hi! When running a remote task on the agent, clearml installs additional system packages. how can this be disabled? the variable agent.package_manager.system...
9 months ago
0 Votes
2 Answers
834 Views
0 Votes 2 Answers 834 Views
Hi everyone! I have a ClearML dataset that takes up 10 Tb. Its local download (get_local_copy) takes about a month. Can you tell me how to speed up this proc...
9 months ago
0 Votes
1 Answers
1K Views
0 Votes 1 Answers 1K Views
Why can't I find the task created for the pipeline in the project through the main dashboard?
one year ago
0 Votes
7 Answers
908 Views
0 Votes 7 Answers 908 Views
Hello everyone! The cache for pip does not work for agent in k8s mode. I specify agent.docker_pip_cache as /mnt/pip_cache in the clearml.conf. But nothing is...
10 months ago
0 Votes
1 Answers
913 Views
0 Votes 1 Answers 913 Views
11 months ago
0 Votes
5 Answers
762 Views
0 Votes 5 Answers 762 Views
Hello! Is there a way to launch clearml apps (for example clearml schedulers) via API or code with status tracking on ClearML application tab? If we run clea...
8 months ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
Hello! How to determine the cache for an agent in Kubernetes? I'm going to mount s3 as a cache folder as a local path using s3fs. What variable needs to be s...
10 months ago
0 Votes
3 Answers
194 Views
0 Votes 3 Answers 194 Views
Hello everyone! I tried to remove models from ClearML using clearml.Model.remove( model=model_id, delete_weights_file=True, force=True, raise_on_errors=True,...
one month ago
0 Votes
3 Answers
632 Views
0 Votes 3 Answers 632 Views
Hello everyone! Can I create a report via API or SDK? Сan the model inference task generate a report that will be displayed in the reports tab?
8 months ago
0 Votes
1 Answers
1K Views
0 Votes 1 Answers 1K Views
Hello everyone! How can I conveniently pass a large number of parameters to the pipeline in order to re-run it through ui?
one year ago
0 Votes
7 Answers
37 Views
0 Votes 7 Answers 37 Views
Hello! I need to run clearml pipeline with caching of steps. I specify cache_executed_step=True for each step, but my steps are not cached and ended with the...
4 days ago
0 Votes
2 Answers
395 Views
0 Votes 2 Answers 395 Views
5 months ago
0 Hello Everyone! Can I Create A Report Via Api Or Sdk? Сan The Model Inference Task Generate A Report That Will Be Displayed In The Reports Tab?

Hi @<1523701087100473344:profile|SuccessfulKoala55> where can I get examples of REST API requests for creating reports?

8 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

for example, global rank from failed task in first scenario
image
image

9 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

Hi @<1523701435869433856:profile|SmugDolphin23> ! I set NODE_RANK in the environment and now

  • if gpus=2, node=2, task.launch_multi_node(node) : three tasks are created, and two of which are completed, but one is failed. In this case, are created (gpus*nodes-1) of tasks, some of which crashes with an error, or they all fall with an error. the behavior is inconsistent.
  • if gpus=2, node=2, task.launch_multi_node(node*gpus) : seven tasks are created.I n this case, all tasks are failed except t...
9 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

@<1523701435869433856:profile|SmugDolphin23>
Logs of rank0:

Environment setup completed successfully
 
Starting Task Execution:
 
 
1718702244585 gpuvm-01:gpu3,0 DEBUG InsecureRequestWarning: Certificate verification is disabled! Adding certificate verification is strongly advised. See: 

ClearML results page: 
 /projects/0eae440b14054464a3f9c808ad6447dd/experiments/beaa8c380f3c46f0b6f5a3feab514dc8/output/log
task id [beaa8c380f3c46f0b6f5a3feab514dc8]
world=4
...
9 months ago
0 Hi Everyone! I'M Trying To Use

@<1523701435869433856:profile|SmugDolphin23> Everything worked after setting the variables: --env NCCL_IB_DISABLE=1 --env NCCL_SOCKET_IFNAME=ens192 --env NCCL_P2P_DISABLE=1. But previously, these variables were not required for a successful launch. When I run ddp training with two nodes , everything works for me now. But as soon as I increase their number ( nodes > 2 ), I get the following error.

Traceback (most recent call last):
  File "/root/.clearml/venvs-builds/3.11/code/light...
3 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

@<1523701435869433856:profile|SmugDolphin23> Two tasks were created when gpus=2, nodes=2, task.launch_multi_node(node). But their running status does not end, and model training does not begin.
image

9 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

@<1523701435869433856:profile|SmugDolphin23> it work with gpus=1 and node=2 and there are only two tasks is created

9 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

@<1523701435869433856:profile|SmugDolphin23> if task.aunch_multi_node(4) , then all 4 tasks are failed
image

9 months ago
0 Hello! I Need To Run Clearml Pipeline With Caching Of Steps. I Specify Cache_Executed_Step=True For Each Step, But My Steps Are Not Cached And Ended With The Status - Completed.

@<1523701070390366208:profile|CostlyOstrich36> If I run the pipeline with the same input parameters, all the steps will also be re-run, nothing will be taken from the cache

4 days ago
0 Hello! I Need To Run Clearml Pipeline With Caching Of Steps. I Specify Cache_Executed_Step=True For Each Step, But My Steps Are Not Cached And Ended With The Status - Completed.

I create a pipeline via PipelineController with adding a step as a function

pipe = PipelineController(
        name=cfg.clearml.pipeline_name,
        project=cfg.clearml.project_name,
        target_project=True,
        version=cfg.clearml.version,
        add_pipeline_tags=True,
        docker=cfg.clearml.dockerfile,
        docker_args=DefaultMLPLATparam().docker_arg,
        packages=packages,
        retry_on_failure=3
    )

for parameter in cfg.clearml.params:
        pipe.add_...
4 days ago
0 Hi Everyone! I'M Trying To Use

@<1523701435869433856:profile|SmugDolphin23> It is possible to request up to 5 workers in the toy example with Feed Forward and MNIST, BUT it is not possible to request more than 2 workers on a real large model

3 months ago
9 days ago
0 Hi All! I Write A Data Processing Pipeline. It Is Necessary To Define Many Hyperparameters That Are Inconvenient To Redefine In A Pop-Up Window When Restarting The Pipeline From Ui. Is It Possible To Overrided The Parameters Through The Configuration File

Hi @<1523701205467926528:profile|AgitatedDove14>
I define a pipeline through functions. I have a lot of parameters, about 40. It is inconvenient to overwrite them all from the window that is on the screen.
image

one year ago
0 Hello! Can You Help Me With Model Endpoints Tab - How To Connect It To Existing Clearml-Serving Instance?

@<1523701070390366208:profile|CostlyOstrich36> i have 2 clearm-serving instances with endpoints

11 days ago
0 Hi Everyone! I'M Trying To Use

@<1523701435869433856:profile|SmugDolphin23> This error occurs when a secondary task is created with launch_multi_node. And this error disappears when I add the reuse_last_task_id=False flag when initializing the task. But now I have a new problem. I can't request more than 2 nodes. The training logs freezes after several iterations of first epoch with three workers. And if i request four workers i get this error:

DEBUG Epoch 0:   8%|▊         | 200/2484 [04:43<53:55,  0.71it/s, v_num=...
3 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

Hi @<1523701205467926528:profile|AgitatedDove14>
I started an experiment with gpus=2 and node=2 and I have the following logs
image
image
image

9 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

Hi @<1523701435869433856:profile|SmugDolphin23> Thank you for your reply!
I use 2 machines.
I set these parameters, but unfortunately, the training has not started.

torch.distributed.DistStoreError: Timed out after 1801 seconds waiting for clients. 2/4 clients joined.
8 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

The errors that occur in the second case are presented in this screenshots.
image
image

9 months ago
0 Hi! I'M Running Launch_Multi_Mode With Pytorch-Lightning

@<1523701435869433856:profile|SmugDolphin23> hi! it works! thanks!

8 months ago
0 Hello Everyone! The Cache For Pip Does Not Work For Agent In K8S Mode. I Specify Agent.Docker_Pip_Cache As /Mnt/Pip_Cache In The Clearml.Conf. But Nothing Is Saved Along This Path
kubectl exec -it clearml-agent-85fd8ccc6d-7fdk7 -n clearml bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulted container "k8s-glue" out of: k8s-glue, init-k8s-glue (init)
root@clearml-agent-85fd8ccc6d-7fdk7:~# cat /root/clearml.conf 
agent.git_user=gitlab_agent
agent.git_pass=682S-pH9ay1nidsxBGyT
agent.cuda_version=118
#agent.docker_internal_mounts.venv_build=/home/s3_cache/venvs-builds
#agent.do...
10 months ago
10 months ago
Show more results compactanswers