Hi Team, I Am Trying To Run A Pipeline Remotely Using Clearml Pipeline And I’M Encountering Some Issues. Could Anyone Please Assist Me In Resolving Them?

Answered

Hi Team,

I am trying to run a pipeline remotely using ClearML pipeline and I’m encountering some issues. Could anyone please assist me in resolving them?

Issue 1 : After executing the code, the pipeline is initiated on the “queue_remote_start” queue and the tasks of the pipeline are initiated on the “queue_remote” queue. However, the creation of the dataset failed because it couldn’t find the Python modules from the current directory.

Issue 2 : I also attempted to use the same queue for both pipe.start and pipe.set_default_execution_queue . However, the tasks of the pipeline remained in the pending and queued state and didn’t proceed to the next step.

To run the pipeline remotely, I have created two different queues and assigned a worker to each using the following commands:

clearml-agent daemon --detached --create-queue --queue queue_remote
clearml-agent daemon --detached --create-queue --queue queue_remote_start

I then executed the following command to run the pipeline remotely:

python3 pipeline.py

The code for the Pipeline from Functions is as follows:

# Create the PipelineController object
    pipe = PipelineController(
        name="pipeline",
        project=project_name,
        version="0.0.2",
        add_pipeline_tags=True,
    )

pipe.set_default_execution_queue('queue_remote')

pipe.add_function_step(
    name='step_one',
    function=step_one,
    function_kwargs={
            "train_file": constants.TRAINING_DATASET_PATH,
            "validation_file": constants.VALIDATAION_DATASET_PATH,
            "s3_output_uri": constants.CLEARML_DATASET_OUTPUT_URI,
            "dataset_project": project_name,
            "dataset_name": constants.CLEARML_TASK_NAME,
            "use_dummy_dataset": use_dummy_model_dataset,
        },
        project_name=project_name,
        task_name=create_dataset_task_name,
        task_type=Task.TaskTypes.data_processing,
    )

pipe.start(queue="queue_remote_start")

Could anyone please provide a solution on how to successfully run the pipeline remotely? Any help would be greatly appreciated.

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					FreshFly37
				
					0
					 × 1

Votes Newest

Answers 39

@<1523701435869433856:profile|SmugDolphin23> I run the code in order to step1, step2 and step3. And then I run the "pipeline_from_task.py" scripts. I follow the ClearML documentation so whole of the codes taken from github repo.

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

@<1657556312684236800:profile|ManiacalSeaturtle63> can you share how you are creating your pipeline?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1523701435869433856:profile|SmugDolphin23> I retry the same scenario with clearml==1.14.1 package but still it does not show me the pipelines not showing in the UI :(

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

@<1523701435869433856:profile|SmugDolphin23> I used clearml==1.13.2 and now I am upgrading to clearml=1.14.1 version.Also I would give extra information about Clearml-server docker-compose file images versions is latest right now.

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

@<1657556312684236800:profile|ManiacalSeaturtle63> what clearml SDK version are you using? I believe there was a bug related to pipelines not showing in the UI, but that was fixed in clearml==1.14.1

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1523701435869433856:profile|SmugDolphin23> you are right. When I added more worker to queue and it released from pending status. However when I click the pipelines in the screenshoot, I could not see pipeline schema. It shows me "no pipeline to show" text like in below. Do you have any idea ? I should see each step box when I click the pipeline right ?

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

Regarding pending pipelines: please make sure a free agent is bound to the queue you wish to run the pipeline in. You can check queue information by accessing the INFO section of the controller (as in the first screenshort)
then by pressing on the queue, you should see the worker status. There should be at least one worker that has a blank "CURRENTLY EXECUTING" entry

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

Hi!
It is possible to use the same queue for the controller and the steps, but there needs to be at least 2 agents that pull tasks from that queue. Otherwise, if there is only 1 agent, then that agent will be busy running the controller and it won't be able to fetch the steps.

Regarding missing local packages: the step is ran in a temporary directory that is different than the directory the script is originally in. To solve this, you could add all the modules/files you are interested in in a git repository. If you do, that repository will be cloned by the agent when running the steps, which will make the packages accessible.

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

I also encountered a similar problem. When I run pipeline code I could not see in the deployed pipeline in the ClearML UI and the pipeline's first step does not start in the remote agent machine. It is queued with pending status.

  				
Posted 
	11 months ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

Show more results

Write your answer

34K Views

39 Answers

11 months ago

10 months ago