Hi Team, I Am Trying To Run A Pipeline Remotely Using Clearml Pipeline And I’M Encountering Some Issues. Could Anyone Please Assist Me In Resolving Them?

Answered

Hi Team,

I am trying to run a pipeline remotely using ClearML pipeline and I’m encountering some issues. Could anyone please assist me in resolving them?

Issue 1 : After executing the code, the pipeline is initiated on the “queue_remote_start” queue and the tasks of the pipeline are initiated on the “queue_remote” queue. However, the creation of the dataset failed because it couldn’t find the Python modules from the current directory.

Issue 2 : I also attempted to use the same queue for both pipe.start and pipe.set_default_execution_queue . However, the tasks of the pipeline remained in the pending and queued state and didn’t proceed to the next step.

To run the pipeline remotely, I have created two different queues and assigned a worker to each using the following commands:

clearml-agent daemon --detached --create-queue --queue queue_remote
clearml-agent daemon --detached --create-queue --queue queue_remote_start

I then executed the following command to run the pipeline remotely:

python3 pipeline.py

The code for the Pipeline from Functions is as follows:

# Create the PipelineController object
    pipe = PipelineController(
        name="pipeline",
        project=project_name,
        version="0.0.2",
        add_pipeline_tags=True,
    )

pipe.set_default_execution_queue('queue_remote')

pipe.add_function_step(
    name='step_one',
    function=step_one,
    function_kwargs={
            "train_file": constants.TRAINING_DATASET_PATH,
            "validation_file": constants.VALIDATAION_DATASET_PATH,
            "s3_output_uri": constants.CLEARML_DATASET_OUTPUT_URI,
            "dataset_project": project_name,
            "dataset_name": constants.CLEARML_TASK_NAME,
            "use_dummy_dataset": use_dummy_model_dataset,
        },
        project_name=project_name,
        task_name=create_dataset_task_name,
        task_type=Task.TaskTypes.data_processing,
    )

pipe.start(queue="queue_remote_start")

Could anyone please provide a solution on how to successfully run the pipeline remotely? Any help would be greatly appreciated.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FreshFly37
				
					0
					 × 1

Votes Newest

Answers 39

what about import clearml; print(clearml.__version__)

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1626028578648887296:profile|FreshFly37> can you share also logs of task ? It may give an idea.

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

@<1657556312684236800:profile|ManiacalSeaturtle63> can you share how you are creating your pipeline?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1523701435869433856:profile|SmugDolphin23> I retry the same scenario with clearml==1.14.1 package but still it does not show me the pipelines not showing in the UI :(

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

@<1626028578648887296:profile|FreshFly37> can you please screenshot this section of the task? Also, how does your project's directory structure look like?

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

what do you get when you run this code?

from clearml.backend_api import Session
print(Session.check_min_api_server_version("2.17"))

  				
Posted 
	one year ago

					More
				  		
  Report
		
					SmugDolphin23
				
					0

@<1523701435869433856:profile|SmugDolphin23> , I’ve updated both the ClearML server and client to the latest version, 1.14.0, as per our previous conversation. However, I’m still encountering the same issue as described earlier.
WebApp: 1.14.0-431
Server: 1.14.0-431
API: 2.28

I attempted to use the same queue for both the controller and the steps, and assigned two workers to this queue. Upon executing the code, the pipeline was initiated on the “queue_remote” queue, and the tasks of the pipeline were also initiated on another worker in the “queue_remote” queue. However, the dataset creation failed because it was unable to locate the Python modules from the current directory as shown in the below screenshot.

Note: I stored the code and its dependencies in a GitHub repository when I executed the pipeline.

Please refer to the attached error screenshot and the code I used to run the pipeline for more details

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FreshFly37
				
					0
					 × 1

sure, I'll add those details & check. Thank you

  				
Posted 
	one year ago

					More
				  		
  Report
		
					FreshFly37
				
					0
					 × 1

This print string like in below. """
if not self._task:
task_name = name or project or '{}'.format(datetime.now())
if self._pipeline_as_sub_project:
parent_project = (project + "/" if project else "") + self._pipeline_section
project_name = "{}/{}".format(parent_project, task_name)
else:
parent_project = None
project_name = project or 'Pipelines'
# if user disabled the auto-repo, we force local script storage (repo="" or repo=False) """

  				
Posted 
	one year ago

					More
				  		
  Report
		
					ManiacalSeaturtle63
				
					0
					 × 1

Show more results

Write your answer

123K Views

39 Answers

one year ago