Can you please screenshot the INFO
tab on the pipeline controller task?
what do you get when you run this code?
from clearml.backend_api import Session
print(Session.check_min_api_server_version("2.17"))
@<1523701435869433856:profile|SmugDolphin23> I run the code in order to step1, step2 and step3. And then I run the "pipeline_from_task.py" scripts. I follow the ClearML documentation so whole of the codes taken from github repo.
@<1626028578648887296:profile|FreshFly37> can you please screenshot this section of the task? Also, how does your project's directory structure look like?
what about import clearml; print(clearml.__version__)
@<1523701435869433856:profile|SmugDolphin23> I have tried another way by including pipeline.py in the root directory of the code and executed “python3 pipeline.py” & still faced same issue
I also encountered a similar problem. When I run pipeline code I could not see in the deployed pipeline in the ClearML UI and the pipeline's first step does not start in the remote agent machine. It is queued with pending status.
@<1657556312684236800:profile|ManiacalSeaturtle63> can you share how you are creating your pipeline?
@<1523701435869433856:profile|SmugDolphin23> you are right. When I added more worker to queue and it released from pending status. However when I click the pipelines in the screenshoot, I could not see pipeline schema. It shows me "no pipeline to show" text like in below. Do you have any idea ? I should see each step box when I click the pipeline right ?
For the clearml-server installation I follow the documentation steps one by one. Link is : None
@<1523701435869433856:profile|SmugDolphin23> I retry the same scenario with clearml==1.14.1 package but still it does not show me the pipelines not showing in the UI :(
@<1523701435869433856:profile|SmugDolphin23> I used clearml==1.13.2 and now I am upgrading to clearml=1.14.1 version.Also I would give extra information about Clearml-server docker-compose file images versions is latest right now.
@<1657556312684236800:profile|ManiacalSeaturtle63> what clearml SDK version are you using? I believe there was a bug related to pipelines not showing in the UI, but that was fixed in clearml==1.14.1
@<1523701435869433856:profile|SmugDolphin23> I have attached two screenshots, One is pipeline initialization & other one is the task of the pipeline.
The project's directory is as follows:
The pipeline.py includes the code to run the pipeline & tasks of the pipeline.
├── Makefile
├── README.md
├── ev_xxxxxx_detection
│ ├── __init__.py
│ ├── __pycache__
│ │ └── __init__.cpython-311.pyc
│ ├── clearml
│ │ ├── __pycache__
│ │ ├── clearml_wrapper.py
│ │ ├── constants.py
│ │ ├── data_loader.py
│ │ ├── ev_trainer.py
│ │ ├── pipeline.py
│ │ └── util.py
├── poetry.lock
├── pyproject.toml
Regarding pending pipelines: please make sure a free agent is bound to the queue you wish to run the pipeline in. You can check queue information by accessing the INFO section of the controller (as in the first screenshort)
then by pressing on the queue, you should see the worker status. There should be at least one worker that has a blank "CURRENTLY EXECUTING" entry
Hi!
It is possible to use the same queue for the controller and the steps, but there needs to be at least 2 agents that pull tasks from that queue. Otherwise, if there is only 1 agent, then that agent will be busy running the controller and it won't be able to fetch the steps.
Regarding missing local packages: the step is ran in a temporary directory that is different than the directory the script is originally in. To solve this, you could add all the modules/files you are interested in in a git repository. If you do, that repository will be cloned by the agent when running the steps, which will make the packages accessible.
sure, I'll add those details & check. Thank you
@<1523701435869433856:profile|SmugDolphin23> Can you please help me out here
This print string like in below. """
if not self._task:
task_name = name or project or '{}'.format(datetime.now())
if self._pipeline_as_sub_project:
parent_project = (project + "/" if project else "") + self._pipeline_section
project_name = "{}/{}".format(parent_project, task_name)
else:
parent_project = None
project_name = project or 'Pipelines'
# if user disabled the auto-repo, we force local script storage (repo="" or repo=False) """
Oh I see. I think there is a mismatch between some clearml versions on your machine? How did you run these scripts exactly? (like the CLI, for example python test.py
?)
Or if you ran it via an IDE, what is the interpreter path?
@<1626028578648887296:profile|FreshFly37> I see that create_dataset
doesn't have a repo set. Can you try setting it manually via the repo
repo_branch
repo_commit
arguments in the add_function_step
method?
@<1626028578648887296:profile|FreshFly37> can you share also logs of task ? It may give an idea.
@<1523701435869433856:profile|SmugDolphin23> , I’ve updated both the ClearML server and client to the latest version, 1.14.0, as per our previous conversation. However, I’m still encountering the same issue as described earlier.
WebApp: 1.14.0-431
Server: 1.14.0-431
API: 2.28
I attempted to use the same queue for both the controller and the steps, and assigned two workers to this queue. Upon executing the code, the pipeline was initiated on the “queue_remote” queue, and the tasks of the pipeline were also initiated on another worker in the “queue_remote” queue. However, the dataset creation failed because it was unable to locate the Python modules from the current directory as shown in the below screenshot.
Note: I stored the code and its dependencies in a GitHub repository when I executed the pipeline.
Please refer to the attached error screenshot and the code I used to run the pipeline for more details
I have attached the screenshot of logs earlier