Hey All, I Am Having An Issue With Clearml That Never Happened Before. It Looks Like It Happens On The Version

Answered

Hey all, I am having an issue with ClearML that never happened before. It looks like it happens on the version clearml-1.14.4 but not in clearml-1.14.1 .

File "/venv/lib/python3.11/site-packages/clearml/backend_interface/util.py", line 89, in get_or_create_project
return _get_or_create_project(session, project_name, description=description, system_tags=system_tags, project_id=project_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/lib/python3.11/site-packages/clearml/backend_interface/util.py", line 112, in _get_or_create_project
    project_id = res.response.projects[0].id
                 ~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'StubObject' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/code/app/pipeline/training_pipeline.py", line 502, in <module>
    execute_pipeline(
  File "/venv/lib/python3.11/site-packages/clearml/automation/controller.py", line 4388, in internal_decorator
    a_pipeline = PipelineDecorator(
                 ^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/clearml/automation/controller.py", line 3413, in __init__
    super(PipelineDecorator, self).__init__(
  File "/venv/lib/python3.11/site-packages/clearml/automation/controller.py", line 342, in __init__
    get_or_create_project(
  File "/venv/lib/python3.11/site-packages/clearml/backend_interface/util.py", line 96, in get_or_create_project
    return _get_or_create_project(session, project_name, description=description, system_tags=system_tags, project_id=project_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/clearml/backend_interface/util.py", line 112, in _get_or_create_project
    project_id = res.response.projects[0].id
                 ~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'StubObject' object is not subscriptable

Does someone ever have the problem too ?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					EnchantingOctopus35
				
					0
					 × 1

Votes Newest

Answers 4

I didn't have the time yet. But here is what I found: I have a process that runs a first function in which I init a ClearML Task. Then, later in my process I run a sub process that triggers a ClearML Pipeline (execute_pipeline in my example is triggered). And then the problem happens.
If I don't run the first function that inits a ClearML Task, I don't have the issue. I think it is related to the cache of clearml ?

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					EnchantingOctopus35
				
					0
					 × 1

Can you try running with the latest version of clearml ?

  				
Posted 
	3 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

👋 hi @<1523701070390366208:profile|CostlyOstrich36> I am running a training pipeline of my model on a remote machine.

The way I do it, is through a cron job that triggers my API and then run a subprocess. The CLI of this subprocess launches the pipeline.

With something like


if __name__ == '__main__':
    args = parse_args(sys.argv)

    if args.is_local:
        PipelineDecorator.run_locally()

    execute_pipeline(...)

where execute_pipeline is decorated with PipelineDecorator.pipeline

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					EnchantingOctopus35
				
					0
					 × 1

Hi @<1787653566505160704:profile|EnchantingOctopus35> , what are you running?

  				
Posted 
	4 months ago

					More
				  		
  Report
		
					CostlyOstrich36
				
					0

Write your answer

374 Views

4 Answers

4 months ago

3 months ago