Hey! Starting An Mlops Director Position In 2 Weeks. I'M Thinking About Architecture. Has Anyone Ever Tried To Use Clearml As An Experiment Tracker, But Used A Different Orchestrator Like Metaflow, Airflow, Prefect, Etc.? I'M Struggling To Find Guides Or

Answered

Hey! Starting an MLOps Director position in 2 weeks. I'm thinking about architecture.

Has anyone ever tried to use ClearML as an experiment tracker, but used a DIFFERENT orchestrator like Metaflow, Airflow, Prefect, etc.? I'm struggling to find guides or "hot takes" online for this.

  				
Posted 
	one day ago

					More
				  		
  Report
		
					BattyCrocodile47
				
					0
					 × 1

Votes Newest

Answers 2

I have tried:
Airflow - Pain to setup, old UI and other problems

Prefect - Literaly just tried to setup a simple distributed system, took me a week, I do not recommend this tool at all, horrible documentation, noone helps at slack.

Dagster - Absolute beauty, nice UI, easy to setup (as a pip package or just a docker + postgres), i highly recommend this tool. Takes a bit to get used to it. I will in coming week try this combo of dagster + clearml, where i periodically check some things and if i met some criteria I will spawn clearml jobs that will be put into clearml queue and executed.

  				
Posted 
	one day ago

					More
				  		
  Report
		
					AmiableSeaturtle81
				
					0
					 × 1

@<1590514584836378624:profile|AmiableSeaturtle81> Cool to see the community building such things! 🙂 If this works out for you, we'll be happy if you share your process!

A question both to you and @<1541954607595393024:profile|BattyCrocodile47> , what compels you to use a different orchestrator? Anything missing from the ClearML orchestration layer?

  				
Posted 
	16 hours ago

					More
				  		
  Report
		
					AnxiousSeal95
				
					0
					 × 1

Write your answer

11 Views

2 Answers

one day ago

8 hours ago