Hi All, Just Started Tinkering With Clearml And Wondering If This Is Suitable To Deploy An Etl/Elt Pipeline Using The Agent On My 2Nd Machine? Or Is Clearml Designed To Pick Up After The Data Has Been Loaded And Cleaned? I Cant See Any Mention Of Feature

Answered

hi all, just started tinkering with clearml and wondering if this is suitable to deploy an ETL/ELT pipeline using the agent on my 2nd machine? or is clearml designed to pick up after the data has been loaded and cleaned? i cant see any mention of feature engineering in the docs?

my desired pipeline is:

pull data from API
store in postgres
run transformations and create datasets etc
should this be handled with clearml agent or should i look to orchestrate this externally from clearml?

cheers

  				
Posted 
	15 hours ago

					More
				  		
  Report
		
					PanickyParrot17
				
					0
					 × 1

Votes Newest

Answers 2

Hi @<1773158059758063616:profile|PanickyParrot17> ,

You can do that with ClearML pipelines . step 1 will be pulling the data, step 2 will store the data, step 3 will create the dataset.
The pipeline controller can have the parameters of which data to pull, the name of the created dataset and more, so you can run it again from the UI and just change the data source

All the steps can run with the clearml agent, and you can also specify using only cpu in the machines (search for the --cpu-only param)

  				
Posted 
	10 hours ago

					More
				  		
  Report
		
					RoundElephant20
				
					0

thanks, ill have a read through the docs again. im just trying to think through the dev and staging setup to develop this efficiently

  				
Posted 
	3 hours ago

					More
				  		
  Report
		
					PanickyParrot17
				
					0
					 × 1

Write your answer

3 Views

2 Answers

15 hours ago

2 hours ago