barebones, can a step in a pipeline refer to a previous step by name and get it?

I am going to be experimenting a bit as well, will get back on this topic in a couple of weeks 🙂

Thanks

I am using the clearml-agent from pypi version

Essentially, if I have a dataset on which I am performing transformations and then creating other downstream datasets

Got it, thanks

AlertBlackbird30 - i don’t understand why it can’t be a focus though. Probably missing some context.

AgitatedDove14 - this was an interesting one. I think I have found the issue, but verifying the fix as of now.

One of the devs was using shutil.copy2 to copy parts of dataset to a temporary directory in a with block - something like:

with TemporaryDirectory(dir=temp_dir) as certificates_directory: for file in test_paths: shutil.copy2(f"{dataset_local}/{file}", f"{certificates_directory}/file")
My suspicion is since copy2 copies with full data and symlin...

My question is - I have this in a notebook now. How can i make it such that any update to the upstream database triggers this data transformation step

AgitatedDove14 - thanks for the quick reply. automation.Monitor is the abstraction i could use?

Thanks let me try playing with these!

Not able to understand what’s really happening in the links

Also the pipeline ran as per this example - https://github.com/allegroai/clearml/blob/master/examples/pipeline/pipeline_controller.py

The agent ip? Generally what’s the expected pattern to deploy and scale this for multiple models?

AgitatedDove14 - added it in bucket_config.py and sdk.conf but somehow value is not being picked up

I am essentially creating a EphemeralDataset abstraction and creating controlled lifecycle for it such that the data is removed after a day in experiments. Additionally and optionally, data created during a step in a pipeline can be cleared once the pipeline completes

Something like this - https://aws.amazon.com/blogs/machine-learning/processing-pdf-documents-with-a-human-loop-using-amazon-textract-and-amazon-augmented-ai/

# Python 3.6.13 | packaged by conda-forge | (default, Feb 19 2021, 05:36:01) [GCC 9.3.0] argparse == 1.4.0 boto3 == 1.17.70 minerva == 0.1.0 torch == 1.7.1 torchvision == 0.8.2

in the repo whereas the docs are https://allegroai.github.io/clearml-server-helm/

You might have to setup a service (systemd) etc

I guess this is a advantage with docker mode. Will try that out as well sometime.

I can contribute as well as needed

Latest version was released 11 hours ago - https://github.com/jpadilla/pyjwt/releases/tag/2.2.0

Ok, got it thanks. Would be cool to let it get untracked as well, especially if we want to as an option

yeah

Sorry if it was confusing. Was asking if people have setup pipelines automatically triggered on update to datasets

AgitatedDove14 - where does automation.controller.PipelineController fit in?

now if dataset1 is updated, i want process to update dataset2

Trying to understand these, maybe playing around will help

I just want to change git remote like https://gitserver.com/path/to.git -> mailto:git@gitserver.com :path/to.git