Thanks for the reply anyways 😄

Nevermind, I figured out the problem. I needed to specify the --docker flag when running the clearml-agent

Right so I figured out why it was calling it multiple times. Everytime a dataset is serialiazed, it calls the _serialize() function inside of clearml/datasets/dataset.py file, the _serialize() method calls self.get(parent_dataset_id) which is the same get() method. This means that the user will always be prompted with the log, even if they are not "getting" a dataset. So anytime a user creates, uploads, finalizes a dataset, they will be prompted with the message...

I set my local laptop as an agent for testing purposes. I run the code on my laptop, it gets sent to the server which sends it back to my laptop. So the conf file is technically on the worker right?

Thank you so much for your reply, will give that a shot!

Hi again @<1523701435869433856:profile|SmugDolphin23> ,

I was able to run the pipeline remotely on an agent, but I am still facing the same problem with the code breaking on the exact same step that requires the docker container. Is there a way to debug what is happening? Currently there is no indication from the logs that it is running the code in the docker container. Here are the docker related logs:

agent.docker_pip_cache = /home/amerii/.clearml/pip-cache
agent.docker_apt_cache =...

Thanks for the reply. I was trying out this feature on a dummy example. I used the following command
dataset = Dataset.get( dataset_project="palmer penguins", dataset_name="raw palmer penguins", alias="my_test_alias_name", overridable=True)That was the only time I called the get() command. I still got the message that I should specify the alias. I can try and do a bit of debugging to see why it gets called multiple times.

I knew that, I was just happy that we have an updated example 😁

Wow, that was fast. Thanks a lot for your prompt response! Will check it out now :D

I'm actually trying that as we speak 😛

Just waiting for the changes to be completed

Hi AgitatedDove14 ,

I am planning to use terraform to retrieve the secrets from AWS, after I retrieve the user list from the secrets manager, I am going to pass them as Environment variables.

The reason I am passing them as environment variables is that, I couldn't find a way to automatically upload files to AWS EFS from Terraform. Since the config file needs to be mounted as an EFS volume to the ECS task definition.

I was able to make the web authentication work while passing the followi...

SuccessfulKoala55 That seemed to do the trick, thanks for your help! 😄

So what's the point of the alias? It's not very clear.. Even after specifying an alias I am still getting the following message: Dataset.get() did not specify alias. Dataset information will not be automatically logged in ClearML Server

I would like to see it used in a clear example as it was intended to be used before giving my opinion on it, if that makes sense

Hi @<1523701205467926528:profile|AgitatedDove14> ,
Thank you for your prompt response.

I am using the functional pipeline API to create the steps. Where each step calls a function. My functions are stored in the files under the ap_pipeline directory ( filters.py , features.py , etc..)

These are packaged as part of this repo.

The modules are imported inside of the clearml_pipeline.py so it would look something like:

from ap_pipeline.features import func1, func2 ....
This...

Ohh, thanks! Will give it a shot now!