SuccessfulKoala55 That seemed to do the trick, thanks for your help! 😄

I added the following to the clearml.conf file

agent {
    package_manager: {
        # supported options: pip, conda, poetry
        type: pip,
        extra_index_url: ["my_url"],
    },
}

For some reason the changes were not reflected, here are the logs from the agent:

agent.package_manager.type = pip
agent.package_manager.pip_version.0 = <20.2 ; python_version < '3.10'
agent.package_manager.pip_version.1 = <22.3 ; python_version >\= '3.10'
agent.package_manager.sys...

As you can see, it eventually manages to reach the apiserver however, it still says that access key was not provided and that the service will not be started. I get the same behaviour whether I set the flag to 0 or 1.

so basically the flag also prevents it from shutting down

Just waiting for the changes to be completed

Wow, that was fast. Thanks a lot for your prompt response! Will check it out now :D

The community server is working again.

The extra_index_url is not even showing..

@<1523701205467926528:profile|AgitatedDove14> So I was able to get it to pull the package by defining packages= None

The second problem that I am running into now, is that one of the dependencies in the package is actually hosted in a private repo.

I tried getting around it by defining the environment PIP_INDEX_URL and passing it using log_os_environments in the clearml.conf and I am now getting this message:

md-ap-feature-engineering/.venv/lib/p...

Wow!

22.04

Thank you for your reply SuccessfulKoala55 . 😄

It is currently set to 1, so I am assuming setting it to 0 would mute the errors from logging?

The current behaviour is, if I keep it set to 1 the services agent would automatically shutdown if the access key is not configured. Assuming I set it to 0, then the agent services should not shutdown anymore, right?

Hi @<1523701205467926528:profile|AgitatedDove14> ,
Thank you for your prompt response.

I am using the functional pipeline API to create the steps. Where each step calls a function. My functions are stored in the files under the ap_pipeline directory ( filters.py , features.py , etc..)

These are packaged as part of this repo.

The modules are imported inside of the clearml_pipeline.py so it would look something like:

from ap_pipeline.features import func1, func2 ....
This...

Hi again @<1523701435869433856:profile|SmugDolphin23> ,

I was able to run the pipeline remotely on an agent, but I am still facing the same problem with the code breaking on the exact same step that requires the docker container. Is there a way to debug what is happening? Currently there is no indication from the logs that it is running the code in the docker container. Here are the docker related logs:

agent.docker_pip_cache = /home/amerii/.clearml/pip-cache
agent.docker_apt_cache =...

Nevermind, I figured out the problem. I needed to specify the --docker flag when running the clearml-agent

The above output is on the clearml community server

@<1523701070390366208:profile|CostlyOstrich36> I am facing the same issue:

{"meta":{"id":"90841d05dfb1431a8d9dfc6bfdb39f9e","trx":"90841d05dfb1431a8d9dfc6bfdb39f9e","endpoint":{"name":"events.debug_images","requested_version":"2.23","actual_version":"2.7"},"result_code":200,"result_subcode":0,"result_msg":"OK","error_stack":"","error_data":{}},"data":{"metrics":[]}}

My server is hosted on AWS Fargate

Right so I figured out why it was calling it multiple times. Everytime a dataset is serialiazed, it calls the _serialize() function inside of clearml/datasets/dataset.py file, the _serialize() method calls self.get(parent_dataset_id) which is the same get() method. This means that the user will always be prompted with the log, even if they are not "getting" a dataset. So anytime a user creates, uploads, finalizes a dataset, they will be prompted with the message...

So I added the snippet above to the code,
and now the preview for the first 10 rows shows up. However, the automatic preview is still not working.

Thanks for the reply. I was trying out this feature on a dummy example. I used the following command
dataset = Dataset.get( dataset_project="palmer penguins", dataset_name="raw palmer penguins", alias="my_test_alias_name", overridable=True)That was the only time I called the get() command. I still got the message that I should specify the alias. I can try and do a bit of debugging to see why it gets called multiple times.

I would like to see it used in a clear example as it was intended to be used before giving my opinion on it, if that makes sense

The thing is, even on the community server, not all the datasets have automatic previews. So for the same code/dataset, some of the runs have previews and some of them don't.

Above is the response for the events.debug_images

I'm actually trying that as we speak 😛

So what's the point of the alias? It's not very clear.. Even after specifying an alias I am still getting the following message: Dataset.get() did not specify alias. Dataset information will not be automatically logged in ClearML Server

Thanks for the reply anyways 😄

I set my local laptop as an agent for testing purposes. I run the code on my laptop, it gets sent to the server which sends it back to my laptop. So the conf file is technically on the worker right?

I have been using them for a while, they've always had autopreviews.

I am having the same problem on both the self hosted and free community clearml.