@<1523701087100473344:profile|SuccessfulKoala55> in the file example here there is no reference to console_cr_flush_period

not sure what that means to be honest @<1523701070390366208:profile|CostlyOstrich36>

yes it is

Hi @<1523701070390366208:profile|CostlyOstrich36> , I am using the community server, what happens if i change to a self hosting server?
I'm using Tensorboard to report everything, nothing special besides that.

sadly the teammate that had the problem re-ran the experiments so i don't have the taskids but I do have the cpu and gpu usage of the agent that ran the experiment:

Yes, same one

yeah i see it now in the requirements of the task, that's weird, I'll create a new environment and check it again, thanks

It’s running a agent without docker, we aren’t using docker

@<1523701087100473344:profile|SuccessfulKoala55> and @<1523701070390366208:profile|CostlyOstrich36> Ok so I found the problem but its weird,
when the agent is setting up the enviorment its installing torch=1.11.0 and not installing the one in the requirements which is torch=1.11.0+cu113,
I've checked the clearml.conf and i do have this flag set:

force_repo_requirements_txt: true

and I have a local whl of torch=1.11.0+cu113 with a path set to its location in the requirements.txt ...

@<1523701087100473344:profile|SuccessfulKoala55> But when i use this setting it the packages download only from the torch repo and not a local repo correct? or does it use the url-extra-link? and is there a way to cancel the auto cuda detect?

Hi @<1523701070390366208:profile|CostlyOstrich36> ,
but how do I configure this if I'm not hosting the clearml server?
where can i find the services.conf file?

Hi @<1523701205467926528:profile|AgitatedDove14> ,
That solved my problem thank you, from my deep dive I've found the problem there was a package called install that changed it's name to pip-install and its requirement was setuptools .
Thanks again for the help.

Yes, here is the log file.

yes that's exactly my issue.

Thanks John, I read the one about the pip timeout, the problem is that I'm assume clearml runs the following command :
"pip install -r requirments.txt" and I want to know if I make clearml add the timeout flag.

@<1523701087100473344:profile|SuccessfulKoala55> and @<1523701070390366208:profile|CostlyOstrich36> , in the end I've found the problem, it was due to me running the pipeline locally and when running the pipeline locally it, doesn't copy all the dir but only the script that is running None

Oh so in that case I'll need to change every agent's pip config file.

yes sometimes I suffer from small network issues, is there a way to make clearml have a bigger timeout when installing packages?
and if not is there a way to point it to a local package for installation or a local virtual enviroment?

@<1523701087100473344:profile|SuccessfulKoala55> yes the working dir is set to the correct path and yet it cannot import the train module

Hi @<1523701070390366208:profile|CostlyOstrich36> , it is part of the repository, do pipelines run differently then normal tasks? what I mean is when i run a task it has a working directory do pipelines also have that or are their working directory the root of the repo?

@<1523701070390366208:profile|CostlyOstrich36> After discussing with my TL, we think the plan we are subscribed to might not be for us, can you point me to a person who we can have a meeting with and advice us the best plan for my team?

I reviewed this example and sadly there isn't anything about how to upload a path as a string only.

Also looked at it but its only supported registered artifact object type is a pandas.DataFrame and not strings.
I think I'll keep it with ':' in the start of the string and that way it won't upload the folder

@<1523701070390366208:profile|CostlyOstrich36> my repo is like this and both the files are located at the same dir so its weird that they cannot find train:

.
├── pytorch
├── tensorflow
│   ├── Project A
│   │   └── src
│   ├── Project B
│   │   ├── data
│   │   ├── model
│   │   ├── reports
│   │   └── utils
│   ├── hand_validator_boxes
│   │   ├── src
│   │   ├── train.py (the module i need)
│   │   └── clearml_pipeline.py (where the pipeline is initilizied
└── utils

Wow, thanks a lot @<1523701070390366208:profile|CostlyOstrich36> for pointing me in the right direction. I also see that i can use sdk.development.worker.log_stdout if i really need to kill my api calls before I'll Host my own server.
BTW what does suppress_update_message do? I mean which kind of messages does it suppress?

@<1523701087100473344:profile|SuccessfulKoala55> After going into the steps full details I reset the step and enqueued it

@<1523701087100473344:profile|SuccessfulKoala55> did i do something wrong?

Thanks I'll look into that, but in the end we decided to add a private repo with the pytorch libraries that we need.
I've added the extra_index_url to point to our https and we changed to requirements.txt to look for that https however I'm getting this that warning I've attached and its still trying to download the packages not from my path.
how do i enable clearml-agent to look for private repos?

we had a few experiments that were stuck for a few hours until we noticed that and we also had 1 that was stuck for 2 days (on the weekend). and they weren't auto aborted.

Thank you @<1523701070390366208:profile|CostlyOstrich36> and @<1523701205467926528:profile|AgitatedDove14> , after that bit on information, can you tell me where I can find the differences between the community server and self hosted server?
Are there any additional downsides to migrating to a self hosted server?