
Reputation
Badges 1
45 × Eureka!No, until now we used the default server that is handled by Clearml and we want to transfer to a self hosted one
Yes, same one
when i tried doing with the decorators it threw me an error that it cannot run task init in side a working task (the pipe lines task)
Thanks @<1523701070390366208:profile|CostlyOstrich36> , but doesn’t the agent create/caches an environment from the requirements.txt when running? I’m reproducing an old project that used to work like that, and also my ClearML.conf set to work that way
Ok cool, I'll try that, Thanks
not sure what that means to be honest @<1523701070390366208:profile|CostlyOstrich36>
@<1523701087100473344:profile|SuccessfulKoala55> But when i use this setting it the packages download only from the torch repo and not a local repo correct? or does it use the url-extra-link? and is there a way to cancel the auto cuda detect?
its working now, thanks that was the problem.
Yes it does, thank you @<1523701070390366208:profile|CostlyOstrich36>
@<1523701087100473344:profile|SuccessfulKoala55> did i do something wrong?
Thank you @<1523701070390366208:profile|CostlyOstrich36> and @<1523701205467926528:profile|AgitatedDove14> , after that bit on information, can you tell me where I can find the differences between the community server and self hosted server?
Are there any additional downsides to migrating to a self hosted server?
@<1523701087100473344:profile|SuccessfulKoala55> yes the working dir is set to the correct path and yet it cannot import the train module
Yes, here is the log file.
Thanks John, I read the one about the pip timeout, the problem is that I'm assume clearml runs the following command :
"pip install -r requirments.txt" and I want to know if I make clearml add the timeout flag.
Hi @<1523701070390366208:profile|CostlyOstrich36> , it is part of the repository, do pipelines run differently then normal tasks? what I mean is when i run a task it has a working directory do pipelines also have that or are their working directory the root of the repo?
yeah i see it now in the requirements of the task, that's weird, I'll create a new environment and check it again, thanks
The flow is: Training.py (which creates and runs a training task) -> conversion_task.py (converts the outputs of the models into a format of our choosing) -> testing.py (testing the model after conversion).
I tried using the decorators and fucntions but they both threw me errors that i cannot do task init in side a running task.
yes sometimes I suffer from small network issues, is there a way to make clearml have a bigger timeout when installing packages?
and if not is there a way to point it to a local package for installation or a local virtual enviroment?
Hi @<1523701070390366208:profile|CostlyOstrich36> , Here is a better explanation of my situation, in my IDE the working directory is where my code starts and I'm importing from common_utils my custom augmentations and locally the code is working with the import I've added in my previous message, however when i run from ClearML agent the import from point a to point b isn't working however they are both in the same git repo and i don't want to copy the files into project_1 as to not have unne...
@<1523701087100473344:profile|SuccessfulKoala55> in the file example here there is no reference to console_cr_flush_period
Wow, thanks a lot @<1523701070390366208:profile|CostlyOstrich36> for pointing me in the right direction. I also see that i can use sdk.development.worker.log_stdout
if i really need to kill my api calls before I'll Host my own server.
BTW what does suppress_update_message
do? I mean which kind of messages does it suppress?
And I'm looking at None as an example of a clearml.conf file and i can't seem to find sdk.development.worker.console_cr_flush_period
this flag.
Hi @<1523701070390366208:profile|CostlyOstrich36> , I am using the community server, what happens if i change to a self hosting server?
I'm using Tensorboard to report everything, nothing special besides that.
Hi @<1523701070390366208:profile|CostlyOstrich36> ,
but how do I configure this if I'm not hosting the clearml server?
where can i find the services.conf file?
@<1523701070390366208:profile|CostlyOstrich36> After discussing with my TL, we think the plan we are subscribed to might not be for us, can you point me to a person who we can have a meeting with and advice us the best plan for my team?
@<1523701087100473344:profile|SuccessfulKoala55> and @<1523701070390366208:profile|CostlyOstrich36> Ok so I found the problem but its weird,
when the agent is setting up the enviorment its installing torch=1.11.0 and not installing the one in the requirements which is torch=1.11.0+cu113,
I've checked the clearml.conf and i do have this flag set:
force_repo_requirements_txt: true
and I have a local whl of torch=1.11.0+cu113 with a path set to its location in the requirements.txt ...
@<1523701087100473344:profile|SuccessfulKoala55> After going into the steps full details I reset the step and enqueued it
Just upgraded to clearml-agent==1.5.1 and I still get this error.