Reputation
Badges 1
25 × Eureka!That makes no sense to me?!
Are you absolutely sure the nntrain is executed on the same queue? (basically could it be that the nntraining is executed on a different queue in these two cases ?)
My apologies you are correct 1.8.1rc0 π
error in my-package setup command:
Okay this seems like an error in the setup.py you have in the "mypackage" folder
RoundMosquito25 do notice the agent is pulling the code from the remote repo, so you do need to push the local commits, but the uncommitted changes clearml will do for you. Make sense?
It takes 20mins to build the venv environment needed by the clearml-agent
You are Joking?! π
it does apt-get install python3-pip , and pip install clearml-agent, how is that 20min?
Hi WickedGoat98
I try to write an article on medium about ClearML and face some a problem with plotly figures.
This is awesome !
I ran the plotly_reporting.py example locally and the uploaded plot was ok.
So are you saying the same example code from the repository worked okay on your server but showed nothing on the hosted server ?
BattyLion34
Maybe something inside the task is different?!
Could you run these lines and send me the result:from clearml import Task print(Task.get_task(task_id='failing task id').export_task()) print(Task.get_task(task_id='working task id').export_task())
This should have worked with the latest clearml RC.
And you verified it is not working?
Do you have two agents pulling from the same queue ?
Maybe one of them is configured differently ?
BattyLion34 is this consistent?
(Really I can't see eny difference, one time it is able to create the venv and another it is failing with permission error)
Thanks BattyLion34 I fixed the code snippet :)
BattyLion34 let me see if I understand.
The same base_task_id when cloned by the UI and enqueues on the same queue as the pipeline, will work but when the pipeline runs the same Task it fails?!
Could it be that you enqueue them on different queues ?
Just call the Task.init before you create the subprocess, that's it π they will all automatically log to the same Task. You can also call the Task.init again from within the subprocess task, it will not create a new experiment but use the main process experiment.
MagnificentSeaurchin79 no need for the detection api (yes definitely a mess to setup), it will be more helpful to get a toy example.
Hi, is there a way to force the requirements.txt?
You mean to ignore the "Installed Packages" ?
BattyLion34
if I simply clone nntraining stage and run it in default queue - everything goes fine.
When you compare the Task you clone manually and the Task created by the pipeline , what's the difference ?
RobustRat47
What exactly is the error you are getting ? (I remember only the latest Triton solved some issue there)
Hmm I think this was the fix (only with TF2.4), let me check a sec
Hi @<1644147961996775424:profile|HurtStarfish47>
. I see
Add image.jpg
being printed for all my data items ...
I assume you forgot to call upload
? the sync "marks" files for uploaded / deletion but the upload call actually does the work,
Kind of like git add / push , if that makes sense ?
for future reference this is indeed a PEP-610 related bug, f
π
can we also set theΒ
poetry
Β version used?....
Actually the agent assumes poetry is preinstalled (so whatever you already have on the docker) ...
That said, maybe we should install a specific version (after installing pip, we could do that if poetry is selected)
wdyt ?
JitteryCoyote63 how can I reproduce it? (obviously when I tested it was okay)
WackyRabbit7
Cool - so that means the fileserver which comes with the host will stay emtpy? Or is there anything else being stored there?
Debug Images and artifacts will be automatically stored to the file server.
If you want your models to be automagically uploaded add the following :task=Task.init('example', 'experiment', output_uri='
')
(You can obviously point it to any other http/S3/GS/Azure storage)
ResponsiveCamel97
could you attach the full log?
Then as you suggested, I would just use sys.path it is probably the easiest and actually very safe (because the subfolders are Always next to the "main" source code)
oh dear π if that's the case I think you should open an Issue on pypa/pip , I'm not sure what we can do other than that ...
Interesting use case, do you already have the connect_configuration
in the code? or do we need to somehow create it ?
If you spin two agent on the same GPU, they are not ware of one another ... So this is expected behavior ...
Make sense ?