Here it is @<1523701070390366208:profile|CostlyOstrich36>
Thanks for your feedback

Hi @<1523701205467926528:profile|AgitatedDove14>
Yes, it was indeed in our code! after looking in depth, the loading of .cu and .cpp files was the root of the issue, slowing down the batch inference. Thanks a lot for your support!!

Also, as can be seen in docker args, I tried using CLEARML_AGENT_SKIP_PIP_VENV_INSTALL and CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL, to avoid installing packages as the container conatins everything needed to run the task, but not sure that it had any effect.

Hi Damjan, thank you for your message.
But If I understand correctly, that doc would be great for online serving. I am looking a solution for batch inference instead.

I will try to create the base docker image using the "Exporting a Task into a Standalone Docker Container" section from
from None

Hi @<1523701070390366208:profile|CostlyOstrich36>
Did you have the time to check the full log I sent you?
Most appreciated!

Hi @<1523701205467926528:profile|AgitatedDove14> Thanks for your reply.
Is there any way to avoid creating a task to get the user ID?

Yesss!! got the info I needed. Thanks a lot!!

Hello @<1523701205467926528:profile|AgitatedDove14> , thank you for addressing my concern. It seems that the aspect of avoiding the venv is functioning correctly, and everything within the container is properly configured to initiate. However, there is still a delay of approximately 2 minutes between the completion of setup, thus the appearance of the console log indicating "Starting Task Execution" and the actual commencement of the inference logic. During this period, no additional logs ar...

Hi @<1523701205467926528:profile|AgitatedDove14> would you be so kind to take a look at this issue?
we still have 2 minutes between the log of
"

Starting Task Execution:

"
and actually starting our inference logic. We have no extra info from the log to check to improve this slow task start time.
thanks a lot for any feedback!

We are thinking to host our own server later on, so this wont be a problem in that case? but I was asking just know what was going on. thanks again

Hi again @<1523701070390366208:profile|CostlyOstrich36>
One further question. Is it possible also to avoid using a git project ? (given that the code could be already present inside the docker container).
hat would be the benefits of setting it in the clearml task config instead? thanks again!

Thanks for the fast reply! I will look into it.

yes, the network seems fine. And it finally ends by getting the dataset. It just seems to hang more than it should between zips files.

Hi @<1523701205467926528:profile|AgitatedDove14> !
Thanks againg for following up this thread.

Perhaps It is not clear to read the delay in the log, but is just after "Starting Task Execution:"

Environment setup completed successfully
Starting Task Execution:

Here this new entry in the log is 2 min after env completed =>
1702378941039 box132 DEBUG 2023-12-12 11:02:16,112 - clearml.model - INFO - Selected model id: 9be79667ca644d7dbdf26732345f5415

So, the environment is creat...

Hi John, on the fileserver