Even if you had any packages, I'm pretty sure there is nothing for you to worry about, it will just list them, and if they are preinstalled, the preinstalled will be used
Hmm maybe different numpy version? ( numpy==1.22.1
maybe the Task needs a diff version) ? Can you post the Task log ?
Hi @<1636175432829112320:profile|PlainSealion45>
I am trying to automatically generate an online endpoint for inference when manually adding tag
released
to a model.
So the "automatic" here means that the model endpoint will be updated with the latest model, but not that a new endpoint will be created.
Does that make sense ?
To add a new endpoint on Tagging a model, you should combine it with ModelTrigger
and have a fucntion that calls the clearml-serving to cr...
Thanks for the logs @<1627478122452488192:profile|AdorableDeer85>
Notice that the log you attached means the preprocessing is executed and the GPU backend is returning an error.
Could you provide the log of the docker compose specifically the intersting part is the Triton container, I want to verify it loads the model properly
Different question. How can I pass PYTHONPATH env variable to a task, run by agent (so python can find classes inside m subdirectories)?
Hi HelpfulHare30
By default the working directory will be added to the python path, this means if I have under execution:Working Dir: "." Script: "src/script.py"
The root git repo will be added to the python path.
BTW: next RC you could add a flag to the agent to always add the git repo
@<1587253076522176512:profile|HollowPeacock33>
Is this a commercial ad? this seems like out of scope for this channel
Can you expand?
Could you test with the same file? Maybe timeout has something to do with the file size ?
VirtuousFish83
could that be that "inplace-abn" while installing the package needs torch ?
And is Task.init called on all processes ?
now realise that the ignite events callbacks seem to not be fired
So this is an ignite issue ?
Sounds good, I assumed that was the case but I was not sure.
Let's make sure that in the clearml.conf
we write it in the comment above the use_credentials_chain
option, so that when users look for IAM roles configuration they can quick search for it 🙂
sdk.conf will add it to the default loaded values (as I think you deduced).
can copy paste the sdk.conf here? (maybe something is missing there?)
Amazing! 🎉
Let me know how we can help 🙂
Is is across the board for any Task ?
What would you expect to happen if you clone a Task that used the requirements.txt, would you ignore the full "pip freeze" and use the requirements .txt again, or is this thime we want to use the "installed packages" ?
And it works correctly when running on my computer, and if I use colab, then for some reason it has no effect.
I think I'm lost on this one, when running in colab, is this continuing a previous experiment ?
MagnificentPig49 that's a good question, I'll ask the guys 🙂
BTW, I think the main issues is actually making sure there is enough documentation on how to compile it...
Anyhow I'll update here
give me a minute to test
okay that's good, that means the agent could run it.
Now it is a matter of matching the TF with cuda (and there is no easy solution for that). Basically I htink that what you need is "nvidia/cuda:10.2-cudnn7-runtime-ubuntu16.04"
If we have the time maybe we could PR a fix?!
Hi MortifiedCrow63
I finally got GS credentials, there is something weird going on. I can verify the issue, with model upload I get timeout error while upload_artifacts just works.
Just updating here that we are looking into it.
You can try callingtask._update_repository()
I'm still trying to figure out how to reproduce it...
Yes RipeGoose2 you are totally correct 🙂 if you want the models to be auto uploaded in the offline session you have to pass output_uri (or default_output_uri).
Thanks!
In the conf file, I guess this will be where ppl will look for it.
os.system
Yes that's the culprit, it actually runs a new process and clearml
assumes that there are no other scripts in the repository that are used, so it does not analyze them
A few options:
Manually add the missing requirement Task.add_requirements('package_name')
make sure you call it before the Task.init
2. import the second script from the first script. This will tell clearml to analyze it as well.
3. Force the entire clearml to analyze the whole repository: https://g...
JitteryCoyote63 How can I reproduce it quickly?
i hope can run in same day too.
Fix should be in the next RC 🙂
are you using matplotlib ? could it be the binding check if matplotlib exists ? could it be you are running it with DEBUG on (i.e. global log level debug) ?
Any recommended way to make a task/pipeline “pause” until some external condition is met?
RoughTiger69 I would setup a trigger on the Dataset (i.e. new version)
https://github.com/allegroai/clearml/blob/df3d3b269acd2df0f31bfe804eb54ddc84d807c0/examples/scheduler/trigger_example.py#L44
wdyt?
GrievingTurkey78 Actually it is in progress, see the GitHub issue for details:
https://github.com/allegroai/trains/issues/219