I've finally gotten the triton engine to run. I'll be going through nvidia triton docs to find how to make an inference request. If you have an example inference request, I'll appreciate if you can share it with me.
I basically go to the model from the experiment first, then when in the model, I'm trying to download it but can't. I've screenshotted the situation.
I'll read the 3 examples now. Am I right to assume that I should drop Pipeline_Controller.py
Still unsure between finalize and publish? Since upload should upload the data to the server
Basically, right now when I save the model, it just goes in draft mode. What I want to do is that save the model only if it is better than the previous one, and once saved, publish it and have a name and tags that I want to add.
Thus I wanted to pass the model id from the prior step to the next one.
'dataset' is the name of my Dataset Object
they're also enqueued
I actually just asked about this in another thread. Here's the link. Asking about the usage of the upload_artifact
Ok since its my first time working with pipelines, I wanted to ask. Does the pipeline controller run endlessly or does it run from start to end with me telling it when to start based on a trigger?
Like there are files in a specific folder on Machine A. A script on Machine A, creates a Dataset, adds files located in that folder, and publishes it. Now can you look at that dataset on the server machine? Not from the ClearML interface but inside normal directories, like in /opt/clearml etc. this directory mentioned is just an example.
Can you take a look here?
https://clearml.slack.com/archives/CTK20V944/p1637914660103300
This is where I've mentioned the anonymous task spawn issue. I kind of want to understand what's causing the problem, if it is a problem etc
Basically the environment/container the agent is running in needs to have specific cuda installed. Is that correct CostlyOstrich36 ?
Thanks for the help.
Considering I don't think the function itself requires Venv to run normally but in this case it says it can't find venv
In this case, it would wait until any change occurred in the project mentioned here. Even if a task started, and not a dataset change or publish or anything, it would trigger.
You could be right, I just had a couple of packages with this issue so I just removed the version requirement for now. Another issue that might be the case, might be that I'm on ubuntu some of the packages might've been for windows thus the different versions not existing
do I just post the issue on the main clearml repo?
Let me try to be a bit more clear.
If I have a training task in which I'm getting multiple ClearML Datasets from multiple ClearML IDs. I get local copies, train the model, save the model, and delete the local copy in that script.
Does ClearML keep track of which data versions were gotten and used from ClearML Data?
Anyway I restarted the triton serving engine.
Also what's the difference between Finalize vs Publish?
I think I get what you're saying yeah. I don't know how I would give each server a different cookie name. I can see this problem being resolved by clearing cookies or manually entering /login at the end of the url
So in my case where I schedule a task every time I publish a data, when I publish my dataset once, it triggers and starts a new task.
I just assumed it should only be triggered by dataset related things but after a lot of experimenting i realized its also triggered by tasks, if the only condition passed is dataset_project and no other specific trigger condition like on publish or on tags are added.
Basically there is an agent still listening to a queue on a machine which i might've started at some point but i can't seem to stop it.
Understandable. My main concern was that I needed initial requirements for experimentation.