With my team we found a solution: to execute tasks with agent, we use clearml-task in CLI. We add the argument --output-uri : ***:1234 where *** is the link to our self-hosted server. Then models in pickle are automatically exported to the server, and not the path of the agent
With the screenshots above, the locally run experiment (left), does it have an http url for the model url field? The one you whited out?
No problem, I tried with this code :
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_blobs
from joblib import dump
from clearml import Task, OutputModel
task = Task.init(project_name="serving examples", task_name="train sklearn model", output_uri=True)
# generate 2d classification dataset
X, y = make_blobs(n_samples=100, centers=2, n_features=2, random_state=1)
# fit final model
model = LogisticRegression()
model.fit(X, y)
#dump(model, filename="sklearn-model.pkl", compress=9)
output_model = OutputModel(task=task, framework="ScikitLearn")
output_model.update_weights(weights_filename='sklearn-model.pkl')
And sadly I got the same error 😞 : $ curl -X POST "***/serve/sklearn_slack" -H "accept: application/json" -H "Content-Type: application/json" -d '{"x0": 1, "x1": 2}'{"detail":"Error processing request: expected str, bytes or os.PathLike object, not NoneType"}
I'm also working in http with my server, could it be the reason of it not working ?
Hi @<1523701118159294464:profile|ExasperatedCrab78> !
Here there are (left: locally, right: remotely)
That might make things harder indeed 🙂 It does explain some things
Hi @<1546303293918023680:profile|MiniatureRobin9> !
Would you mind sending me a screenshot of the model page (incl the model path) both for the task you trained locally as well as the one you trained on the agent?
Ah I see. So then I would guess it is due to the remote machine (the clearml agent) not being able to properly access your clearml server
Oh okay, it could explain a lot of stuff. Thank you for your answer 👍 My server isn't on 0.0.0.0, so would I need to setup a new one to solve this problem, or is there an alternative ?
I checked the logs as you suggested, I didn't find any error of this type (maybe I didn't put an important parameter). My agent is setup as a docker. Here are the logs.
Check your agent logs (through clearml console tab) and check if there isn't any error thrown.
What is probably happening is that your agent tries to upload the model but fails due to some kind of networking/firewall/port issue. For example: make sure you host your self-hosted server on 0.0.0.0 host so it's able to accept external connections other than localhost
Thanks! I know that you posted these locations before in text, I just wanted to make sure that they are the ones I was thinking. It seems like the model isn't properly uploaded to the clearml server. Instead, it's saving only the local path to the model file.
Normally that's what the output_uri=True in the Task.init(...) call is for, but it seems there is a bug that's not uploading the model.
Would you mind testing out manual model uploading ?
So in your case it should be adding these lines to the end of the train_model.py
output_model = OutputModel(task=task, framework="scikitlearn")
output_model.update_weights(weights_filename='sklearn-model.pkl')
They should make sure your model-url is a https link to a file saved on the clearml server
Oh wait, you have a self-hosted server?