task = Task.init(...) for foo in foos: data = foo.run() for key,value in data.items(): if key in 'auc', 'f1', etc: logger.log(key, value) elif key.startswith('model'): savemodel etc

Apparently our devops guy figured it out that you needed to have a different port number and a different docker container given 8080 was already occupied

Is it possible that the Prerocess class is the problem?

I guess so, this was done by our DevOps guy and he said he is following instructions

Is there a more detailed logging about what's going on?

TBH our Preprocess class has an import in it that points to a file that is not part of the preprocess.py so I have no idea how you think this can work.

apparently it did not work

Yeah, I found it, thanks.

I also found that you should have a deterministic ordering before you apply a fixed seed random sampling or else you will have a lot of head-scratching and assertion errors...

What I try to do is that DSes have some lightweight baseclass that is independent of clearml they use and a framework have all the clearml specific code. This will allow them to experiment outside of clearml and only switch to it when they are in an OK state. This will also help not to pollute clearml spaces with half backed ideas

but here I can tell them: return a dictionary of what you want to save

Is there any restriction on what can be in preprocess.py?

everything is weird about this. I put two models in the same endpoint, then only one was running, then I started another docker container having a different port number and then the curls with the new model endpoint (with the new port) started working

I want the model to be stored in a way that clearml-serving can recognise it as a model

use_current_task is False by default, let me see if this helps

I passed an env variable to the docker container so I figure this out

while in our own code:
if model_type == 'XGBClassifier': model = XGBClassifier() model.load_model(filename)

now on to the next pain point:

The DSes would expect the same interface as they used in the code that saved the model (me too TBH)

I'd rather just fail if they try to use a model that is unknown.

it worked!!!!

I think I figured this out but now I have a problem:

auto_connect_frameworks={ 'xgboost': False, 'scikitlearn': False }

I put two models in the same endpoint, then only one was running,

Sorry I wanted to say "service id"

Same service-id but different endpoints

also random tasks are popping up in the DevOps project in the UI

And then get_model is what I wrote above, just uses the CML API to pick up the right model from the task_id and model_name and the model config contains the class name so get_model has an if/else structure in it to create the right class.

yeah so in docker run:
-e TASK_ID='b5f339077b994a8ab97b8e0b4c5724e1' \ -e MODEL_NAME='best_model' \and then in Preprocess:
self.model = get_model(task_id=os.environ['TASK_ID'], model_name=os.environ['MODEL_NAME'])

Ok, but I need to think with the head of the DS, this way they only need to remember (and me only need to teach them where to find) one id.

I expect the task to be the main entry point for all their work, and the above interface is easy to remember, check etc etc. Also it is the same as getting artifacts so less friction.

` def get_task(task_id):
return Task.get_task(task_id)

def get_artifact(task_id, artifact_name):
task = Task.get_task(task_id)
return task.artifacts[artifact...

Yes, this is exactly how I solved it at the end

because we already had these get_artifact(), get_model() functions that the DSes use to get the data into notebooks to further analyse their stuff, I might as well just use those with a custom preprocess and call the predict myself.

I pass the IDs to the docker container as environment variables, so this does need restart for the docker container but I guess we can live with that for now

TBH ClearML doesn't seem to be picking the model up so I need to do it manually

That's because then I need to teach every DS how to use the ClearML api