interesting if I run the script from the repo main directory with
python code/run.py
it still gives me the same error message

clearml.Repository Detection - WARNING - Can't get diff information for git repo in repo/code

python='python3' ~/anaconda3/envs/.venv/bin/python3

clearml==1.7.1

yeah

clearml-serving needs a "Preprocess" class in a file that is added to the endpoint.

wait

Python 3.8.13

git status gives correct information

well, it just shoved the dataset files with cryptic data_1/data_2 names among the artifacts.

I am not sure I want to do that

but I am one level lower than top. so:

~/work/repo is the main repo dir

~/work/repo/code/run.py and I am running python run.py in ~/work/repo/code

I guess so, this was done by our DevOps guy and he said he is following instructions

Is there a more detailed logging about what's going on?

nah, it runs about 1 minute of standards SQL->dataframes->xgboost pipeline with some file saving

This is very different than from in the xgboost example:

https://github.com/allegroai/clearml-serving/blob/main/clearml_serving/preprocess/preprocess_template.py

ahh, because task_id is the "real" id of a task

I am running a script

Yeah, I found it, thanks.

I also found that you should have a deterministic ordering before you apply a fixed seed random sampling or else you will have a lot of head-scratching and assertion errors...

This receives the payload from the server and turns it into something that can be fed to the model. This in out case depends on a data structure that is stored on the clearml server as an artifact. I need to communicate this to the class so it can pick it up and use it when called

This was not something I was expecting to break.

If you enable nbdime globally and switch virtual environments, then git diff will fail.

{"detail":"Error processing request: ('Expecting data to be a DMatrix object, got: ', <class 'pandas.core.frame.DataFrame'>)"}

🙂

docker-compose I guess

also random tasks are popping up in the DevOps project in the UI

Apparently our devops guy figured it out that you needed to have a different port number and a different docker container given 8080 was already occupied

git-nbdiffdriver diff: git-nbdiffdriver: command not found fatal: external diff died, stopping at ...

I didn't realise that pickling is what triggers clearml to pick it up. I am actually saving a dictionary that contains the model as a value (+ training datasets)

What I try to do is that DSes have some lightweight baseclass that is independent of clearml they use and a framework have all the clearml specific code. This will allow them to experiment outside of clearml and only switch to it when they are in an OK state. This will also help not to pollute clearml spaces with half backed ideas

python run.py param1 param2