like replace a model in staging seldon with this model from clearml; push this model to prod seldon, but in shadow mode

we are just entering the research phase for a centralized serving solution. Main reasons against clearml-serving triton are: 1) no support for kafka 2)no support for shadow deployments (both of these are supported by Seldon, which is currently the best=looking option for us)

in the far future - automatically. In the nearest future - more like semi-manually

self-hosted. Just upgraded to latest version today (1.1.1). The problem appeared when we were still using 1.0.2

I think they appeared when I had a lot of HPO tasks enqueued and not started yet, and then I decided to either Abort or Archive them - I don’t remember already

no new unremovable entries have appeared (although I haven’t tried)

I want to have 2 instances of scheduler - 1 starts reporting jobs for staging, another one for prod

I’ll make it more visible though

and this can break a lot of things, when somebody start the scheduler with an older version of clearml, saves the state, then upgrades and new clearml expects the state in another format

I already added to the task:
Workaround: Remove limit_execution_time from scheduler.add_task

I can try, but difficult to verify correctness without a publicly available test suite

in cloned tasks, the correct model is being applied, but the original one stays registered as input model

no, I’m providing the id of task which generated the model as a “hyperparam”

I had a bunch of training tasks each of which outputted a model. I want to apply each one of them to a specific dataset. I have a clearml task ( apply_model ) for that, which takes dataset_id and model-producing task_id as input. First time I initiate apply model by hardcoding ids and starting the run from my machine (it is then goes into cloud, when it reaches execute_remotely )

I am not registering a model explicitly in apply_model . I guess it is done automatically when I do this:
output_models = train_task_with_model.models["output"] model_descriptor = output_models[0] model_filename = model_descriptor.get_local_copy()

clearml==1.5.0
WebApp: 1.5.0-192 Server: 1.5.0-192 API: 2.18

I create the dataset like this:
` project_name = "Sandbox"
task_name = "get_raw_data"
task = Task.init(project_name=project_name,
task_name=task_name,
task_type=Task.TaskTypes.data_processing,

             )

dataset = Dataset.create(use_current_task=True)

adding some files here

dataset.upload(verbose=True)

dataset.finalize(verbose=True) `

I do see the “Data Processing” type task in UI together with all other dataset-related features, like lineage plot

system tags cannot be seen from UI. right?

here: https://github.com/allegroai/clearml/blob/master/clearml/datasets/dataset.py#L75

I see that in the end, both query functions are calling Task._query_tasks

I guess so, lemme double-check

Basically, my problem is that it returns empty result. In the same code I can get dataset by its ID and I can get the task (which created the dataset) using
Task.get_tasks()(without mentioning th ID explicitly)

if the task is of wrong type (not data_processing) - then it’ll get both correct type and correct system tag

but I don’t get to this line, because my task is already of type data_processing

So I thought, maybe I can tell clearml-session to use hostname from ngrok

For others, who haven’t heard about ngrok:
Ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels.

I guess that’s because ngrok is not like a Dynamic DNS

Here’s my workaround - ignore the fail messages, and manually create an SSH connection to the server with Jupyter port forwarded.

https://github.com/allegroai/clearml-session/issues/1