
Reputation
Badges 1
147 × Eureka!no, Iām providing the id of task which generated the model as a āhyperparamā
also, I donāt see an edit button near input models
I think we need logging here: https://github.com/allegroai/clearml-session/blob/bf1851cd3831c19cc0eadd9b2ffc0613f97f16e1/clearml_session/main.py#L564
clearml==1.5.0
WebApp: 1.5.0-192 Server: 1.5.0-192 API: 2.18
For others, who havenāt heard about ngrok:Ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels.
well, I first run clearml-session to start everything on the remote machine, then I close the local process (while Interactive is still running on the remote machine)
but I donāt get to this line, because my task is already of type data_processing
and this can break a lot of things, when somebody start the scheduler with an older version of clearml, saves the state, then upgrades and new clearml expects the state in another format
I tried this, but didnāt help:input_models = current_task.models["input"] if len(input_models) == 1: input_model_as_input = {"name": input_models[0].name, "type": ModelTypeEnum.input} response = current_task.send(DeleteModelsRequest( task=current_task.task_id, models=[input_model_as_input] ))
yes, Iāll try it out
yeah, I think Iāll go with schedule_function
right now, but your proposed idea would make it even clearer.
like replace a model in staging seldon with this model from clearml; push this model to prod seldon, but in shadow mode
I think Iāll skip with PR: there is a related problem, that makes the fix (and especially its testing much more difficult): https://github.com/allegroai/clearml/issues/648#issuecomment-1102595620
gotcha, thanks!
I am not registering a model explicitly in apply_model
. I guess it is done automatically when I do this:output_models = train_task_with_model.models["output"] model_descriptor = output_models[0] model_filename = model_descriptor.get_local_copy()
Then I ssh into the remote machine using ngrok hostname and tunnel the port for Jupyter
` Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel
Warning: Permanently added '<CENSORED>' (ECDSA) to the list of known hosts.
Enter passphrase for key '/Users/jevgenimartjushev/.ssh/id_rsa': <CENSORED>
SSH tunneling failed, retrying in 3 seconds `
Did a small update: added a workaround and renamed the issue to include more client_facing conditionlimit_execution_time is present
instead of an implementation detail conditiontimeout_jobs are present
I can try, but difficult to verify correctness without a publicly available test suite
ideally, I want to hardcode, e.g. use_staging = True, enqueue it; and then via clone-edit_user_properties-enqueue in UI start the second instance
slightly related follow-up question: can I add user properties to a scheduler configuration?
not sure I fully get it. Where will the connection between task and scheduler appear?
so I assume itās somehow related to remote connection form VS Code
we are just entering the research phase for a centralized serving solution. Main reasons against clearml-serving triton are: 1) no support for kafka 2)no support for shadow deployments (both of these are supported by Seldon, which is currently the best=looking option for us)