clearml.utilities.pigar.main.GenerateReqs.extract_reqs

` Remote machine is ready
Setting up connection to remote session
Starting SSH tunnel
Warning: Permanently added '<CENSORED>' (ECDSA) to the list of known hosts.
Enter passphrase for key '/Users/jevgenimartjushev/.ssh/id_rsa': <CENSORED>

SSH tunneling failed, retrying in 3 seconds `

Also added implementation thought to the issue

is it possible to override this?

SuccessfulKoala55 any ideas or should we restart?

“To have the Full pip freeze as “installed packages” - that’s exactly what I’m trying to prevent. Locally my virtualenv has all the dependencies for all the clearml tasks, which is fine because I don’t need to download and install them every time I launch a task. But remotely I want to keep the bare minimum needed for the concrete task. Which clearml successfully does, as long as I don’t import any local modules.

I would probably like to see a fully-blown example with other market leading technologies covering parts which are missing from clear-ml. E.g. clearml+feast+seldon

I’m rather sure that after restart everything will be back to normal. Do you want me to invoke smth via SDK or REST while the server is still in this state?

when I go into Dataset.list_datasets with the debugger and remove system_tags=[“dataset”] from api call params - I get the correct response back

for the tasks that are not deleted, log is different:
[2021-09-09 12:19:07,718] [8] [WARNING] [clearml.service_repo] Returned 400 for tasks.dequeue in 4ms, msg=Invalid task id: status=stopped, expected=queued

For me - workaround is totally acceptable, thus scheduler is once again usable for me.

https://github.com/allegroai/clearml-session/issues/1

Maybe it makes sense to use schedule_function instead of schedule_task_id and then the schedule function will perform the cloning of the last task and starting the clone?

thanks for info

self-hosted. Just upgraded to latest version today (1.1.1). The problem appeared when we were still using 1.0.2

but we run everything in docker containers. Will it still help?

if you call Task.init in your entire repo (serve/train) you end up with "installed packages" section that contains all the required pacakges for both use cases ?
yes, and I thought that it is looking at what libraries are installed in virtualenv, but you explained that it rather doing a static analysis over whole repo.

first analyze the entry point script, if it does not contain other to local files

I am importing a module which is in the same folder as the main one (i.e. in the same package)

[.]$ /root/.clearml/venvs-builds/3.8/bin/python -u '/root/.clearml/venvs-builds/3.8/code/-m filprofiler run catboost_train.py' doesn’t look good

I think they appeared when I had a lot of HPO tasks enqueued and not started yet, and then I decided to either Abort or Archive them - I don’t remember already

I tried both, didn’t work

and ClearML should strive to be clear, amirite? 😄

Also, installed packages are also incorrect (not including ones that I install fmor within the notebook using !pip install package_name_here )

` # Python 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
azure_storage_blob == 12.8.0
boto3 == 1.17.30
clearml == 0.17.5
google_cloud_storage == 1.36.2
ipykernel == 5.5.0

Detailed import analysis

**************************

IMPORT PACKAGE azure_storage_blob

clearml.storage: 0

IMPORT PACKAGE boto3

clearml.storage: 0

IMPORT PACKA...

exactly what I’m talking about

we are just entering the research phase for a centralized serving solution. Main reasons against clearml-serving triton are: 1) no support for kafka 2)no support for shadow deployments (both of these are supported by Seldon, which is currently the best=looking option for us)

all subsequent invocations are done by cloning this task in UI and changing the model task_id

ok, so if it goes over whole repository, then my question transforms into: how to make sure it will traverse only current package? I have separate packages for serving and training in a single repo. I don’t want serving requirements to be installed.

need to check with infra engineers

as I understand, it uses tensorboard from C++ code