This worked ✔

Cool, now I understand the auto detection better

Very nice thanks, I'm going to try the SA server + agents setup this week, let's see how it goes ✌

I don't even know where trains is coming from... While using the same environment I can't even import trains, see

I never installed trains on this environment

no this is from the task execution that failed

` alabaster==0.7.12
appdirs==1.4.4
apturl==0.5.2
attrs==21.2.0
Babel==2.9.1
bcrypt==3.1.7
blinker==1.4
Brlapi==0.7.0
cachetools==4.0.0
certifi==2019.11.28
chardet==3.0.4
chrome-gnome-shell==0.0.0
clearml==1.0.5
click==8.0.1
cloud-sptheme==1.10.1.post20200504175005
cloudpickle==1.6.0
colorama==0.4.3
command-not-found==0.3
cryptography==2.8
cupshelpers==1.0
cycler==0.10.0
Cython==0.29.24
dbus-python==1.2.16
decorator==4.4.2
defer==1.0.6
distlib==0.3.1
distro==1.4.0
distro-info===0.23ubuntu1
doc...

So the scale will also appear?

Gotcha

` # Python 3.8.10 (default, Jun 2 2021, 10:49:15) [GCC 9.4.0]
clearml == 1.0.5
hyperopt == 0.2.5
matplotlib == 3.4.3
numpy == 1.21.2
pandas == 1.3.2
plotly == 5.3.0
python_dateutil == 2.8.2
scikit_learn == 0.24.2
statsmodels == 0.12.2
tqdm == 4.62.2

Detailed import analysis

**************************

IMPORT PACKAGE clearml

tasks/data_projection.py: 9

tasks/hp_optimization.py: 6

tasks/hpo_n_best_evaluation.py: 6

tasks/pipelines/monthly_predictions.py: 4

IMPORT PACKAGE hypero...

Continuing on this line of thought... Is it possible to call task.execute_remotely on a CPU only machine (data scientists' laptop for example) and make the agent that fetches this task to run it using GPU? I'm asking that because it is mentioned that it replicates the running environment on the task creator... which is exactly what I'm not trying to do 😄

I see it's a plotly plot, even though I report a matplotlib one

TimelyPenguin76 this fixed it, using the detect_with_pip_freeze as true solves the issue

I think a good idea is to add to the error message when the clearml agent fails due to import error, a suggestion ot try out with pip freeze

But does it disable the agent? or will the tasks still wait for the agent to dequeue?

Can you lend a few a words about how the not-pip freeze mechanism of detecting packages work?

In the larger context I'd look on how other object stores treat similar problems, I'm not that advanced in these topics.

But adding a simple force_download flag to the get_local_copy method could solve many cases I can think of, for example I'd set it to true in my case as I don't mind the times it will re-download when not necessary as it is quite small (currently I always delete the local file, but it looks pretty ugly)

actually i was thinking about model that werent trained uaing clearml, like pretrained models etc

found it here:
https://github.com/allegroai/trains/blob/98ea965e6d0a39019b31e5fec146ab55da0d52e8/trains/storage/manager.py#L41

Also being able to separate their configurations files would be good (maybe there is and I don't know?)

I'm quite confused... The package is not missing, it is in my environment and executing tasks normally ( python my_script.py.... ) works

And once this is done, what is the file server IP good for? will it redirect to the bucket?

Thia is just keeping getting better and better.... 🤩

There are many ohter packages in my environment which are not listed

😄

pgrep -af trains shows that there is nothing running with that name

TimelyPenguin76 I think our problem is that the agent is not using this environment, I'm not sure which one he does... Is there a way to hard-code the agent environment?

I'll try it

Saving part from task A:

pipeline = trials.trials[index]['result']['pipeline'] output_prefix = 'best_iter_' if i == 0 else 'iter_' task.upload_artifact(name=output_prefix + str(index), artifact_object=pipeline)

I double checked that the task object is the correct one