I am noticing that the files are saved locally, is there any chance that the files are over-written during the run or get deleted at some point and then replaced?
Yes they are local - I don't think there is a possibility they are getting overwritten... But that depends on how clearml names them. I showed you the code that saves the artifacts, but this code runs multiple times from a given template with different values - essentially it creates like 10 times the same task with different param...

Happens on my Firefox aswell

It's working! 😄

:face_palm: 🤔 :man-tipping-hand:

But does it disable the agent? or will the tasks still wait for the agent to dequeue?

I can easily see how this problem can reoccur

happens on my phone as well :man-facepalming:

😶‍🌫️

inference table is a pandas dataframe

yeah I guessed so

O_O

Version 1.1.1

Snippet of which part exactly?

Yeah, logs saying "file not found", here is an example

Here you go

I assume we are talking about the IP I would find here right?
https://www.whatismyip.com/

hahaha

When you are inside a project, the search bar searches for experiments

so if you want to search inside a specific project, go to that project and use the search bar, if you want to search all over, go to the project called "All Experiments" and search there

moreover, in each pipeline I have 10 different settings of task A -> Task b (and then task C), each run 1-2 fails randomly

sorry I think it trimmed it

this

alabaster==0.7.12 appdirs==1.4.4 apturl==0.5.2 attrs==21.2.0 Babel==2.9.1 bcrypt==3.1.7 blinker==1.4 Brlapi==0.7.0 cachetools==4.0.0 certifi==2019.11.28 chardet==3.0.4 chrome-gnome-shell==0.0.0 clearml==1.0.5 click==8.0.1 cloud-sptheme==1.10.1.post20200504175005 cloudpickle==1.6.0 colorama==0.4.3 command-not-found==0.3

I don't even know where trains is coming from... While using the same environment I can't even import trains, see

` alabaster==0.7.12
appdirs==1.4.4
apturl==0.5.2
attrs==21.2.0
Babel==2.9.1
bcrypt==3.1.7
blinker==1.4
Brlapi==0.7.0
cachetools==4.0.0
certifi==2019.11.28
chardet==3.0.4
chrome-gnome-shell==0.0.0
clearml==1.0.5
click==8.0.1
cloud-sptheme==1.10.1.post20200504175005
cloudpickle==1.6.0
colorama==0.4.3
command-not-found==0.3
cryptography==2.8
cupshelpers==1.0
cycler==0.10.0
Cython==0.29.24
dbus-python==1.2.16
decorator==4.4.2
defer==1.0.6
distlib==0.3.1
distro==1.4.0
distro-info===0.23ubuntu1
doc...

Anyway I checked the base task, and this is what it has in installed packages (seems like it doesn't list all the real packages in the environment)

I'm quite confused... The package is not missing, it is in my environment and executing tasks normally ( python my_script.py.... ) works

TimelyPenguin76 I think our problem is that the agent is not using this environment, I'm not sure which one he does... Is there a way to hard-code the agent environment?

` # Python 3.8.10 (default, Jun 2 2021, 10:49:15) [GCC 9.4.0]
clearml == 1.0.5
hyperopt == 0.2.5
matplotlib == 3.4.3
numpy == 1.21.2
pandas == 1.3.2
plotly == 5.3.0
python_dateutil == 2.8.2
scikit_learn == 0.24.2
statsmodels == 0.12.2
tqdm == 4.62.2

Detailed import analysis

**************************

IMPORT PACKAGE clearml

tasks/data_projection.py: 9

tasks/hp_optimization.py: 6

tasks/hpo_n_best_evaluation.py: 6

tasks/pipelines/monthly_predictions.py: 4

IMPORT PACKAGE hypero...

this is the full one TimelyPenguin76

I set it to true and restarted by agent

cool the /login worked

when spinning up the ami i just went for trains recommended settings