What's the "working dir" ? (where in the repo the script is executed from)
Let me try it another time. Maybe something else went wrong.
Wait, nvm. I just tried it again and now it worked.
Thanks for your help again. I will just use detect_with_conda_freeze: true . Seems like a perfect solution for me!
For example in our case we do reinforcement learning and the we would call a script like this: python run_openai_gym.py some_ http://package.my _agent .
Okay. It works now. I don't know what went wrong before. Probably a user error 😅
test_clearml , so directly from top-level.
ReassuredTiger98 regrading the agent error, can you see the package some_packge in the "Installed Packages" in the UI? Was it installed ? are you using pip or conda as package manager in the agent (check the clearml.conf) is the agent running in docker mode ?
The package is just subdir by the way. So it should not be in installed packages anyways, right?
Correct, also when the agent is spinning the code it will automatically add the root of the git repository to the pythonpath so you should be able to load the package.
Good to know!
I think the current solutions are fine. I will try it first and probably will have some more questions/problems 🙂
Perfect! That sounds like a good solution for me.
ReassuredTiger98
It seems like clearml is not able to fetch the dependencies correctly whenÂ
importlib
 is used.
If you have an example please let me know we'll try to fix it :)
Is it possible to read the dependencies manually from a conda environment.yml?
You can set detect_with_conda_freeze: true in clearml.conf, it will just use the entire conda env
https://github.com/allegroai/clearml/blob/28b85028fe4da3ab963b69e8ac0f7feef73cfcf6/docs/clearml.conf#L170
Don't know whether I do something wrong. Locally it works, but when executed via queue I get:
File "run_task.py", line 14, in <module> main() File "run_task.py", line 9, in main printme = importlib.import_module("some_package.file_to_import").printme File "/home/tim/.clearml/venvs-builds/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked ModuleNotFoundError: No module named 'some_package'
ReassuredTiger98 oh wow I did not realize you actually call importlib to import your libraries (any reason not to call import ?)
And yes, I think we will miss it as the package analysis is actually static text analysts of the code
Nono, I got to thank you for this awesome tool!
The package is just subdir by the way. So it should not be in installed packages anyways, right?
I will create a minimal example.
Many thanks ReassuredTiger98 !
ReassuredTiger98 I can verify the code snippet reproduces the issues with packages missing from "installed package".
If you feel this is important, please open a GitHub issue.
Also, you can manually add packages:
Task.add_requirements('package_name_here', 'optional version here')
So when you manually load the package you can make sure it will be listed, do remember to call it before the Task.init call.
When experimenting we use a entrypoint script which we pass the specific experiment to.
Here is the minimal example: https://github.com/mctigger/test_clearml