The package is just subdir by the way. So it should not be in installed packages anyways, right?
Correct, also when the agent is spinning the code it will automatically add the root of the git repository to the pythonpath so you should be able to load the package.
ReassuredTiger98
It seems like clearml is not able to fetch the dependencies correctly when
importlib
is used.
If you have an example please let me know we'll try to fix it :)
Is it possible to read the dependencies manually from a conda environment.yml?
You can set detect_with_conda_freeze: true in clearml.conf, it will just use the entire conda env
https://github.com/allegroai/clearml/blob/28b85028fe4da3ab963b69e8ac0f7feef73cfcf6/docs/clearml.conf#L170
When experimenting we use a entrypoint script which we pass the specific experiment to.
Perfect! That sounds like a good solution for me.
Don't know whether I do something wrong. Locally it works, but when executed via queue I get:
File "run_task.py", line 14, in <module> main() File "run_task.py", line 9, in main printme = importlib.import_module("some_package.file_to_import").printme File "/home/tim/.clearml/venvs-builds/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked ModuleNotFoundError: No module named 'some_package'
ReassuredTiger98 regrading the agent error, can you see the package some_packge in the "Installed Packages" in the UI? Was it installed ? are you using pip or conda as package manager in the agent (check the clearml.conf) is the agent running in docker mode ?
Here is the minimal example: https://github.com/mctigger/test_clearml
Nono, I got to thank you for this awesome tool!
I will create a minimal example.
Many thanks ReassuredTiger98 !
test_clearml , so directly from top-level.
Let me try it another time. Maybe something else went wrong.
Wait, nvm. I just tried it again and now it worked.
Thanks for your help again. I will just use detect_with_conda_freeze: true . Seems like a perfect solution for me!
The package is just subdir by the way. So it should not be in installed packages anyways, right?
Okay. It works now. I don't know what went wrong before. Probably a user error 😅
For example in our case we do reinforcement learning and the we would call a script like this: python run_openai_gym.py some_ http://package.my _agent .
Good to know!
I think the current solutions are fine. I will try it first and probably will have some more questions/problems 🙂
ReassuredTiger98 oh wow I did not realize you actually call importlib to import your libraries (any reason not to call import ?)
And yes, I think we will miss it as the package analysis is actually static text analysts of the code
What's the "working dir" ? (where in the repo the script is executed from)
ReassuredTiger98 I can verify the code snippet reproduces the issues with packages missing from "installed package".
If you feel this is important, please open a GitHub issue.
Also, you can manually add packages:
Task.add_requirements('package_name_here', 'optional version here')
So when you manually load the package you can make sure it will be listed, do remember to call it before the Task.init call.