Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Another Question: Is It Possible To Read The Dependencies Manually From A Conda Environment.Yml? It Seems Like Clearml Is Not Able To Fetch The Dependencies Correctly When

Another question: Is it possible to read the dependencies manually from a conda environment.yml? It seems like clearml is not able to fetch the dependencies correctly when importlib is used.

  
  
Posted 3 years ago
Votes Newest

Answers 22


ReassuredTiger98

It seems like clearml is not able to fetch the dependencies correctly when 

importlib

 is used.

If you have an example please let me know we'll try to fix it :)

Is it possible to read the dependencies manually from a conda environment.yml?

You can set detect_with_conda_freeze: true in clearml.conf, it will just use the entire conda env
https://github.com/allegroai/clearml/blob/28b85028fe4da3ab963b69e8ac0f7feef73cfcf6/docs/clearml.conf#L170

  
  
Posted 3 years ago

The package is just subdir by the way. So it should not be in installed packages anyways, right?

Correct, also when the agent is spinning the code it will automatically add the root of the git repository to the pythonpath so you should be able to load the package.

  
  
Posted 3 years ago

Here is the minimal example: https://github.com/mctigger/test_clearml

  
  
Posted 3 years ago

For example in our case we do reinforcement learning and the we would call a script like this: python run_openai_gym.py some_ http://package.my _agent .

  
  
Posted 3 years ago

What's the "working dir" ? (where in the repo the script is executed from)

  
  
Posted 3 years ago

test_clearml , so directly from top-level.

  
  
Posted 3 years ago

Wait, nvm. I just tried it again and now it worked.

  
  
Posted 3 years ago

Just like in the git repo.

  
  
Posted 3 years ago

Thanks for your help again. I will just use detect_with_conda_freeze: true . Seems like a perfect solution for me!

  
  
Posted 3 years ago

Okay. It works now. I don't know what went wrong before. Probably a user error 😅

  
  
Posted 3 years ago

Let me try it another time. Maybe something else went wrong.

  
  
Posted 3 years ago

ReassuredTiger98 I can verify the code snippet reproduces the issues with packages missing from "installed package".
If you feel this is important, please open a GitHub issue.
Also, you can manually add packages:
Task.add_requirements('package_name_here', 'optional version here')
So when you manually load the package you can make sure it will be listed, do remember to call it before the Task.init call.

  
  
Posted 3 years ago

I will create a minimal example.

Many thanks ReassuredTiger98 !

  
  
Posted 3 years ago

Perfect! That sounds like a good solution for me.

  
  
Posted 3 years ago

I will create a minimal example.

  
  
Posted 3 years ago

Don't know whether I do something wrong. Locally it works, but when executed via queue I get:

File "run_task.py", line 14, in <module> main() File "run_task.py", line 9, in main printme = importlib.import_module("some_package.file_to_import").printme File "/home/tim/.clearml/venvs-builds/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked ModuleNotFoundError: No module named 'some_package'

  
  
Posted 3 years ago

ReassuredTiger98 oh wow I did not realize you actually call importlib to import your libraries (any reason not to call import ?)
And yes, I think we will miss it as the package analysis is actually static text analysts of the code

  
  
Posted 3 years ago

When experimenting we use a entrypoint script which we pass the specific experiment to.

  
  
Posted 3 years ago

Nono, I got to thank you for this awesome tool!

  
  
Posted 3 years ago

The package is just subdir by the way. So it should not be in installed packages anyways, right?

  
  
Posted 3 years ago

ReassuredTiger98 regrading the agent error, can you see the package some_packge in the "Installed Packages" in the UI? Was it installed ? are you using pip or conda as package manager in the agent (check the clearml.conf) is the agent running in docker mode ?

  
  
Posted 3 years ago

Good to know!
I think the current solutions are fine. I will try it first and probably will have some more questions/problems 🙂

  
  
Posted 3 years ago
555 Views
22 Answers
3 years ago
one year ago
Tags