Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Any Idea Why Only A Single Instance Of Mujoco Can Be Run With Clearml-Agent? I Run 2 Clearm-Agents, One Per Gpu On My Workstation. However, The Second Task Failes With One Of The Following Errors:

Any idea why only a single instance of mujoco can be run with clearml-agent? I run 2 clearm-agents, one per GPU on my workstation. However, the second task failes with one of the following errors:

File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/suite/acrobot.py", line 20, in <module> from dm_control import mujoco File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/mujoco/__init__.py", line 18, in <module> from dm_control.mujoco.engine import action_spec File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/mujoco/engine.py", line 41, in <module> from dm_control.mujoco import index File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/mujoco/index.py", line 88, in <module> from dm_control.mujoco.wrapper import util File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/mujoco/wrapper/__init__.py", line 20, in <module> from dm_control.mujoco.wrapper.core import callback_context File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/mujoco/wrapper/core.py", line 29, in <module> from dm_control.mujoco.wrapper.mjbindings import functions File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/mujoco/wrapper/mjbindings/functions.py", line 17, in <module> mjlib = util.get_mjlib() File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/site-packages/dm_control/mujoco/wrapper/util.py", line 120, in get_mjlib return ctypes.cdll.LoadLibrary(library_path) File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/ctypes/__init__.py", line 442, in LoadLibrary return self._dlltype(name) File "/home/me/.clearml/venvs-builds.1/3.7/lib/python3.7/ctypes/__init__.py", line 364, in __init__ self._handle = _dlopen(self._name, mode) OSError: /home/me/.mujoco/mujoco200_linux/bin/libmujoco200.so: undefined symbol: __glewBindBuffer
or

WARNING:absl:mjbindings failed to import mjlib and other functions. libmujoco.so may not be accessible.

However, when I start the scripts directly on the workstation without using clearml-agent, everything works fine.

EDIT:
The cause of the error itself is not so important. It seems like this is because my script needs MUJOCO_GL="egl" to be set. In "Configuration -> EnvironmentI can see that the Task has registered the environment variable. However, the script fails. When I do export MUJOCO_GL="egl" ` before starting the clearml-agent, my script works! Any idea why this is the case?

  
  
Posted 3 years ago
Votes Newest

Answers 19


Hi ReassuredTiger98
Are you running the agent in venv mode ?

  
  
Posted 3 years ago

How can I see that?
I have venv_update.enabled: true and detect_with_conda_freeze: true

  
  
Posted 3 years ago

ReassuredTiger98 what is the command line you use to start the ClearML Agents?

  
  
Posted 3 years ago

Just tested it again. Here is my config:
https://gist.github.com/mctigger/086c5f8071a604605e9f7a172800b51d
In the Web UI under Configuration -> Hyper Parameters -> Environment I can see the following:
MUJOCO_GL osmesa

  
  
Posted 3 years ago

ReassuredTiger98 the environment is currently only set in runtime of the process (not before), this will change in the next RC of trains-agent (due is a few days)

  
  
Posted 3 years ago

What exactly does this mean? The environment is set after the script is started?

  
  
Posted 3 years ago

via Python?

  
  
Posted 3 years ago

Well (yes, I think), the environment section is used mostly for logging, the next version will have full support by the clearml-agent (due next week), and the next release of clearml-server will add basj-script support.

  
  
Posted 3 years ago

Since you are running in venv mode, adding the OS environment before the clearml-agent, will basically make sure it will propagate to the process itself.
ReassuredTiger98 make sense ?

  
  
Posted 3 years ago

I forgot to add this:
` Here is my error:

Traceback (most recent call last):
File "src/run_gym.py", line 25, in <module>
print(os.environ["MUJOCO_GL"])
File "/home/tim/.clearml/venvs-builds/3.7/lib/python3.7/os.py", line 681, in getitem
raise KeyError(key) from None
KeyError: 'MUJOCO_GL' `
This is at the top of my script.

  
  
Posted 3 years ago

If I understood correctly, if I tried to print(os.environ["MUJOCO_GL"]) after the clearml Task is created, this should be set?

  
  
Posted 3 years ago

Let me check ....

  
  
Posted 3 years ago

Correct

  
  
Posted 3 years ago

So the environment variables are not set by the clearml-agent, but by clearml itself

  
  
Posted 3 years ago

Yes

  
  
Posted 3 years ago

Okay, now I understand!

  
  
Posted 3 years ago

Thank you very much.

  
  
Posted 3 years ago

My pleasure

  
  
Posted 3 years ago

As always, great help!

  
  
Posted 3 years ago