Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi, With Clearml-Agent 1.5.1, I Tried To Run An Experiment Within A Docker With Image Python3:8 And It Failed Executing The Task While Trying To Call Python3.9. I Am Not Sure Why It'S Using Python3.9, Since The Agent.Default_Python Is 3.8 And The Image Is

Hi, with clearml-agent 1.5.1, I tried to run an experiment within a docker with image python3:8 and it failed executing the task while trying to call python3.9. I am not sure why it's using python3.9, since the agent.default_python is 3.8 and the image is python:3.8
I created the task locally with python3.9, could that be the reason? (task.data.script.binary == 'python3.9')
Here is the log of the experiment.

  
  
Posted one year ago
Votes Newest

Answers 19


Hi SmugDolphin23 thanks for the input! Will try now but that seems hacky: to have it working I have to specify python3.8 two times:
one in the agent config file (agent.default_python is already python3.8, but seems to be ignored) + make sure it is available (using python:3.8 docker image)Is there a way to prevent this redundancy? Ie. If I want to change the python version, I can control it from a single place?

  
  
Posted one year ago

Yes, so even if you use a docker image with 3.8, the agent doesn't really know that you have 3.8 installed. If it is ran with 3.9, it will assume that is the desired version you want to use. So you need to change it in the config.
Not really sure why default_python is ignored (we will need to look into this), but python_binary should work...

  
  
Posted one year ago

SmugDolphin23 Actually adding agent.python_binary didn't work, it was not read by the clearml agent (in the logs dumped by the agent, agent.python_binary = (no value)

  
  
Posted one year ago

So either I specify in the clearml-agent agent.python_binary: python3.8 as you suggested, or I enforce the task locally to run with python3.8 using task.data.script.binary

  
  
Posted one year ago

And since I ran the task locally with python3.9, it used that version in the docker container

  
  
Posted one year ago

JitteryCoyote63 very odd, it seems to work just fine on my machine

  
  
Posted one year ago

can you share your config? (make sure to remove any credentials)

  
  
Posted one year ago

JitteryCoyote63 This is odd you have both python3.9 and python3.8 on the container, and since it says (probably) ob the task the agent should run python3.9 it's trying to use it for creating the enthronement (it does not matter that agent is using python3.8).
1673431344706 agent-1 DEBUG /usr/bin/python3.9 /usr/bin/python3.9: No module named pipThis is the main issue, pip is missing for python3.9 and this is why it reverted to python 3.8 when it was setting the environment.
It should probably have the same logic when running the actual Task (not just when installing the agent inside the container)
Does that make sense ?

  
  
Posted one year ago

I think my problem is that I am launching an experiment with python3.9 and I expect it to run in the agent with python3.8. The inconsistency is from my side, I should fix it and create the task with python3.8 with:
task.data.script.binary = "python3.8" task._update_script(convert_task.data.script)Or use python:3.9 when starting the agent

  
  
Posted one year ago

Or use python:3.9 when starting the agent

This is probably the best solution 🙂

  
  
Posted one year ago

SmugDolphin23 which agent version did you use?

  
  
Posted one year ago

What I put in the clearml.conf is the following:

agent.package_manager.pip_version = "==20.2.3" agent.package_manager.extra_index_url: [" "] agent.python_binary = python3.8

  
  
Posted one year ago

Should I open an issue in github clearml-agent repo?

  
  
Posted one year ago

SuccessfulKoala55 1.5.1

  
  
Posted one year ago

Not really because this is difficult to control: I use the AWS autoscaler with ubuntu AMI and when an instance is created, packages are updated, and I don't know which python version I get, + changing the python version of the OS is not really recommended

  
  
Posted one year ago

I have a mental model of the clearml-agent as a module to spin my code somewhere, and the python version running my code should not depend of the python version running the clearml-agent (especially for experiments running in containers)

  
  
Posted one year ago

packages are updated, and I don't know which python version I get, + changing the python version of the OS is not really recommended

Wait I'm confused, this is inside a container, no?

and the python version running my code should not depend of the python version running the clearml-agent (especially for experiments running in containers)

Generally speaking you are correct, but some packages will not have the same version for all python versions

Specifically in this case I think we could improve the agent so it will not get "confused" between the multiple python versions
Could you open an Issue for that? and also help in understanding how to quickly reproduce it ?

  
  
Posted one year ago

# Set the python version to use when creating the virtual environment and launching the experiment # Example values: "/usr/bin/python3" or "/usr/local/bin/python3.6" # The default is the python executing the clearml_agent python_binary: "" # ignore any requested python version (Default: False, if a Task was using a # specific python version and the system supports multiple python the agent will use the requested python version) # ignore_requested_python_version: trueHo this explains it well actually. My surprise is that using python:3.8 docker image, it finds a python3.9 executable, I was not expecting it

  
  
Posted one year ago

Hi JitteryCoyote63 ! Your clearml-agent is likely ran with python3.9. Can try setting this entry https://github.com/allegroai/clearml-agent/blob/ebb955187dea384f574a52d059c02e16a49aeead/docs/clearml.conf#L48 in your clearml.conf to python3.8 , or the full path to python3.8 if that doesn't work

  
  
Posted one year ago
612 Views
19 Answers
one year ago
one year ago
Tags
Similar posts