Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Is There Any Way To Clear The Installed Packages Of A Task Programmatically? (I.E. Using The Python Sdk And Not The Ui)

Is there any way to clear the installed packages of a task programmatically? (i.e. using the python sdk and not the UI)

  
  
Posted 3 years ago
Votes Newest

Answers 22


task = Task.init(...) if task.running_locally(): # wait for the repo detection and requirements update task._wait_for_repo_detection() # reset requirements task._update_requirements(None)Regarding this, does this work if the task is not running locally and is being executed by the trains agent?

  
  
Posted 3 years ago

Regrading the missing packages, you might want to test with:
force_analyze_entire_repo: falsehttps://github.com/allegroai/trains/blob/c3fd3ed7c681e92e2fb2c3f6fd3493854803d781/docs/trains.conf#L162

Or if you have a full venv you like to store instead:
https://github.com/allegroai/trains/blob/c3fd3ed7c681e92e2fb2c3f6fd3493854803d781/docs/trains.conf#L169

BTW:
What is the missed package?

  
  
Posted 3 years ago

I also tried detection with pip freeze and that didn’t work

  
  
Posted 3 years ago

AgitatedDove14 I tried your code snippet, something like this:
task = Task.init(...) if task.running_locally(): # wait for the repo detection and requirements update task._wait_for_repo_detection() # reset requirements task._update_requirements(None) task.execute_remotely(queue="default") ... rest of task here ...It’s able to create a draft successfully with no items in the installed packages section, but it only installed Cython and began the execution immediately. None of the requirements from requirements.txt were installed. Any workaround for this?

  
  
Posted 3 years ago

Regarding this, does this work if the task is not running locally and is being executed by the trains agent?

This line: "if task.running_locally():" makes sure that when the code is executed by the agent it will not reset it's own requirements (the agent updates the requirements/installed_packages after it installs them from the requiremenst.txt, so that later you know exactly which packages/versions were used)

  
  
Posted 3 years ago

This line: “if task.running_locally():” makes sure that when the code is executed by the agent it will not reset it’s own requirements (the agent updates the requirements/installed_packages after it installs them from the requiremenst.txt, so that later you know exactly which packages/versions were used)

got it! Thanks for explaining

  
  
Posted 3 years ago

I was thinking more along the lines of a draft task that has yet to be enqueued(so it can still be edited). The installed packages grabs most packages except for a few, which is why I am resorting to clearing the packages so that requirements.txt is used. I’ve been using the UI so far to do it, but would like to move the process to a script

  
  
Posted 3 years ago

Let me experiment with force_analyze_entire_repo and I’ll get back to you! It was missing google-cloud-bigquery , an import from a separate file in the repository. Hopefully that should resolve the problem. A flag would be really cool, just in case if theres any problem with the package analysis.

  
  
Posted 3 years ago

GiddyTurkey39

A flag would be really cool, just in case if theres any problem with the package analysis.

Trying to think if this is a system wide flag (i.e. trains.conf) or a flag in task.init.
What do you think?

  
  
Posted 3 years ago

Regrading resetting it via code, if you need I can write a few lines for you to do that , although that might be a bit hacky.
Maybe we should just add a flag saying, use requirements.txt ?
What do you think?

  
  
Posted 3 years ago

Hi GiddyTurkey39
Are you referring to an already executed Task or the current running one?
(Also, what is the use case here? is it because the "installed packages are in accurate?)

  
  
Posted 3 years ago

GiddyTurkey39
BTW: you can always add the missing package via code:
Task.add_requirements('torch', optional_version)

  
  
Posted 3 years ago

Since the package requirements vary from task to task, setting it up in trains.conf might be a little tedious to go back and change. Most of the time, the package analysis has worked very well, but in this particular use case it failed to detect. I think task.init flag would be great!

  
  
Posted 3 years ago

just reran and got the same results with the flag for force_analyze_entire_repo set to false 😞 still has the following import error:
ImportError: cannot import name 'bigquery' from 'google.cloud' (unknown location)

  
  
Posted 3 years ago

task = Task.init(...) if task.running_locally(): # wait for the repo detection and requirements update task._wait_for_repo_detection() # reset requirements task._update_requirements(None)🙂

  
  
Posted 3 years ago

If you have a few lines for resetting, I would love to have that as an option as well!

  
  
Posted 3 years ago

I think task.init flag would be great!

👍

  
  
Posted 3 years ago

It was set to true earlier, I changed it to false to see if there would be any difference but doesn’t seem like it

  
  
Posted 3 years ago

GiddyTurkey39 my bad 😞 try this one
task._update_requirements({})

  
  
Posted 3 years ago

force_analyze_entire_repo to to True 🙂
(false is the default)

  
  
Posted 3 years ago

AgitatedDove14 worked like a charm, thank you!

  
  
Posted 3 years ago

It was set to true earlier, I changed it to false to see if there would be any difference but doesn’t seem like it

I would actually just add:
Task.add_requirements('google.cloud')Before the Task.init call (Notice, it has to be before the the init call)

  
  
Posted 3 years ago
929 Views
22 Answers
3 years ago
one year ago
Tags