Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Hi

Hi !

https://github.com/allegroai/clearml-agent/releases/tag/0.17.2 is out, with an exciting new feature:

🎉 Virtual Environments Caching 🎉

Also supported for conda and when using docker!
To enable, see https://github.com/allegroai/clearml-agent/blob/205f9dd81620fcec5aa155991afbbf711f3cd648/docs/clearml.conf#L101 . For more information, see the https://github.com/allegroai/clearml-agent/releases/tag/0.17.2 🙂

  
  
Posted 3 years ago
Votes Newest

Answers 30


I didn't know about that one! I'll try it, thanks!!

  
  
Posted 3 years ago

Hi! I'm tryin to find a workaround for this: can't do pip install <name_of_package>
I executed the task, and it created a cache venv.
But when running the code, it couldn't import the package because it wasn't listed.
I then sourced the venv, and manually installed the package
so if I do python -c 'import object_detection' if works

  
  
Posted 3 years ago

can this be solved by using a docker image with the preinstalled packages at a user level?

  
  
Posted 3 years ago

You might need to update it to the latest Detection API 😞

  
  
Posted 3 years ago

should this have worked?

  
  
Posted 3 years ago

so, the thing is that to install the object_detection package you need to manually run some commands, copy a setup.py file, etc. There is no git repo that does that for you...is that more clear?

  
  
Posted 3 years ago

so does the container install anything, or just runs the script? how is the setup done there?

  
  
Posted 3 years ago

I see, thanks!

  
  
Posted 3 years ago

so I need to run a sed command to replace some lines in one of the tensorflow files..do you know if I can do this as part of the setup.py install?

  
  
Posted 3 years ago

so if it lasts executes python setup.py install, I can do stuff like add a line to a file in my venv inside the setup script

  
  
Posted 3 years ago

it worked! thanks AgitatedDove14 !!

  
  
Posted 3 years ago

How would you like me to share it?
So far I have this:
tensorflow_object_detection_autoinstall.sh
Before running:
You need to set your venv
install numpy
export TF_DIR=$HOME/tensorflow mkdir $TF_DIR cd $TF_DIR echopwdwget unzip protoc-3.14.0-linux-x86_64.zip -d protoc export PATH=$PATH:pwd`/protoc/bin

git clone
cd models
git checkout 8a06433
cd $TF_DIR/models/research
protoc object_detection/protos/*.proto --python_out=.

git clone
cd cocoapi/PythonAPI
make
cp -r pycocotools/ $TF_DIR/models/research/

cd $TF_DIR/models/research
cp object_detection/packages/tf2/setup.py .

force tensorflow==2.2

sed -i "s/'tf-models-official'/'tf-models-official', 'tensorflow==2.2'/" setup.py
python -m pip install .
pip install numpy==1.20

finally fix bug in tensorflow array_ops file

file=python -c 'from tensorflow.python.ops import array_ops; print(array_ops.__file__)'
sed -i "s/np.prod/tf.math.reduce_prod/g" $file
sed -i 's/import numpy as np/import numpy as np; import tensorflow as tf/' $file
echo "Done" `

  
  
Posted 3 years ago

this creates the whl package that I then use in my requirements.txt file directly

  
  
Posted 3 years ago

oh it uses tf 1.15 😕

  
  
Posted 3 years ago

so does the container install anything,

The way the agent works with dockers:
spin the docker Install the base stuff inside the docker (like git and make sure it has python etc) Create a new venv inside the docker, inheriting everything from the docker's system wide python packages, this means that if you have the "object_detection" package installed, it will be available inside the new venv. Install specific python package your Task requires (inside the venv). This allows you to override/add missing packages on the base docker container Run the code

  
  
Posted 3 years ago

MagnificentSeaurchin79 You could also just fork the tensorflow repo, make changes in a specific branch and specify your forked repo with your custom branch in the install_requires of your setup.py

  
  
Posted 3 years ago

does that mean that it will install my package lastly?

It will install last, but not because it was last in the list, but because it is local/repo package 🙂

Can I do in setup.py the modifications to the tensorflow code?

You mean like have the changes as part of the "uncommitted changes" section ?

  
  
Posted 3 years ago

so I have a couple of questions regarding setup.py.
If I add the requirement '.' as the last entry, does that mean that it will install my package lastly? Can I do in setup.py the modifications to the tensorflow code?I need to see how I can change the tensoflow code after it was installed and prevent other tensorflow installation to overwrite it..is it clear?

  
  
Posted 3 years ago

great 🙂

  
  
Posted 3 years ago

And if you need a very small change, you can also simply https://www.geeksforgeeks.org/monkey-patching-in-python-dynamic-behavior/ it

  
  
Posted 3 years ago

and can the agent when running locally with no base docker, inherit as well system wide packages?

  
  
Posted 3 years ago

but then I run again the task, it uses the same cache venv, but fails to find object_detection

  
  
Posted 3 years ago

well you can always do:
os.system('sed ...')🙂

  
  
Posted 3 years ago

MagnificentSeaurchin79

Can this be solved by using a docker image with the preinstalled packages at a user level?

Yes 🙂
BTW: I think I missed how you managed to install the object_detection API in the first place?
Is it the git repo of the Task? did you fork it? is it a submodule of your git repo?

p.s.
Yes Slack is quite good at reminding you, but generally saying always prefer @ , it will send me an email if I miss the message :)

  
  
Posted 3 years ago

thanks MagnificentSeaurchin79 , yes that makes it clear.
If that is the case, I think building a container is the easiest solution 🙂
(BTW: You could also build a wheel, if you have setup.py then running is once bdist_wheel will build a wheel, and then install the wheel)

  
  
Posted 3 years ago

ah great! I'll try that then

  
  
Posted 3 years ago

MagnificentSeaurchin79 YEY!!!!
Very cool!

Do you feel like making it public, I have the feeling a lot of people will appreciate it, this is very useful 🙂

  
  
Posted 3 years ago

btw, do you see these messages AgitatedDove14 when they are inside an old thread? or should I start a new message?

  
  
Posted 3 years ago

yes, that would work, except that I need to modify tensorflow as well..I'm currently working on creating a wheel for modified tf..but it's taking a while...

  
  
Posted 3 years ago
648 Views
30 Answers
3 years ago
one year ago
Tags