Hi! I'm tryin to find a workaround for this: can't do pip install <name_of_package>
I executed the task, and it created a cache venv.
But when running the code, it couldn't import the package because it wasn't listed.
I then sourced the venv, and manually installed the package
so if I do python -c 'import object_detection' if works

yes, that would work, except that I need to modify tensorflow as well..I'm currently working on creating a wheel for modified tf..but it's taking a while...

ah great! I'll try that then

great 🙂

MagnificentSeaurchin79 You could also just fork the tensorflow repo, make changes in a specific branch and specify your forked repo with your custom branch in the install_requires of your setup.py

And if you need a very small change, you can also simply https://www.geeksforgeeks.org/monkey-patching-in-python-dynamic-behavior/ it

but then I run again the task, it uses the same cache venv, but fails to find object_detection

oh it uses tf 1.15 😕

so if it lasts executes python setup.py install, I can do stuff like add a line to a file in my venv inside the setup script

I see, thanks!

so I need to run a sed command to replace some lines in one of the tensorflow files..do you know if I can do this as part of the setup.py install?

I didn't know about that one! I'll try it, thanks!!

so, the thing is that to install the object_detection package you need to manually run some commands, copy a setup.py file, etc. There is no git repo that does that for you...is that more clear?

well you can always do:
os.system('sed ...')🙂

should this have worked?

this creates the whl package that I then use in my requirements.txt file directly

MagnificentSeaurchin79 YEY!!!!
Very cool!

Do you feel like making it public, I have the feeling a lot of people will appreciate it, this is very useful 🙂

MagnificentSeaurchin79

Can this be solved by using a docker image with the preinstalled packages at a user level?

Yes 🙂
BTW: I think I missed how you managed to install the object_detection API in the first place?
Is it the git repo of the Task? did you fork it? is it a submodule of your git repo?

p.s.
Yes Slack is quite good at reminding you, but generally saying always prefer @ , it will send me an email if I miss the message :)

thanks MagnificentSeaurchin79 , yes that makes it clear.
If that is the case, I think building a container is the easiest solution 🙂
(BTW: You could also build a wheel, if you have setup.py then running is once bdist_wheel will build a wheel, and then install the wheel)

and can the agent when running locally with no base docker, inherit as well system wide packages?

You might need to update it to the latest Detection API 😞

so I have a couple of questions regarding setup.py.
If I add the requirement '.' as the last entry, does that mean that it will install my package lastly? Can I do in setup.py the modifications to the tensorflow code?I need to see how I can change the tensoflow code after it was installed and prevent other tensorflow installation to overwrite it..is it clear?

it worked! thanks AgitatedDove14 !!

How would you like me to share it?
So far I have this:
tensorflow_object_detection_autoinstall.sh
Before running:
You need to set your venv
install numpy
export TF_DIR=$HOME/tensorflow mkdir $TF_DIR cd $TF_DIR echopwdwget unzip protoc-3.14.0-linux-x86_64.zip -d protoc export PATH=$PATH:pwd`/protoc/bin

git clone
cd models
git checkout 8a06433
cd $TF_DIR/models/research
protoc object_detection/protos/*.proto --python_out=.

git clone
cd cocoapi/PythonAPI
make
cp -r pycocotools/ $TF_DIR/models/research/

cd $TF_DIR/models/research
cp object_detection/packages/tf2/setup.py .

force tensorflow==2.2

sed -i "s/'tf-models-official'/'tf-models-official', 'tensorflow==2.2'/" setup.py
python -m pip install .
pip install numpy==1.20

finally fix bug in tensorflow array_ops file

file=python -c 'from tensorflow.python.ops import array_ops; print(array_ops.__file__)'
sed -i "s/np.prod/tf.math.reduce_prod/g" $file
sed -i 's/import numpy as np/import numpy as np; import tensorflow as tf/' $file
echo "Done" `

btw, do you see these messages AgitatedDove14 when they are inside an old thread? or should I start a new message?

Yes, just set system_site_packages: true in your clearml.conf
https://github.com/allegroai/clearml-agent/blob/d9b9b4984bb8a83914d0ec6d53c86c68bb847ef8/docs/clearml.conf#L57

so does the container install anything,

The way the agent works with dockers:
spin the docker Install the base stuff inside the docker (like git and make sure it has python etc) Create a new venv inside the docker, inheriting everything from the docker's system wide python packages, this means that if you have the "object_detection" package installed, it will be available inside the new venv. Install specific python package your Task requires (inside the venv). This allows you to override/add missing packages on the base docker container Run the code

so does the container install anything, or just runs the script? how is the setup done there?

does that mean that it will install my package lastly?

It will install last, but not because it was last in the list, but because it is local/repo package 🙂

Can I do in setup.py the modifications to the tensorflow code?

You mean like have the changes as part of the "uncommitted changes" section ?

can this be solved by using a docker image with the preinstalled packages at a user level?