oh it uses tf 1.15 😕

I see, thanks!

Hi! I'm tryin to find a workaround for this: can't do pip install <name_of_package>
I executed the task, and it created a cache venv.
But when running the code, it couldn't import the package because it wasn't listed.
I then sourced the venv, and manually installed the package
so if I do python -c 'import object_detection' if works

it worked! thanks AgitatedDove14 !!

so does the container install anything, or just runs the script? how is the setup done there?

so I have a couple of questions regarding setup.py.
If I add the requirement '.' as the last entry, does that mean that it will install my package lastly? Can I do in setup.py the modifications to the tensorflow code?I need to see how I can change the tensoflow code after it was installed and prevent other tensorflow installation to overwrite it..is it clear?

so I need to run a sed command to replace some lines in one of the tensorflow files..do you know if I can do this as part of the setup.py install?

well you can always do:
os.system('sed ...')🙂

MagnificentSeaurchin79

Can this be solved by using a docker image with the preinstalled packages at a user level?

Yes 🙂
BTW: I think I missed how you managed to install the object_detection API in the first place?
Is it the git repo of the Task? did you fork it? is it a submodule of your git repo?

p.s.
Yes Slack is quite good at reminding you, but generally saying always prefer @ , it will send me an email if I miss the message :)

does that mean that it will install my package lastly?

It will install last, but not because it was last in the list, but because it is local/repo package 🙂

Can I do in setup.py the modifications to the tensorflow code?

You mean like have the changes as part of the "uncommitted changes" section ?

so does the container install anything,

The way the agent works with dockers:
spin the docker Install the base stuff inside the docker (like git and make sure it has python etc) Create a new venv inside the docker, inheriting everything from the docker's system wide python packages, this means that if you have the "object_detection" package installed, it will be available inside the new venv. Install specific python package your Task requires (inside the venv). This allows you to override/add missing packages on the base docker container Run the code

should this have worked?

but then I run again the task, it uses the same cache venv, but fails to find object_detection

And if you need a very small change, you can also simply https://www.geeksforgeeks.org/monkey-patching-in-python-dynamic-behavior/ it

How would you like me to share it?
So far I have this:
tensorflow_object_detection_autoinstall.sh
Before running:
You need to set your venv
install numpy
export TF_DIR=$HOME/tensorflow mkdir $TF_DIR cd $TF_DIR echopwdwget unzip protoc-3.14.0-linux-x86_64.zip -d protoc export PATH=$PATH:pwd`/protoc/bin

git clone
cd models
git checkout 8a06433
cd $TF_DIR/models/research
protoc object_detection/protos/*.proto --python_out=.

git clone
cd cocoapi/PythonAPI
make
cp -r pycocotools/ $TF_DIR/models/research/

cd $TF_DIR/models/research
cp object_detection/packages/tf2/setup.py .

force tensorflow==2.2

sed -i "s/'tf-models-official'/'tf-models-official', 'tensorflow==2.2'/" setup.py
python -m pip install .
pip install numpy==1.20

finally fix bug in tensorflow array_ops file

file=python -c 'from tensorflow.python.ops import array_ops; print(array_ops.__file__)'
sed -i "s/np.prod/tf.math.reduce_prod/g" $file
sed -i 's/import numpy as np/import numpy as np; import tensorflow as tf/' $file
echo "Done" `

great 🙂

ah great! I'll try that then

can this be solved by using a docker image with the preinstalled packages at a user level?

btw, do you see these messages AgitatedDove14 when they are inside an old thread? or should I start a new message?

MagnificentSeaurchin79 YEY!!!!
Very cool!

Do you feel like making it public, I have the feeling a lot of people will appreciate it, this is very useful 🙂

so if it lasts executes python setup.py install, I can do stuff like add a line to a file in my venv inside the setup script

MagnificentSeaurchin79 You could also just fork the tensorflow repo, make changes in a specific branch and specify your forked repo with your custom branch in the install_requires of your setup.py

so, the thing is that to install the object_detection package you need to manually run some commands, copy a setup.py file, etc. There is no git repo that does that for you...is that more clear?

thanks MagnificentSeaurchin79 , yes that makes it clear.
If that is the case, I think building a container is the easiest solution 🙂
(BTW: You could also build a wheel, if you have setup.py then running is once bdist_wheel will build a wheel, and then install the wheel)

You might need to update it to the latest Detection API 😞

I didn't know about that one! I'll try it, thanks!!

yes, that would work, except that I need to modify tensorflow as well..I'm currently working on creating a wheel for modified tf..but it's taking a while...

this creates the whl package that I then use in my requirements.txt file directly

Yes, just set system_site_packages: true in your clearml.conf
https://github.com/allegroai/clearml-agent/blob/d9b9b4984bb8a83914d0ec6d53c86c68bb847ef8/docs/clearml.conf#L57

and can the agent when running locally with no base docker, inherit as well system wide packages?