MagnificentSeaurchin79
Can this be solved by using a docker image with the preinstalled packages at a user level?
Yes 🙂
BTW: I think I missed how you managed to install the object_detection API in the first place?
Is it the git repo of the Task? did you fork it? is it a submodule of your git repo?
p.s.
Yes Slack is quite good at reminding you, but generally saying always prefer @ , it will send me an email if I miss the message :)
thanks MagnificentSeaurchin79 , yes that makes it clear.
If that is the case, I think building a container is the easiest solution 🙂
(BTW: You could also build a wheel, if you have setup.py then running is once bdist_wheel will build a wheel, and then install the wheel)
and can the agent when running locally with no base docker, inherit as well system wide packages?
You might need to update it to the latest Detection API 😞
Yes, just set system_site_packages: true
in your clearml.conf
https://github.com/allegroai/clearml-agent/blob/d9b9b4984bb8a83914d0ec6d53c86c68bb847ef8/docs/clearml.conf#L57
so does the container install anything,
The way the agent works with dockers:
spin the docker Install the base stuff inside the docker (like git and make sure it has python etc) Create a new venv inside the docker, inheriting everything from the docker's system wide python packages, this means that if you have the "object_detection" package installed, it will be available inside the new venv. Install specific python package your Task requires (inside the venv). This allows you to override/add missing packages on the base docker container Run the code
so does the container install anything, or just runs the script? how is the setup done there?
does that mean that it will install my package lastly?
It will install last, but not because it was last in the list, but because it is local/repo package 🙂
Can I do in setup.py the modifications to the tensorflow code?
You mean like have the changes as part of the "uncommitted changes" section ?
can this be solved by using a docker image with the preinstalled packages at a user level?
this creates the whl package that I then use in my requirements.txt file directly
MagnificentSeaurchin79 YEY!!!!
Very cool!
Do you feel like making it public, I have the feeling a lot of people will appreciate it, this is very useful 🙂
yes, that would work, except that I need to modify tensorflow as well..I'm currently working on creating a wheel for modified tf..but it's taking a while...
And if you need a very small change, you can also simply https://www.geeksforgeeks.org/monkey-patching-in-python-dynamic-behavior/ it
MagnificentSeaurchin79 You could also just fork the tensorflow repo, make changes in a specific branch and specify your forked repo with your custom branch in the install_requires of your setup.py
Hi! I'm tryin to find a workaround for this: can't do pip install <name_of_package>
I executed the task, and it created a cache venv.
But when running the code, it couldn't import the package because it wasn't listed.
I then sourced the venv, and manually installed the package
so if I do python -c 'import object_detection' if works
so if it lasts executes python setup.py install, I can do stuff like add a line to a file in my venv inside the setup script
so I need to run a sed command to replace some lines in one of the tensorflow files..do you know if I can do this as part of the setup.py install?
I didn't know about that one! I'll try it, thanks!!
so, the thing is that to install the object_detection package you need to manually run some commands, copy a setup.py file, etc. There is no git repo that does that for you...is that more clear?
well you can always do:os.system('sed ...')
🙂
but then I run again the task, it uses the same cache venv, but fails to find object_detection
so I have a couple of questions regarding setup.py.
If I add the requirement '.' as the last entry, does that mean that it will install my package lastly? Can I do in setup.py the modifications to the tensorflow code?I need to see how I can change the tensoflow code after it was installed and prevent other tensorflow installation to overwrite it..is it clear?
it worked! thanks AgitatedDove14 !!
How would you like me to share it?
So far I have this:
tensorflow_object_detection_autoinstall.sh
Before running:
You need to set your venv
install numpyexport TF_DIR=$HOME/tensorflow mkdir $TF_DIR cd $TF_DIR echo
pwdwget
unzip protoc-3.14.0-linux-x86_64.zip -d protoc export PATH=$PATH:
pwd`/protoc/bin
git clone
cd models
git checkout 8a06433
cd $TF_DIR/models/research
protoc object_detection/protos/*.proto --python_out=.
git clone
cd cocoapi/PythonAPI
make
cp -r pycocotools/ $TF_DIR/models/research/
cd $TF_DIR/models/research
cp object_detection/packages/tf2/setup.py .
force tensorflow==2.2
sed -i "s/'tf-models-official'/'tf-models-official', 'tensorflow==2.2'/" setup.py
python -m pip install .
pip install numpy==1.20
finally fix bug in tensorflow array_ops file
file=python -c 'from tensorflow.python.ops import array_ops; print(array_ops.__file__)'
sed -i "s/np.prod/tf.math.reduce_prod/g" $file
sed -i 's/import numpy as np/import numpy as np; import tensorflow as tf/' $file
echo "Done" `
btw, do you see these messages AgitatedDove14 when they are inside an old thread? or should I start a new message?