If the only issue is this linetask.execute_remotely(..., exit_process=True)It has to finish the static analysis of the entire repository (which usually happens in the background but now we have to wait for it). If the repo is large this could actually take 20sec (depending on CPU/drive of the machine itself)
Hi ResponsiveCamel97
Let me explain how it works, essentially it creates a new venv inside the docker, inheriting all the packages form the main system packages.
This allows it to use the installed packages if the version match, and upgrade/change if you need, all without the need to rebuild a new container. Make sense ?
How so? they are in one place? the creation of the venv is transparent, and the packages that are there are everything you have in the docker, plus the ability to override them from the UI.
What am I missing here ?
sdk.storage.cache.size.cleanup_margin_percent
Hi ReassuredTiger98
This is actually future proofing the cache mechanism and allowing it be "smarter" i.e. clean based on cache folder size instead of cache folder entries, this is currently not available
sdk.storage.cache
parameters for the agent?
For both local execution and with an agent
When are datasets deleted if I run local execution?
When you hit the cache entry limit (100 if I recall). This can a...
Can you share the storagemanager usage, and error you are getting ?
a task of queue B if the next task is of type A it will have to wait,
It seems you imply there are two types of Tasks and they need to be executed one after the other ?
BTW: there is still the bug with the env merging, correct ?
well, it's only when adding a
- name
to the template
Nonetheless it should not break it 🙂
If it cannot find the Task ID I'm guessing it is trying to connect to the demo server and not your server (i.e. configuration is missing)
Hi BattyLion34
The windows issue seems like it is coming from missing QT installed on the Host machine
Check the pyqt5 version in your "Installed packages"
see here:
https://superuser.com/questions/1433913/qtpy-pythonqterror-no-qt-bindings-could-be-found
Regrading the linux, it seems your are missing the object_detection package, where do you usually install it from ?
FrothyShark37 what was different in your script ?
RoundMosquito25 good news, no no need to open any ports 🙂
Basically B_i agents are always polling the server for "jobs" create an http/s request from them to the server, so all connections are out connections. Firewall is intact 🙂
GreasyPenguin66 Nice !!!
Very cool setup, and kudos on making it work with multiple users!
Quick question, shouldn't the JUPYTERHUB_API_TOKEN env variable be enough to gain access to the server? Why did you need to add it to the 'nbserver-x.json' as well?
Could it be it was never allocated to begin with ?
(I mean new logs, while we are here did it report any progress)
Oh dear, I think your theory might be correct, and this is just the mongo preallocating storage.
Which means the entire /opt/trains just disappeared
No worries, and I hope you manage to get that backup.
I want to download an exact folder/batch of the dataset to my local machine to check data out without downloading whole dataset.
TeenyBeetle18 the closest you can get is to download only one part of the dataset, if this is a multi part dataset (i.e. the dataset version is larger than the default 500MB, so you have multiple izp files, and you just want to download one of them, not all of them).
This can actually be achieved with:Dataset.get_local_copy(..., part=0)
https://githu...
ShinyWhale52 any time 🙂
Feel free to followup with more questions
If you are using user/pass for the git (i.e. not ssh key) we are not passing it to the pip install (and come to think about it, we probably should?!)
unless the domain is different
?
Imagine that you are working with both github and bitbucket for example, if you are using git-ssh than git will know which of the domains to send the key to. Currently there is a single user/pass entry so all domains will get the same credentials. But I think this is a rare use case.
how did you try to restart them ?
Yes, but how did you restart the agent on the remote machine ?