Sure, is it reproducible ?

SuperiorDucks36 from code ? or UI?
(You can always clone an experiment and change the entire thing, the question is how will you get the data to fill in the experiment, i.e. repo / arguments / configuration etc)
There is a discussion here, I would love to hear another angle.
https://github.com/allegroai/trains/issues/230

https://stackoverflow.com/questions/37743683/why-is-an-empty-mongodb-database-so-big

With pleasure 🙂

pip cache & git cache & venvs cache
Are all supported, you just need to map the folders.
If you do not want to spin a PVC with NFS mount, you can just mount an S3 bucket with s3fs as part of the container extra bash script,
https://github.com/allegroai/clearml-agent/blob/b39b54bbafab39e6731cb742fdf317bc6dcae54a/docs/clearml.conf#L140

s3 FUSE fuse filesystems:
https://github.com/kahing/goofys
https://github.com/s3fs-fuse/s3fs-fuse

WDYT?

As a hack you can try DEFAULT_VERSION
(it's just a flag and should basically do Store)
EDIT: sorry that won't work 😞

Correct, (if this is running on k8s it is most likely be passed via env variables , CLEARML_WEB_HOST etc,)

Thanks OutrageousGrasshopper93
I will test it "!".
By the way the "!" is in the project or the Task name?

Yes , both work :(

Hi JuicyDog96
The easiest way is:
from trains.backend_api.session.client import APIClient client = APIClient() client.projects.get_all()You can just run it from a python console and check what you are getting.
Full API is https://github.com/allegroai/trains/tree/master/trains/backend_api/services/v2_8

Hmm I'm assuming something wrong here:
https://github.com/allegroai/clearml-server/blob/a64c4d264d00eadd2d11818b37151d3cc6266d99/docker/docker-compose.yml#L119
What's the host machine OS ?

This smells like a driver/image issue on the instance VM
What are you getting if add this inside your code?

os.system('nvidia-smi')

Hi @<1630377234361487360:profile|RoughSeaturtle43>

code from gitlab repo with ssl cert.

what do you mean by ssl secret? is it SSH or app-token ?

HealthyStarfish45
No, it should work 🙂

but actually that path doesn't exist and it is giving me an error

So you are saying you only uploaded the "meta-data" i.e. a text file with links to the files, and this is why it is missing?

Is there a way to change the path inside the .txt file to clearml cache, because my images are stored in clearml cache only

I think a good solution would be to store the path in the txt file as relative path, i.e. instead of /Users/adityachaudhry/data/folder... as ./data/folder

Hi TenseOstrich47
Does the .ssh folder on the user running the agent contain the correct credentials ?
Basically from the user running the agent on the agent's machine can you clone the repo with:
ssh://git@github.com/15gifts/py-db.git

PS. I just noticed that this function is not documented. I'll make sure it appears in the doc-string.

So in theory you can clone yourself 2 extra times and push into an execution queue, but the issue might be actually making sure the resources are available. what did you have in mind?

.I am using pipeline from tasks method and not pipeline from decorator.

Wait I'm confused nowm if this is a pipeline from Tasks then the Tasks themselves should have clearml in the "installed packages", no? and if they do not, how were they created?

what's the clearml package version and clearml-session version ?

Hi WickedGoat98 ,
I think you are correct 😞
I would guess it is something with the ingress configuration (i.e. ConfigMap)

Correct

Failed to initialize NVML: Unknown Error

yeah this is a driver issue. I think you need to check the VM image if the drivers match the GPU on that machine

Hi @<1557899668485050368:profile|FantasticSquid9>
There is some backwards compatibility issue with 1.2 (I think).
Basically what you need it to spin a new one on a new session ID and rergister the endpoints

MelancholyElk85

How do I add files without uploading them anywhere?

The files themselves need to be packaged into a zip file (so we have an immutable copy of the dataset). This means you cannot "register" existing files (in your example, files on your S3 bucket?!). The idea is to make sure your dataset is protected against changes on the one hand, but on the other to allow you to change it, and only store the changeset.
Does that make sense ?

1633204289496 clearml-services DEBUG docker: invalid reference format.

This is the strange message, like the execution command is not valid...

but it fails during env setup due to trying to install an obscure version of pytorch. Been trying to solve this for three days!

AdventurousButterfly15 it tries to resolve the correct pytorch version based on the cuda inisde the container

ERROR: torch-1.12.1+cu116-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.

seems like it is trying to install pytoch for python 3.10 with cuda 11.6 support, this seems reasonable, no?

MysteriousBee56 that is so weird ... last one, I promise 🙂
docker run -t --rm nvidia/cuda:10.1-base-ubuntu18.04 bash -c "echo 'Binary::apt::APT::Keep-Downloaded-Packages \"true\";' > /etc/apt/apt.conf.d/docker-clean && apt-get update && apt-get install -y git python3-pip && python3 -m pip install trains-agent && echo \$(which python3) && echo \$(which trains-agent)"

No worries 🙂 glad it worked

Hi ScaryLeopard77
I think the error message you are getting is actually "passed" from Triton. Basically someone needs to tell it what the Model in/out look like (matrix size/type) this is essentially the content of the "config.pbtxt" , and this has to be set when spinning the model endpoint. does that make sense to you?