Hi @<1574931891478335488:profile|DizzyButterfly4> , I think if you have a pandas object pd then the usage would be something like ds.set_metadata(metadata=pd, metadata_name="my pandas object")

I think you would be referencing the entire thing using the metadata_name parameter

Hi SpotlessPenguin79 , can you please elaborate on this?

for non-aws cloud providers?

What exactly are you trying to do?

Hi @<1789465500154073088:profile|ScaryShrimp8> , can you provide the full log of the run? How did you set up & run the agent? What version and what OS are you on?

Hi TrickyFox41 , are you getting some sort of error?

Hi @<1752501940488507392:profile|SquareMoth4> , you have to bring your own compute. ClearML only acts as a control plane allowing you to manage your compute. Why not use AWS for example as a simple solution?

@<1523701181375844352:profile|ExasperatedCrocodile76> , I think you need to set agent.package_manager.system_site_packages: True
In clearml.conf

ClearML doesn't assume that you have all the necessary packages installed so it does need to have some sort of basis for the packages to install

So If I manually add a dataset (many excels), in a folder, and copy that folder to NFC

How would you do that?

Try running the following script

from clearml import Task 
import time

task = Task.init(output_uri="

")

print("start sleep")
time.sleep(20)
print("end sleep")

Please add the logs

UpsetBlackbird87 , I couldn't reproduce the issue on my end. Can you please send me a self contained example code to try and recreate?

Hi @<1523701875835146240:profile|SkinnyPanda43> , with what email are you trying to login?

That sounds like a fairly large team already. I would suggest considering the Scale version. It would alleviate a lot of devops work & maintenance on your part, provide direct support for users & admins, RBAC, SSO, configuration vaults and many other features.

Hi @<1523701295830011904:profile|CluelessFlamingo93> , if you self host then you're no longer limited on usage. However the downside is of course that you have to manage it yourself (Security, backups etc).

Hi @<1533257278776414208:profile|SuperiorCockroach75> , what do you mean? ClearML logs automatically scikit learn

Hi @<1792726992181792768:profile|CloudyWalrus66> , can you provide the full log of the ec2 instance?

The setting for the python binary should be explicit since the agent can't really 'detect' where you installed your python
For example:
agent.python_binary: "C:\ProgramData\Anaconda3\python.exe"

Hi @<1523701868901961728:profile|ReassuredTiger98> , you can fetch the task object, there one of the attributes of the task is it's worker. This way you can see on what machine it is running 🙂

Hi @<1571308079511769088:profile|GentleParrot65> , ideally you shouldn't be terminating instances manually. However you mean that the autoscaler spins down a machine and still recognizes it as running and refuses to spin up a new machine?

Hi @<1719524650926477312:profile|EncouragingFish95> , I think what you're looking for is the AWS autoscaler - None

None

Hi @<1582179661935284224:profile|AbruptJellyfish92> , connectivity issues should not affect training and should cache everything until connection is restored and everything should be sent to the server. Did you encounter a different behavior?

Also, what happens if you apss it in agent.default_docker.arguments ?

It should be in top level, not environment or agent

Does it go back to working if you revert the changes?

Hi @<1727859576625172480:profile|DeliciousArcticwolf54> , I'd suggest debugging using developer tools in the webUI. Also, are you seeing any errors in the API server or webserver containers? I'd suggest first testing with elasticsearch to make sure that the deployment went through OK and this is not related to something else.

Out of curiosity, why do you want to use opensearch instead of elasticsearch?

Hi GrittyHawk31 , can you elaborate on what you mean by metadata? Regarding models you can achieve this by defining the following in Task.init(output_uri="<S3_BUCKET>")

Hi @<1523704207914307584:profile|ObedientToad56> , the virtual env is constructed using the detected packages when run locally. You can certainly override that. For example use Task.add_requirements - None

There are also a few additional configurations in the agent section of clearml.conf I would suggest going over

Pipelines have id's, you can try using a pipeline ID. I think it should work

@<1524560082761682944:profile|MammothParrot39> , I'm not sure what you mean, if it's in draft, why do you expect it to run?

Hi ObedientToad56 🙂

My question is on how the deployment would be once we have verified the endpoints are working in a local container.

I isn't the deployment just running the inference container? You just open up the endpoints towards where you wanna server, no?

I'm curious as to why you weren't redirected to login page

I think maybe you're right. Let me double check. I might be confusing it with the previous version