AgitatedDove14

49 Questions, 8126 Answers

Active since 10 January 2023

Last activity one year ago

Reputation

Badges 1

25 × Eureka!

Answers 8126

0 Hi! I Am Getting The Following Error On An Agent:

Hi GrievingTurkey78
Can you test with the latest clearml-agent RC (I remember a fix just for that)
pip install clearml-agent==1.2.0rc0

3 years ago

0 Is It Possible To Give The Agent Access To Install Private Pip Packages (Needs To Be Installed From The Repo)?

Can you copy the "Installed Packages" here, and point to the package causing the issue?

4 years ago

0 Trying To Setup A Trains-Agent Worker On A Remote Machine; When I Run Trains-Init And Follow The Steps To Give It Credentials For Our Trains Server I Get This

http vs https ?

4 years ago

0 Hi Guys! Love Using Trains And Love The Great Support In This Channel. Say I Have Two Different Training Experiments Which Report Every 20 Iteration, But The Batch Size Between Them Is Different, Resulting In Different Number Of Iterations Per Epoch. I Wo

ShallowCat10 Thank you for the kind words 🙂

so I'll be able to compare the two experiments over time. Is this possible?

You mean like match the loss based on "images seen" ?

5 years ago

0 When I Try To Enqueue An Experiment To A Queue, It Looks Like Trains-Agent Is Recreating A New Virtual Env Everytime. So, This Causes A Problem In Case Trains Missed A Package And Aborts The Run. Even If I Install This Package Manually, It Gets Removed.

Hi FriendlyKoala70 you can edit the installed package section and add the missing package. See more details on how trains-agent works here (although it's on conda the same rules apply for pip) https://github.com/allegroai/trains-agent/issues/8

5 years ago

0 Hello Periodically Under High Load, We Are Facing Too Long(>1 Sec) Processing Times For Requests Such As: Workers.Status_Report Events.Add_Batch Queues.Get_Next_Task. Also There Are Warnings "Connection Pool Is Full, Discarding Connection: Elasticsearch-S

Hi ItchyJellyfish73
This seems aligned with scenario you are describing, it seems the api server is overloaded with simultaneous connections.
Add an additional apiserver instance to the docker-compose and an nginx as load balancer:
https://github.com/allegroai/clearml-server/blob/09ab2af34cbf9a38f317e15d17454a2eb4c7efd0/docker/docker-compose.yml#L4
`
apiserver:
command:
- apiserver
container_name: clearml-apiserver
image: allegroai/clearml:latest
restart: unless-sto...

4 years ago

0 Hi, While Running My Experiments I Get This Message : "Clearml Monitor: Could Not Detect Iteration Reporting, Falling Back To Iterations As Seconds-From-Start" - I Believe It Happens Due To The Fact I Have A Heavy Calculation During The Run And I Assume C

assume clearml has some period of time that after it, shows this message. am I right?

Yes you are 🙂

is this configurable?

It is 🙂
task.set_resource_monitor_iteration_timeout(seconds_from_start=1800)

4 years ago

0 <image>

What's the OS (Windows/Max/Linux)? What's the chrome version ?

4 years ago

0 Hello, I Have Two Questions About Taskscheduler.

Hi ScaryBluewhale66

TaskScheduler I created. The status is still

running

. Any idea?

The TaskScheduler needs to actually run in order to trigger the jobs (think cron daemon)
Usually it will be executed on the clearml-agent services queue/mahine.
Make sense ?

3 years ago

0 Hey Guys. We Have Been Using Clearml For A While Now And It Has Solved Quite Some Headaches Around Our Operations. We Are Self Hosting It Using Docker Swarm And Were Wondering If This Is Something That The Community Would Be Interested In. This Would Be

that really depends on hoe much data you have there, and the setup. The upside of the file server is you do not need to worry about credentials, the downside is storage is more expensive

2 years ago

0 Hello! I'M Running Clearml-Server On Kubernetes, And It Seems My Models Are Not Really Saved. I See That Doing Task.Init(Output_Uri=True) Should Send Models To Fileserver. The Models Are Visible In The Ui But The Download Button Is Greyed Out And When I D

...I'm not sure I follow, the clearml-task is designed to always be used so that at the end the agent will be running the Task. What am I missing?

3 years ago

0 Hi, I Have Been Getting The Following For A While. Is There A More Detailed Log I Can Look Into? This Happens On Both Https And Http.

In our case this is not possible due to client security (e.g. training data from clients can potentially be 'reverse engineered' from trained models in future).

Hmm I see, wouldn't it make more sense to separate clients like a multi-tenant SAAS solution ?

4 years ago

0 Hello! I Get The Following Error In Results->Console After A Task Is Sent For Remote Execution (Using Sdk):

Yey?

3 years ago

0 Hi All, How Do I Add Additional Data To An Experiment After The Program Is Done? E.G. I Want To Add Some More Charts To The Results Tab For An Experiment That Have Some Report Results Generated By A Separate Process

Now I need to figure out how to export that task id

You can always look it up 🙂
How come you do not have it?

5 years ago

0 Another Question: How Can I Make Clearml-Agent Use Pre-Installed Version From The Nvidia/Pytorch (

ReassuredTiger98 yes this is odd:
also:
Warning, could not locate PyTorch torch==1.12 matching CUDA version 115, best candidate 1.12.0.dev20220407Seems like it found a matching version and did not use it...
Let me check that

3 years ago

0 Am I Doing Something Wrong Or Is Should I Open An Issue About It (Bug?)? I'M Using The

Hi WackyRabbit7
I believe this is fixed in clearml-server 1.1 (this is a plotly color issue), releasing later today or tomorrow 🙂

4 years ago

0 Hello Everyone, I’M Newcomer For Clearml. I Have Question Related To

https://github.com/googleapis/python-storage/issues/74

4 years ago

0 Question: Has Anyone Done Anything With Ray Or Rllib, And Clearml? Would Clearml Be Able To Integrate With Those Out Of The Box?

save off the "best" model instead of the last

Should be relatively easy to update on the main Task the model with the best performance, no?

4 years ago

0 Hey All, Hope You’Re All Doing Well. I’M Running A Self-Deployed Server (0.17, I Think, Where Can You Find The Version In Use?). I’M Having Trouble With The Automatic Plot Capture. If I Run

Hi NastyFox63 could you verify the fix works?
pip install git+

4 years ago

0 Hi! I'M Using Func

. Could you clarify the question for me, please?
...
Could you please point me to the piece of ClearML code related to the downloading process?

I think I mean this part:
https://github.com/allegroai/clearml/blob/e3547cd89770c6d73f92d9a05696018957c3fd62/clearml/datasets/dataset.py#L2134

3 years ago

0 Hi All! I’M Trying To Set Up Remote-Launching Of Training Scripts On Clearml Autoscaler, And I Can’T Figure Out One Thing: How To Make Remote Clearml Agent Do

Hi @<1716987933514272768:profile|SuccessfulPuppy43>

How to make remote ClearML agent do

pip install -e .

in theory there is no need to do that clearml-agent adds the repo root folder to the python path.
If you insist on actually installing it, try to add to your "installed packages" section a "requirement.txt" compatible line:

-e .

one year ago

0 Hello Everyone, Is There Any Way To Remove A Serving Instance?

Hi @<1657918706052763648:profile|SillyRobin38>
You mean remove the entire serving session? is it still running somewhere ?
(for example if you take the docker-compose down it will be marked aborted automatically after 2 hours)

one year ago

0 So I'M In A Colab Notebook, And After Running My Trainer(), How Do I Upload My Test Metrics To Clearml? Clearml Caught These Metrics And Uploaded Them:

can you see these metric on TB ?

4 years ago

0 Is It Possible To Filter Tasks By There Output And Input Names Using .Get_Tasks?

Hmm SuccessfulKoala55 what do you think?

4 years ago

0 Having Issues Running Trains-Server On Win10. Trains-Elastic Exited With Code 137 Trains-Mongo Exited With Code 100 Trains-Apiserver Exited With Code 1 Some Errors=> Requests.Exceptions.Connectionerror: Httpconnectionpool(Host='Elasticsearch', Port=9200

LazyLeopard18 nice. maybe we should add it in the FAQ / Install. Could you send the exact docker-compose you used and command line, I'll ask the guys to add it 🙂

5 years ago

0 Hi There, I'Ve Encountered A Problematic Behavior In Python. When Defining An Argument A Default Value Of

Another question, do you have the argparse with type=str ?

5 years ago

0 Hi! I’M Running An Experiment As Follows:

Yes, actually ensuring pip is there cannot be skipped (I think in the past it cased to many issues, hence the version limit etc.)
Are you saying it takes a lot of time when running? How long is the actual process that the Task is running (just to normalize times here)

3 years ago

0 Hi! I’M Running An Experiment As Follows:

So are you saying why do we need to install a specific pip version ?
You can "disable it" by selecting a very high version
pip_version: "<40"https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L67

3 years ago

0 Hi, I'M Having A Hard Time Trying To Understand The Dataset Class. What I Need Is To Be Able To Get The Dataset, Delete A File, And Upload It Again. But The Problem Is When I Call The

NICE! MagnificentSeaurchin79 could you PR this fix?

4 years ago

0 During Our First Large Hyperpameter Run, We Have Noticed That There Are Some Tasks That Get Aborted With The Following Console Log:

Hi @<1541592204353474560:profile|GhastlySeaurchin98>

During our first large hyperpameter run, we have noticed that there are some tasks that get aborted with the following console log:

This looks like the HPO algorithm doing early stopping, which algo are you using ?

2 years ago

Show more results