AgitatedDove14

48 Questions, 8049 Answers

Active since 10 January 2023

Last activity 6 months ago

Reputation

Badges 1

25 × Eureka!

Answers 8049

0 I’M Wondering If Someone Has An Example Of How To Use The

Hi @<1533620191232004096:profile|NuttyLobster9>
base_task_factory is a function that gets the node definition and returns a Task to be enqueued ,
pseudo code looks like:

def my_node_task_factory(node: PipelineController.Node) -> Task:
  task = Task.create(...)
  return task

Make sense ?

one year ago

0 Hi, What'S The Expected Flow When Running An Agent In Docker Mode, And My Git Repo Requires Ssh Keys For Cloning? E.G.

Hi GracefulDog98
The agent will map the ~/.ssh folder automatically into the docker's /root/.ssh
It will also convert http links to ssh pull if you set force_git_ssh_protocol
in your clearml.conf :
https://github.com/allegroai/clearml-agent/blob/351f0657c3dcf707659875d7e0a52fa387709978/docs/clearml.conf#L25

3 years ago

0 Hi, I’M Getting This Error When I Try To Run Task On A Remote Agent With Docker Mode Web Ui:

EnviousPanda91
in your clearml.conf I think you are missing a section
agent.git_user="" agent.git_pass="" agent.git_host="" agent.force_git_ssh_protocol: true

2 years ago

0 Hi! Maybe Someone Will Be Able To Help Me. Im Trying To Build System: Step 0: Build Pipeline (Run Locally) Step 1: Executes The Taskscheduler, Step 2: Executes The Pipeline (Remotely) I Tried To Use Agent Services As On Image. But It Doesnt Work ( I’M Ab

You can see in the log it tries to download an artifact from a specific IP:URL is that link a valid one?
(this seems like the main cause of the error, first line in the screenshot)

3 months ago

0 When Using Docker Mode (And Specifically K8S Glue), What Are The Options For Caching? One Option Is Definitely Having A Base Image That Has The Things Needed. Anything Else? Thanks!

A few examples:
https://medium.com/@Sushil_Kumar/readwritemany-persistent-volumes-in-google-kubernetes-engine-a0b93e203180
https://docs.openshift.com/enterprise/3.1/install_config/storage_examples/shared_storage.html

3 years ago

0 I Have A Self-Hosted Clearm-Server And And Clearml-Agent Started With

Hi ReassuredTiger98
Could you add some print ? before / after the artifact upload?
Also what's the clearml version you are using ?

3 years ago

0 What Sort Of Integration Is Possible With Clearml And Sagemaker? On The Page

Yep I think you are correct, you should have had the same output as a local jupyter notebook, and it seems that in sagemaker studio it is not working 😞
Let me check something

one year ago

0 Hey There! I’M Having A Problem With Clearml-Sessions, Maybe Someone Had A Similar Problem Already: I’M Running An Agent In Docker Mode On A Remote Machine. When I Run

BitingKangaroo95 can you post here the entire console output of clearml-session (including full command line) ?

2 years ago

0 Hey, Is There A Shortcut On The Dataset Sdk To Directly Get The Latest Version Of A Dataset ?

currently I'm doing it by fetching the latest dataset, incrementing the version and creating a new dataset version

This seems like a very good approach, how would you improve ?

one year ago

0 Hi, Is There Any Way To Get Experiment Debug Images Programmatically?

HandsomeCrow5
client.events.debug_images(metrics=[dict(task='6adb929f66d14731bc76e3493ab89d80', metric='image')])

4 years ago

0 I Have Set

"regular" worker will run one job at a time, services worker will spin multiple tasks at the same time But their setup (i.e. before running the actual task) is one at a time..

4 months ago

0 Hi, When I Save Model Using Tf.Keras.Save_Model Or Using Modelcheckpoint Model Is Not Saved As An Artifact. Output Uri Is Set To Google Cloud Bucket. When Reporting With Logger Everything Is Stored Correctly. Do You Maybe Have Any Idea Why This Would Not

OutrageousGiraffe8 so basically replacing to:
self.d1 = ReLU()

2 years ago

0 Hi, I Am Trying To Upload A Plot To An Existing Task Using The

What's the trains version / trains-server version ?

3 years ago

0 Hi, I Would Like To Pass In Some Pip Arguments That Clearml-Agent Would Include When Setting Up The Venv On The Containers. How Should I Specify This? The Argument In Question Are --Trusted-Host And --Find-Links . I Need Them As I'Ve Installed A Pypi Repo

See here:
https://pip.pypa.io/en/stable/user_guide/#environment-variables

Pass these environment variables as part of the YAML template you are using with the k8s.
Should work for both 🙂

3 years ago

0 Do I Understand Correctly, That Running

Task.get_task(..).system_tags

3 years ago

0 Hi, I Want To Run A Script Remotely On My Agent, But For It To Work I Need It To Download To The Agent The Whole Directory The Script Is In, Is It Possible?

🤞

3 years ago

0 Hi Everyone, I'M Running Into A Weird Error When Trying To Clone And Run And Task That Has Completed Successfully. I Have A Test Task That Loads A Dummy Dataset And Trains A Toy Model With Pytorch. When Running Remotely, I Use My Own Docker Image That Has

Hi @<1533620191232004096:profile|NuttyLobster9>
First nice workaround!
Second could you send the full log? When the venv is skipped then pytorch resolving should be skipped as well, and no error should be raised...
And Lastly could you also send the log of the task that executed correctly (the one you cloned), because you are correct it should have been the same

5 months ago

0 Hi, I'D Like To Know If There Is A Way To Include A Process Like Aws Autoscaler And Its Configurations Inside The Clearml Helm Chart. My Goal Is To Automatically Run The Aws Autoscaler Task On A Clearml-Agent Pod When I Deploy The Clearml Services On The

Nice! TrickySheep9 any chance you can share them ?

3 years ago

0 Hi Guys, We Are Running Clearml-Serving On A Kube Cluster On Aws And We Have Noticed That We Are Getting Some 502 Errors Once In A While That We Can'T Seem To Trace Back.

Hi @<1569858449813016576:profile|JumpyRaven4>

The gunicorn logs do not show anything including any error or trace of the 502 only siege reports the 502 as well as the ALB.Is this an ALB or an ELB ?
What's the timeout its configured?
Do you have GPU instances as well? what's the clearml-serving-inference docker version ?

10 months ago

0 Hi! I Have Local Minio Setup, Via Minio Browser I Can Upload 50-100 Mb Per Second As Its Local. But When I Try To Use Task.Upload_Artifact It Uploads 500 Kb Per Second. Does Anyone Have An Idea About This?

What if I register the artifact manually?

task.upload_artifact('local folder', artifact_object=' ')This one should be quite quick, it's updating the experiment

4 years ago

0 <image>

Let me check, see what can be learned ...

3 years ago

0 It Appears That When I Use Poetry, It Recreates The Environment (Doesn'T Persist The Venv) And Redownloads All The Python Packages (Doesn'T Use A Cache). Is This How The System Currently Works, Or Are There Specific Flags And Such That I Have Not Enabled

Thank you StrangePelican34 !!!

3 years ago

0 Hi Friends! I'M Trying To Upgrade The

I don't have the compose file, or at least can't seem to find it in

/opt

you can manually take down all dockers with:
docker psthen docker stop <container id> for each container id

3 years ago

0 What Sort Of Integration Is Possible With Clearml And Sagemaker? On The Page

What happens when you call:

from clearml.backend_interface.task.repo import ScriptInfo

print(ScriptInfo._ScriptInfo__legacy_jupyter_notebook_server_json_parsing(None))

one year ago

0 Is There A Way To Upload A Dict Object As A Yaml Artifact Instead Of A Json?

Hmm would uploading it as YAML string be better?

3 years ago

0 Hello! Since Today I Get

Hi @<1523701868901961728:profile|ReassuredTiger98> when you get to it...
please download the wheel, then install it with

pip3 install -U clearml_agent-0.17.3rc0-py3-none-any.whl

Then run the daemon with the additional --debug argument, basically:

clearml-agent --debug daemon --foreground ...

Once the agent is running please send the Task's log from your console 🙂

3 years ago

0 Hi Guys, Does Anybody Have The Same Issue Like Me? Is There Any Workaround?

Expected behaviour is that it reads last iteration correctly. At least it is stated in docs so.

This is exactly what should happen, are you saying that for some reason it fails?

2 years ago

0 Roll Call! Who Else Is Here?

What's up @<1523708288686886912:profile|DeliciousBluewhale87>

3 years ago

0 Hello, I use ```'-e', 'CLEARML_AGENT_AGENT_DOCKER_INTERNAL_MOUNTS__SDK_CACHE=/home/username/clearml_agent_cache',``` yet my mount looks like this ```'-v', '/home/clearml/.clearml/cache:/clearml_agent_cache',``` I want clearml to use `/home/username/clearm

Wait @<1715900788393381888:profile|BitingSpider17> are you passing it on a single Task? these values are read by the daemon (i.e. running on the host) which means it is not getting them from the Task context (which leads to zero effect on the mount points)
Notice that in new versions of the clearml-agent the SDK mount point was changed to: sdk_cache: "/clearml_agent_cache" exactly to solve for the non-root containers:
[None](https://github.com/allegroai/clearml-agent/blob/6b31883e4579...

3 months ago

0 Hi, We Saw 2 More Issues With Images Logging: 1. You Need To Use Tf.Summary.Image And Not Summary_Ops_V2.Image 2. Image Needs To Be In Range [0, 1] And Not [0, 255] (Matplotlib And Tensorboard Can Handle Either One) Is That Expected?

I get the same "white" image in both TB & ClearML 😞

3 years ago

Show more results