CLEARML-AGENT version 1.2.2

try
None
or
None

Hi SuccessfulKoala55
Is this section only relevant to AWS or also to GCP?

Version: 1.6.4

BTW - is the CLEARML_HOST_IP relevant for the clearml-agent ?
i can see that we can create a worker with this environment variable . e.g.
CLEARML_WORKER_NAME=MY-WORKDER CLEARML_WORKER_ID=MY-WORKER:0 CLEARML_HOST_IP=X.X.X.X clearml-agent daemon --detachedmy mistake doesn't use it to create a dedicated IP

we want to use the dataset output_uri as a common ground to create additional dataset formats such as https://webdataset.github.io/webdataset/

What key is used for this? I can only see git_user

That is a workaround - but surly not optimal
If we want to generate a dataset from a set of files that are on a local computer (e.g. a local GPU workstation then ran some media transformation) -
then instead of creating the Dataset directly - we need to first upload them and only then use the ClearML sdk.
Do you see any option integrating this kind of workflow into clearml?

1 dataset with over 100 versions.

i've doubled checked the Path- and it is correct

but it is on the host server

yes - the agent is running with --docker
Great - where do I define the volume mount?
Should I build a base image that runs on the server and then use it as the base image in the container?

Btw -after updating clearml.conf do I need to restart the agent?
I can't see the additional tab under https://clearml.slack.com/archives/CTK20V944/p1658199530781499?thread_ts=1658166689.168039&cid=CTK20V944 , and I reran the task and got the same error

How do you do so?
can't find it under
clearml-agent daemon -h

Thx - it worked!
BTW - maybe it worth while to add this comment in the ClearML Agent daemon documentation - that when ever you update the clearml.conf you need to
clearml-agent daemon --stop recreate all the daemonclearml-agent daemon ....

In order to create a webdataset we need to create tar files -
so we need to unzip and then recreate the tar file.
Additionally when the files are in GCS in the raw format you can easily review them with the preview (e.g. a wav file can be directly listened within the GCP console - web browser).
I think the main difference is that I can see a value of having access to the raw format within the cloud vendor and not only have it as an archive

This does not work -
Since all the files are stored as a single ZIP file (which if unzipped will have all the data), but we would like to have access to the raw files in there original format.

add the google.storage parameters to the conf settings
sdk { google.storage { credentials = [ { bucket: "clearml-storage" project: "dev" credentials_json: /path/to/SA/creds/user.json }, ] } }%

opened an https://github.com/allegroai/clearml/issues/668

will do
A work around that worked for me is to explicitly complete the task, seems like the flush has some bug
task = Task.get_task('...')
task.close()
task.mark_completed()

ds.is_final()
True

exactly - (that is how I used it in my initial code) - but if you have to convert it back to the original data type then something is broken...

we reinstalled the clearml-agent$clearml-agent --version CLEARML-AGENT version 1.2.3running top | grep clearmlwe can see the agent running
running clearml-agent listwe can see 2 workers
before running clearml-agent daemon --stopWe updated the clearml.conf and updated the worker_id and worker_name with the relevant name/id that we can see from clearml-agent list
and we get
` Could not find a running clearml-agent instance with worker_name=<clearml_worker_na...

Hi AnxiousSeal95 ,
Is there an estimate when the above feature will be available?

Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

This also may help with the configuration for GCS
https://clearml.slack.com/archives/CTK20V944/p1635957916292500?thread_ts=1635781244.237800&cid=CTK20V944

google.storage { credentials = [ { bucket: "clearml-storage" project: "my-project" credentials_json: "/path/to/creds.json" }, ] }No - just emulating - it is more of /home/... /creds.json

here is the https://clearml.slack.com/archives/CTK20V944/p1657219266857559

AgitatedDove14 -
I also tried to https://github.com/allegroai/clearml-session
running the session within docker but got the same error

clearml-session --docker

--git-credentials

(there is a typo in git - --git-credent ila s -> --git-credent ials)
and still got the same error

clearml_agent: ERROR: Can not run task without repository or literal script in script.diff

Are you running the "cleamrl-session" from your machine? (i.e. not from inside a docker) ?

correct - running it locally - not inside docker . Should I try to run within a docker?

Can you send the full clearml-session console output ?

see above

SmugDolphin23 I'll be happy to checkout the RC once it is fixed