Ok - I can see that if I ran finalize(auto_upload=True) on the dataset - I get all the information in the UI.
Way is this necessary?

I saw https://clear.ml/docs/latest/docs/references/sdk/dataset/#verify_dataset_hash - but I don't think it is the correct one. the https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html property

Sorry - I'm a Helm newbee
when running
helm search repo clearml --versionsI can't see version 3.6.2 - the highest is 3.5.0
This is the repo that we used to get the helm chart
helm repo add allegroaiWhat I'm I missing?

here is a video of the steps

Great - Thx for the clarifications!

Hi

you will have to configure the credentials there (in a local 

clearml.conf

 or using environment variables

This is the part that confuses me - is there a way to configure clearml.conf from the values.yaml ? I would like the GKE to load the cluster with the correct credentials without logging into the pods and manually updating the claerml.conf file

I'm guessing .1 is since there were datasets that I could not see - but actually they were there (as sub projects). so everything is related

Hi AgitatedDove14
OK - the issue was the firewall rules that we had.

Now both of the jupyter lab and vscode servers are up.

But now there is an issue with the Setting up connection to remote session

After the

Environment setup completed successfully
Starting Task Execution:
ClearML results page:

https://app-clearml.XXXX/projects/XXXXXXXXXX/output/log

There is a WARNING

clearml - WARNING - Could not retrieve remote configuration named 'SSH'...

Hi SuccessfulKoala55
Is this section only relevant to AWS or also to GCP?

Version: 1.6.4

In order to create a webdataset we need to create tar files -
so we need to unzip and then recreate the tar file.
Additionally when the files are in GCS in the raw format you can easily review them with the preview (e.g. a wav file can be directly listened within the GCP console - web browser).
I think the main difference is that I can see a value of having access to the raw format within the cloud vendor and not only have it as an archive

Hi SweetBadger76
Further investigation showed that the worker was created with a dedicated CLEARML_HOST_IP - so running the

clearml-agent daemon --stop

didn't kill it (but it did appear in the clearml-agent list But once we added the CLEARML_HOST_IP `

CLEARML_HOST_IP=X.X.X.X clearml-agent daemon --stop

it finally killed it

using the helm charts https://github.com/allegroai/clearml-helm-charts

CLEARML-AGENT version 1.2.2

will do
A work around that worked for me is to explicitly complete the task, seems like the flush has some bug
task = Task.get_task('...')
task.close()
task.mark_completed()

ds.is_final()
True

Hi SmugDolphin23
Do you have a timeline for fixing this https://clearml.slack.com/archives/CTK20V944/p1661260956007059?thread_ts=1661256295.774349&cid=CTK20V944

I've updated the configuration and now i'm able to see sub projects that I didn't see before.
As I can see - each dataset is a separate sub project - is that correct?

1 dataset with over 100 versions.

How do you do so?
can't find it under
clearml-agent daemon -h

BTW SuccessfulKoala55 we are running with
CLEARML-AGENT version 1.3.0

not sure I understand
running
clearml-agent listI get
`
workers:

• company:
  id: d1bd92...1e52b
  name: clearml
  id: clearml-server-...wdh:0
  ip: x.x.x.x
  ... `