using the helm charts https://github.com/allegroai/clearml-helm-charts

I saw https://clear.ml/docs/latest/docs/references/sdk/dataset/#verify_dataset_hash - but I don't think it is the correct one. the https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html property

Also, what version are you on?

of what?

Hi

you will have to configure the credentials there (in a local 

clearml.conf

 or using environment variables

This is the part that confuses me - is there a way to configure clearml.conf from the values.yaml ? I would like the GKE to load the cluster with the correct credentials without logging into the pods and manually updating the claerml.conf file

exactly - (that is how I used it in my initial code) - but if you have to convert it back to the original data type then something is broken...

but it is on the host server

AgitatedDove14 -
I also tried to https://github.com/allegroai/clearml-session
running the session within docker but got the same error

clearml-session --docker

--git-credentials

(there is a typo in git - --git-credent ila s -> --git-credent ials)
and still got the same error

clearml_agent: ERROR: Can not run task without repository or literal script in script.diff

@<1523701205467926528:profile|AgitatedDove14> -
I'm getting the following error when running the following code within the mp_worker

command = ["ffmpeg","-i",f"{url}","-vcodec","libx264", "output.mp4"]
subprocess.run(command, stderr=subprocess.STDOUT)

TypeError: fork_exec() takes exactly 21 arguments (17 given)

Any suggestions?

CostlyOstrich36 - but we will use any method that will allow us to save the files as parquet.
We are not yet using clearml Dataset - i'm not sure if this is a solution

This is my current solution
[ds for ds in dataset.list_datasets() if ds['project'].split('/')[0]==<PROJEFCT_NAME>]

I'm looking for the bucket URI
I think my work flow needs to alter.
get the data into the bucket and then create the Dataset using the add_external_file and then be able to consume the data locally or stream And then I can use - link_entries

But this is not on the pods, isn't it? We're talking about the python code running from COLAB or locally...?

correct - but where is the clearml.conf file?

so running the command clearml-agent -d list returns the https://clearml.slack.com/archives/CTK20V944/p1657174280006479?thread_ts=1657117193.653579&cid=CTK20V944

I found the task in the UI -
and in the UNCOMMITTED CHANGES execution section there is

No changes logged

Any other suggestions?

Ok - I can see that if I ran finalize(auto_upload=True) on the dataset - I get all the information in the UI.
Way is this necessary?

Thx for your reply

We have assets in a GCP bucket.
The dataset is created and then the assets are linked to the dataset via the add_external_files method

Are you running the "cleamrl-session" from your machine? (i.e. not from inside a docker) ?

correct - running it locally - not inside docker . Should I try to run within a docker?

Can you send the full clearml-session console output ?

see above

I think I have a lead.
looking at list of workers from clearml-agent list e.g. https://clearml.slack.com/archives/CTK20V944/p1657174280006479?thread_ts=1657117193.653579&cid=CTK20V944
is there a way to find the worker_name ?
in the above example the worker_id is clearml-server-agent-group-cpu-agent-5df4476cfc-j54gh:0 but I'm not able to stop this worker using the command

clearml-agent daemon --stop

since this orphan worker has no corresponding clearml.conf

Using the https://allegro.ai/clearml/docs/rst/references/clearml_python_ref/task_module/task_task.html?highlight=upload_artifact#clearml.task.Task.upload_artifact method. It works well, but only saves it as a csv (which is very problematic since when loading the artifact none of the data types of the columns are preserved...)

Hi SweetBadger76 ,
Well - apparently I was mistaken.
I still have a ghost worker that i'm mot able to remove (I had 2 workers on the same queue - that caused my confusion).
I can see it in the UI and when I run clearml-agent list
And although I'm stoping the worker specifically
clearml-agent daemon --stop <worker_id>I'm getting
Could not find a running clearml-agent instance with worker_name=<worker_id> worker_id=<worker_id>