Reputation
Badges 1
103 × Eureka!Hi SweetBadger76 ,
Well - apparently I was mistaken.
I still have a ghost worker that i'm mot able to remove (I had 2 workers on the same queue - that caused my confusion).
I can see it in the UI and when I run clearml-agent list
And although I'm stoping the worker specificallyclearml-agent daemon --stop <worker_id>
I'm gettingCould not find a running clearml-agent instance with worker_name=<worker_id> worker_id=<worker_id>
Just for the record - for who ever will be searching for a similar setup with colab
prerequisitecreate a dedicated Service Account (I was not able to authenticate with a regular User credentials (and not SA)) get SA key ( credentials.json ) Upload json to an ephemeral location (e.g. root of colab)login into ClearML Web UI - Create access key for user - https://clear.ml/docs/latest/docs/webapp/webapp_profile#creating-clearml-credentials prepare credentials` %%bash
export api=`ca...
Hi SweetBadger76
Further investigation showed that the worker was created with a dedicated CLEARML_HOST_IP
- so running the
clearml-agent daemon --stop
didn't kill it (but it did appear in the clearml-agent list
But once we added the
CLEARML_HOST_IP `
CLEARML_HOST_IP=X.X.X.X clearml-agent daemon --stop
it finally killed it
will do
A work around that worked for me is to explicitly complete the task, seems like the flush
has some bug
task = Task.get_task('...')
task.close()
task.mark_completed()
ds.is_final()
True
Hi CostlyOstrich36 ,
After verifying - I can confirm that there is no custom certificate .
any other ideas?
Dataset.list_datasets(dataset_project='XXXX')
Always returns an empty list
That is a workaround - but surly not optimal
If we want to generate a dataset from a set of files that are on a local computer (e.g. a local GPU workstation then ran some media transformation) -
then instead of creating the Dataset
directly - we need to first upload them and only then use the ClearML
sdk.
Do you see any option integrating this kind of workflow into clearml?
Sorry -
After updating the repo I can see that the newest chart is 4.1.1
SweetBadger76 should I update to this version?
Hi SweetBadger76 -
I'm I misunderstanding how this tests
worker runs?
exactly - (that is how I used it in my initial code) - but if you have to convert it back to the original data type then something is broken...
I saw https://clear.ml/docs/latest/docs/references/sdk/dataset/#verify_dataset_hash - but I don't think it is the correct one. the https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shape.html property
This does not work -
Since all the files are stored as a single ZIP file (which if unzipped will have all the data), but we would like to have access to the raw files in there original format.
Thx again for your time -
Where the experiment is being executed
Not sure I understand what you mean by this -
Assuming that we are running the ClearML on GKE (we have succeeded doing so) - and running the python code from COLAB or locally. Where do we configure the Google Storage ? how can the helm / k8s dynamically load the clearml.conf
? is it only from values.yaml
?
Where you view your experiment
In mlflow
I was able to view the artifact
directly (a...
Hi,
You may want to consider to do the visualizing while creating the Datasets - see https://github.com/thepycoder/asteroid_example/blob/main/get_data.py#L34 logging the head()
of the dataframe
Hi SuccessfulKoala55
Is this section only relevant to AWS or also to GCP?
Possibly - thinking more of https://github.com/pytorch/data/blob/main/examples/vision/caltech256.py - using clearml dataset as root path.
Thx - it worked!
BTW - maybe it worth while to add this comment in the ClearML Agent daemon documentation - that when ever you update the clearml.conf
you need to
clearml-agent daemon --stop recreate all the daemonclearml-agent daemon ....
Is there any settings that we need to take into account when working with session
?
in the https://clear.ml/docs/latest/docs/apps/clearml_session#accessing-a-git-repository it mentions accessing Git Repository -
Can you run clearml sessions
without accessing Git? Assuming we are using ssh
- what is the correct configuration?
Great - Thx TimelyPenguin76 for your input
google.storage { credentials = [ { bucket: "clearml-storage" project: "my-project" credentials_json: "/path/to/creds.json" }, ] }
No - just emulating - it is more of /home/... /creds.json
Looking in the repo I was not able to see an example - only reference to https://github.com/allegroai/clearml/blob/b9b0a506f35a414f6a9c2da7748f3ec3445b7d2d/docs/clearml.conf#L13 - I just need to add company.id
or user.id
in the credential dict?
Hi AgitatedDove14
OK - the issue was the firewall rules that we had.
Now both of the jupyter lab
and vscode
servers are up.
But now there is an issue with the Setting up connection to remote session
After the
Environment setup completed successfully
Starting Task Execution:
ClearML results page:
There is a WARNING
clearml - WARNING - Could not retrieve remote configuration named 'SSH'...
@<1523701205467926528:profile|AgitatedDove14> -
I'm getting the following error when running the following code within the mp_worker
command = ["ffmpeg","-i",f"{url}","-vcodec","libx264", "output.mp4"]
subprocess.run(command, stderr=subprocess.STDOUT)
TypeError: fork_exec() takes exactly 21 arguments (17 given)
Any suggestions?
Ok - I can see that if I ran finalize(auto_upload=True)
on the dataset - I get all the information in the UI.
Way is this necessary?