Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
UnevenDolphin73
Moderator
106 Questions, 749 Answers
  Active since 10 January 2023
  Last activity 9 months ago

Reputation

0

Badges 1

662 × Eureka!
0 Votes
27 Answers
990 Views
0 Votes 27 Answers 990 Views
Is there an autoscaler equivalent for K8s? That is, a service that will launch pods based on incoming requests?
one year ago
0 Votes
7 Answers
1K Views
0 Votes 7 Answers 1K Views
If I create a dataset with Dataset.create(..., use_current_task=True) , that task holds the dataset. Can I then refer/copy/attach the same dataset to other t...
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
Since v1.4.0, our StorageManager.download_folder(..., local_folder='./') is failing - we've had to revert back to 1.3.2. I saw the changelist includes a fix ...
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
One more follow-up still; we're trying to run non-GPU scaler, and I've finally sorted out subnet and security groups issues, only to run into this: Executing...
2 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
Does the ClearML team still post updates in this channel when new ClearML versions come out? If not, what's the easiest way to keep track of various release ...
2 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
How/where can I change the session timeout time for the WebUI? It's timing out WAY too fast after the latest updates
2 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
When I upgrade the server, do I need to do anything with the agents/queues (redefine them, etc)?
2 years ago
0 Votes
0 Answers
1K Views
0 Votes 0 Answers 1K Views
Any easy way to ensure the pipline (tasks + controller) are not hidden in the projects UI? EDIT: Apparently not; for the time being I created this https://gi...
one year ago
0 Votes
5 Answers
964 Views
0 Votes 5 Answers 964 Views
Getting a weird error on local setup for ClearML Server: 2022-03-30 08:07:07,932 - clearml.Task - ERROR - Action failed <500/0: tasks.get_all/v1.0 (mongo:270...
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
Trying to run AWS autoscaler with poetry queue, and I get: Traceback (most recent call last): File "/root/.local/bin/poetry", line 5, in from poetry.console....
2 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
Can I run an autoscaler listening to a regular queue (i.e. combine autoscaling with on premise machines)? Or does it run the danger of creating an EC2 instan...
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
For some reason the agent is now trying to use Python 2.7 all of a sudden, any idea why? 🤔 I see the following in the task's console output: New python exec...
2 years ago
0 Votes
23 Answers
981 Views
0 Votes 23 Answers 981 Views
Back to this ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start 😢 Hanging for the past ~45m like this
2 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
I encountered a weird edge case with the AWS Auto-scaler, wondering if there are any solutions or if this is a known issue. Something as follows happened: Th...
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
Is it possible to avoid the clearml-agent for local installations, and have the file server automatically use an S3 bucket? I've found https://clearml.slack....
3 years ago
0 Votes
3 Answers
998 Views
0 Votes 3 Answers 998 Views
Can we temporarily disable the autologging feature? (specifically for matplotlib ?)
one year ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
We’re randomly getting the following message - 2023-03-07 11:07:26,123 [WARNING] [urllib3.connectionpool]: Retrying (Retry(total=237, connect=237, read=240, ...
one year ago
0 Votes
16 Answers
1K Views
0 Votes 16 Answers 1K Views
We’re running clearml-agent on k8s and I first noticed some warnings from the pod about Python 3.6..? > /usr/lib/python3/dist-packages/secretstorage/dhcrypto...
one year ago
0 Votes
18 Answers
1K Views
0 Votes 18 Answers 1K Views
What would be the best way to approach this flow? We have a configuration file that defines e.g. the project name to use in ClearML, alongside other experime...
2 years ago
0 Votes
4 Answers
986 Views
0 Votes 4 Answers 986 Views
Weird error; my local execution hung with ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start 2022-03-07 ...
2 years ago
0 Votes
11 Answers
1K Views
0 Votes 11 Answers 1K Views
Pipelines suddenly no longer appear in the Pipelines tab, what could/should I look into? EDIT: Using latest ClearML (agent, server, SDK), and creating the pi...
one year ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
I think something is messed up in my remote agents environment, could someone lend a hand? I'm getting this during remote execution (poetry queue): Summary -...
one year ago
0 Votes
23 Answers
1K Views
0 Votes 23 Answers 1K Views
We're using Ray and ClearML together, and suddenly we're seeing some hanging threads, and finally we got an error message: 2022-01-10 09:58:56,803 [ERROR] [C...
2 years ago
0 Votes
3 Answers
987 Views
0 Votes 3 Answers 987 Views
side question - any plans to include native support for lgbm ?
3 years ago
0 Votes
1 Answers
1K Views
0 Votes 1 Answers 1K Views
We're moving from on-premise to AWS - is there an easy way to migrate wall the tasks, projects, etc to the new instance?
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
I realize I'm asking many niche questions - my apologies 🙈 Is there a way to translate task.data.user (user ID) to the actual username? Use case is that I'd...
2 years ago
0 Votes
11 Answers
1K Views
0 Votes 11 Answers 1K Views
ClearML 1.3.2 I'm running Task.get_tasks(project_name=...) and suddenly I get 2022-04-07 10:28:43,617 - clearml.Task - ERROR - Failed reloading task c2d514f3...
2 years ago
0 Votes
18 Answers
1K Views
0 Votes 18 Answers 1K Views
!! In remote execution, do agents inherit the config from the queue from which they pull the task?
2 years ago
0 Votes
16 Answers
1K Views
0 Votes 16 Answers 1K Views
How would I go about adding multiple credentials in the autoscaler? (i.e. specify multiple sdk.aws.s3.credentials ) It's a bit odd that one would also have t...
2 years ago
0 Votes
1 Answers
1K Views
0 Votes 1 Answers 1K Views
Side note for documentation, Task.project reads > Returns the current Task’s project name. But it actually returns the project ID
one year ago
Show more results questions
0 Back To This

Sorry, been away for a while!
I have no additional information, since it was a bug in my model that I have since eliminated...
Maybe it was just a matplotlib error and can be dropped for now. I'll let you know if it pops up again!

2 years ago
0 Back To This

Still happening with 1.4.1 SuccessfulKoala55

2 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

Each user creates a .env file for their needs or exports them in the shell running the python code. Currently I copy the environment variables to an S3 bucket and download it from there.

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

Thanks AgitatedDove14 , I'll first have to prove viability with the free version :)

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

These are per-user. Essentially we log user DB access as well (for various backtracking afterwards), so it's beneficial for us to pass the user DB secrets to the task and not have it configured once on the agent.

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

I mean, I know I could connect_configuration({k: os.environ.get(k) for k in [...]}) , but then those environment variables would be exposed in the ClearML UI, which is not ideal (the environment variables in question hold usernames and passwords, required for DB access)

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

Maybe. When the container spins, are there any identifiers regarding the task etc available? I create a folder on the bucket per python train.py so that the environment variables files doesn't get overwritten if two users execute almost-simultaneously

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

You mean at the container level or at clearml?

Yes, the container level (when these docker shell scripts run).
The per user ID would be nice, except I upload the .env file before the Task is created (it's only available really early in the code).

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

Thanks for the reply CostlyOstrich36 !
Does the task read/use the cache_dir directly? It's fine for it to be a cache and then removed from the fileserver; if users want the data to stay they will use the ClearML Dataset 🙂

The S3 solution is bad for us since we have to create a folder for each task (before the task is created), and hope it doesn't get overwritten by the time it executes.

Argument augmentation - say I run my code with python train.py my_config.yaml -e admin.env...

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

Also I appreciate the time youre taking to answer AgitatedDove14 and CostlyOstrich36 , I know Fridays are not working days in Israel, so thank you 🙂

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

The S3 bucket credentials are defined on the agent, as the bucket is also running locally on the same machine - but I would love for the code to download and apply the file automatically!

3 years ago
0 For Some Reason The Agent Is Now Trying To Use Python 2.7 All Of A Sudden, Any Idea Why?

One way to circumvent this btw would be to also add/use the --python flag for virtualenv

2 years ago
0 Is There A Way To Interface With Clearml Agent (Cli?) To Handle Model Repositories And Data Versioning (But So, Not Experimentation, Tight Integration, Pipelining, Etc)?

Thanks Alon. In the full/official documentation the clearml-data CLI is not mentioned anywhere, so perhaps it should be refreshed 😉

I think we're referring to different things here.
I won't be using the UI (and neither will my team).
But as mentioned, we've used DVC before and it adds a lot of junk metadata files to each GitHub PR (many dvc.yaml , dvc.lock and .gitignore files). We're trying to avoid that as much as possible, hence my question about GitHub pull...

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

The overall flow I currently have is e.g.
Start an internal task (not ClearML Task; MLOps not initialized yet) Call some pre_init function with args so I can upload the environment file via StorageManager to S3 Call some start_run function with the configuration dictionary loaded, so I can upload the relevant CSV files and configuration file Finally initialize the MLOps (ClearML), start a task, execute remotely
I can play around with 3/4 (so e.g. upload CSVs and configuratio...

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

I'm not sure I follow, how would that solution look like?

3 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

Great, thanks! Any idea about environment variables and/or other files (CSV)? I suppose I could use the task.upload_artifact for the CSVs. but I'm still unsure about the environment variables

3 years ago
0 Weird Error; My Local Execution Hung With

It misses the repository information of course, but the 'configuration/Args' were logged. So something weird in identifying the repository

2 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

I guess the big question is how can I transfer local environment variables to a new Task

3 years ago
0 Can I Run An Autoscaler Listening To A Regular Queue (I.E. Combine Autoscaling With On Premise Machines)? Or Does It Run The Danger Of Creating An Ec2 Instance While An On-Premise Machine Takes The Job?

Thanks CostlyOstrich36 !
And can I make sure the same budget applies to two different queues?
So that for example, an autoscaler would have a resource budget of 6 instances, and it would listen to aws and default as needed?

2 years ago
0 I Have Some Code That Launches Ml Tasks And It Accepts A Yaml File,

Maybe this is part of the paid version, but would be cool if each user (in the web UI) could define their own secrets, and a task could then be assigned to some user and use those secrets during boot?

3 years ago
0 Back To This

Not really, I've only been able to somewhat understand the scope of where it happens, and I'm not sure it's even a ClearML issue (maybe matplotlib)

2 years ago
0 Back To This

Trying now with 1.4.1, but I believe the changes you're referring to SuccessfulKoala55 were also introduced in 1.4.0, right?

2 years ago
0 Back To This

1.3.2

2 years ago
0 Can We Report A Pandas Table With Styling To Be Retained In The Webui? It Would Be Nice To Report E.G.

Thanks AgitatedDove14 , I'll give it a try. Perhaps additional documentation is needed for that extra_layout

2 years ago
0 What Is Being Stored Exactly In

AgitatedDove14

hmmm... they are important, but only when starting the process. any specific suggestion ?
(and they are deleted after the Task is done, so they are temp)

Ah, then no, sounds temporary. If they're only relevant when starting the process though, I would suggest deleting them immediately when they're no longer needed, and not wait for the end of the task (if possible, of course)

2 years ago
0 The Comparison Page Seems To Resize The Experiments So That All Tags Will Fit In The Screen, But Then The Experiments Are Pretty Much Impossible To Compare

I can scroll sideways but if I open any of the comparison items, I pretty much can only see one experiment's values

2 years ago
0 What Happens To File That Are Downloaded To A Remote_Execution Via Storagemanager? Are They Removed At The End Of The Run, Or Does It Continuously Increases Disk Space?

I guess it's mixed. If #340 is resolved, then this initializer task will be a no-op: detach, and init-close new tasks as needed.

2 years ago
0 I Encountered A Weird Edge Case With The Aws Auto-Scaler, Wondering If There Are Any Solutions Or If This Is A Known Issue. Something As Follows Happened:

The instance that took a while to terminate (or has taken a while to disappear from the idle workers)

2 years ago
Show more results compactanswers