Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
UnevenDolphin73
Moderator
106 Questions, 749 Answers
  Active since 10 January 2023
  Last activity 10 months ago

Reputation

0

Badges 1

662 × Eureka!
0 Votes
2 Answers
577 Views
0 Votes 2 Answers 577 Views
Is there a GCP driver similar to aws_driver.py ? None
10 months ago
0 Votes
0 Answers
952 Views
0 Votes 0 Answers 952 Views
Also ubuntu 18.04 is losing support so everything in the image needs an update…
one year ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
Our Mac users are having some issues. They have their respective ~/clearml.conf, and yet they get: ClearML 1.1.5 Traceback (most recent call last): ... File ...
2 years ago
0 Votes
10 Answers
1K Views
0 Votes 10 Answers 1K Views
More clarification on documentation (ClearML Data): > Dataset changes are stored using differentiable storage, meaning a version will store the change-set fr...
2 years ago
0 Votes
16 Answers
1K Views
0 Votes 16 Answers 1K Views
How would I go about adding multiple credentials in the autoscaler? (i.e. specify multiple sdk.aws.s3.credentials ) It's a bit odd that one would also have t...
2 years ago
0 Votes
7 Answers
1K Views
0 Votes 7 Answers 1K Views
Can we report a pandas table with styling to be retained in the WebUI? It would be nice to report e.g. df.style.background_gradient(cmap='coolwarm', axis=Non...
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
Is it possible to avoid the clearml-agent for local installations, and have the file server automatically use an S3 bucket? I've found https://clearml.slack....
3 years ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
Follow up on execute_remotely, I see one can limit the available GPU resources in a worker daemon; could one also limit the number of CPU cores available?
2 years ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
Can I shutdown specific workers somehow? Running clearml-agent .... --stop just iterates over all the local workers and shuts them down one by one
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
Is there any way (or are there any plans) to include some view for datasets in the WebUI? One that is detached from the generating task?
2 years ago
0 Votes
13 Answers
1K Views
0 Votes 13 Answers 1K Views
Two things: The release of clearml-agent was not announced in šŸ¤Æ clearml-agent seems outdated wrt requirements, making poetry and clearml-agent incompatible:...
2 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
If, for starters, I'd only like to use ClearML for logging purposes (that is, experiments run locally, and I report metrics, graphs, etc to ClearML). Do I st...
3 years ago
0 Votes
8 Answers
1K Views
0 Votes 8 Answers 1K Views
Any plans to add support for MultiIndex tables? That comma-separation feels a bit odd. (Also this shows up in plots while in the scalars tab I actually have ...
3 years ago
0 Votes
24 Answers
633 Views
0 Votes 24 Answers 633 Views
How can I send a composed chunk of code for remote execution from within a notebook ?
11 months ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
Sometimes, on shorter tasks, my Task would hang for the 5 minute timeout with e.g. 2022-01-04 08:45:25,181 - clearml.Task - INFO - Waiting for repository det...
2 years ago
0 Votes
6 Answers
1K Views
0 Votes 6 Answers 1K Views
How/where can I change the session timeout time for the WebUI? It's timing out WAY too fast after the latest updates
2 years ago
0 Votes
8 Answers
1K Views
0 Votes 8 Answers 1K Views
The ClearML helm charts page is gone? https://allegroai.github.io/clearml-helm-charts/
one year ago
0 Votes
27 Answers
998 Views
0 Votes 27 Answers 998 Views
Is there an autoscaler equivalent for K8s? That is, a service that will launch pods based on incoming requests?
one year ago
0 Votes
23 Answers
986 Views
0 Votes 23 Answers 986 Views
Back to this ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start šŸ˜¢ Hanging for the past ~45m like this
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
Since v1.4.0, our StorageManager.download_folder(..., local_folder='./') is failing - we've had to revert back to 1.3.2. I saw the changelist includes a fix ...
2 years ago
0 Votes
14 Answers
1K Views
0 Votes 14 Answers 1K Views
Automatic logging for generated matplotlib works nicely, but for some reason it names the graphs as "00 untitled" and uses series "plot image" in ClearML UI ...
3 years ago
0 Votes
18 Answers
1K Views
0 Votes 18 Answers 1K Views
Does ClearML somehow remove any loggers from logging module? We suddenly noticed that we have some handlers missing when running in ClearML
2 years ago
0 Votes
8 Answers
975 Views
0 Votes 8 Answers 975 Views
Is there a guide on how to deploy a services agent on a k8s setup? Specifically, with the CLI we use flags such as --services-mode --cpu-only , and I can’t s...
one year ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
I think something is messed up in my remote agents environment, could someone lend a hand? I'm getting this during remote execution (poetry queue): Summary -...
one year ago
0 Votes
29 Answers
1K Views
0 Votes 29 Answers 1K Views
We're trying to use the AWS autoscaler and have managed to get it up and running with spinning up instances. However, it does not seem to pull any of the tas...
2 years ago
0 Votes
2 Answers
999 Views
0 Votes 2 Answers 999 Views
Can we use S3 buckets to cache environments?
2 years ago
0 Votes
1 Answers
1K Views
0 Votes 1 Answers 1K Views
Side note for documentation, Task.project reads > Returns the current Task’s project name. But it actually returns the project ID
one year ago
0 Votes
11 Answers
1K Views
0 Votes 11 Answers 1K Views
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
Trying to run AWS autoscaler with poetry queue, and I get: Traceback (most recent call last): File "/root/.local/bin/poetry", line 5, in from poetry.console....
2 years ago
0 Votes
30 Answers
1K Views
0 Votes 30 Answers 1K Views
I'm trying to set up some initial experiments within our stack, but when I use the execute_remotely task, I get this error: clearml_agent: ERROR: Failed gett...
3 years ago
Show more results questions
0 Pipelines Suddenly No Longer Appear In The Pipelines Tab, What Could/Should I Look Into? Edit: Using Latest Clearml (Agent, Server, Sdk), And Creating The Pipelines Via The Pipelinecontroller Sdk (

Happens with the latest version indeed.
I canā€™t share our code, but the gist of it is:

pipe = PipelineController(name=..., project=..., version=...)

pipe.add_function_step(...)  # Many calls

pipe.set_default_execution_queue(...)
pipe.start(queue=..., wait=True)
one year ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

At any case @<1537605940121964544:profile|EnthusiasticShrimp49> this seems like a good approach, but itā€™s not quite there yet. For example, even if Iā€™d provide a simple def run_step(ā€¦) function, Iā€™d still need to pass the instance to the function. Passing it along in the kwargs for create_function_task does not seem to work, so now I need to also upload the inputs, etc ā€” Iā€™m bringing this up because the pipelines do already do this for you.

11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

So maybe summarizing (sorry for the spam):

  • Pipelines:- Pros: Automatic upload and serialization of input arguments
  • Cons: Clutter, does not support classes, cannot inject code, does not recognize environment when run from e.g. IPython- Tasks:- Pros: Tidier and matches original idea, recognizes environment even when run from IPython
  • Cons: Does not support classes, cannot inject code, does not automatically upload input arguments
11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

More experiments @<1537605940121964544:profile|EnthusiasticShrimp49> - the core issue with the create_function_step seems to be that the chosen executable will be e.g. IPython or some notebook, and not e.g. python3.10 , so it fails running it as a taskā€¦ šŸ¤”

11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

Interesting, why wonā€™t it be possible? Quite easy to get the source code using e.g. dill .

11 months ago
0 Since V1.4.0, Our

Thanks David! I appreciate that, it would be very nice to have a consistent pattern in this!

2 years ago
0 When Uploading An Artifact, Can I List It In Some Grouping (Like With Parameters, Having E.G.

TimelyPenguin76 that would have been nice but I'd like to upload files as artifacts (rather than parameters).

AgitatedDove14 I mean like a grouping in the artifact. If I add e.g. foo/bar to my artifact name, it will be uploaded as foo/bar .

2 years ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

Consider e.g:

# steps.py

class DataFetchingStep:
    def __init__(self, source, query, locations, timestamps):
        # ...
    def run(self, queue=None, **kwargs):
        # ...

class DataTransformationStep:
    def __init__(self, inputs, transformations):
        # inputs can include instances of DataFetchingStep, or local files, for example
        # ...
    def run(self, queue=None, **kwargs):
        # ...

And then the following SDK usage in a notebook:

from steps imp...
11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

I can elaborate in more detail if you have the time, but generally the code is just defined in some source files.
Iā€™ve been trying to play around with pipelines for this purpose, but as suspected, it fails finding the definition for the pickled objectā€¦

11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

I think also the script path in the created task will cause some issues, but letā€™s seeā€¦

11 months ago
0 We'Re Trying To Use The Aws Autoscaler And Have Managed To Get It Up And Running With Spinning Up Instances. However, It Does Not Seem To Pull Any Of The Tasks For The Remote Instances. We See It Gets

I'll see if we can do that still (as the queue name suggests, this was a POC, so I'm trying to fix things before they give up šŸ˜› ).
Any other thoughts? The original thread https://clearml.slack.com/archives/CTK20V944/p1641490355015400 suggests this PR solved the issue

2 years ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

Hey @<1537605940121964544:profile|EnthusiasticShrimp49> ! Youā€™re mostly correct. The Step classes will be predefined (of course developers are encouraged to add/modify as needed), but as in the DataTransformationStep , there may be user-defined functions specified. Thatā€™s not a problem though, I can provide these functions with the helper_functions argument.

  • The .add_function_step is indeed a failing point. I canā€™t really create a task from the notebook because calling `Ta...
11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

Iā€™ll give the create_function_task one more try šŸ¤”

11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

Maybe create_function_task ? šŸ¤”

11 months ago
0 Can Steps Be Removed From Pipelines, And/Or Can Pipelines Be Generally Modified Other Than Adding Steps To Them?

Dynamic pipelines in a notebook, so I donā€™t have to recreate a pipeline every time a step is changed šŸ¤”

11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

Then I wonder:

  • How to achieve this? The pipeline controller seems to only work with functions, not classes, so running smaller steps remotely seems more difficult then I imagined. I was already prepared to upload artifacts myself etc, but now Iā€™m not sure?
  • Do I really need to recreate the pipeline everytime from scratch? Or can I remove/edit steps? Itā€™s mostly used as aā€¦ controller for notebook-based executions and experimentations, before the actual pipeline is known. That is, it will ...
11 months ago
0 How Can I Send A Composed Chunk Of Code For Remote Execution

Thanks @<1537605940121964544:profile|EnthusiasticShrimp49> ! Thatā€™s definitely the route I was hoping to go, but the create_function_task is still a bit of a mystery, as Iā€™d like to use an entire class with relevant logic and proper serialization for inputs, and potentially Iā€™ll need to add more ā€œhelper functionsā€ (as in the case of DataTransformationStep , for example). Any thoughts on that? šŸ¤”

11 months ago
0 Is There A Way To Generate Usage Stats And Reports For Queues? For Example, How Often Is A Queue Used, How Much Cpu Does

We're wondering how many on-premise machines we'd like to deprecate. For that, we want to see how often our "on premise" queue is used (how often a task is submitted and run), for how long, how many resources it consumes (on average), etc.

2 years ago
0 Can I Somehow Change A Project Settings Via The Python Sdk?

I can also do this via Mongo directly, but I was hoping to skip the K8S interaction there.

one year ago
0 Can I Somehow Change A Project Settings Via The Python Sdk?

An internal project I've accidentally made with a hidden tag while playing around with the ClearML internal code.

one year ago
0 Can I Somehow Change A Project Settings Via The Python Sdk?

Where do I import this APIClient from AgitatedDove14 ? I meanwhile edited it directly in mongo, but editing a db directly on a Friday is a big nono

one year ago
0 Can I Somehow Change A Project Settings Via The Python Sdk?

I'd like to remove the hidden system tag from a project

one year ago
0 For Some Reason The Agent Is Now Trying To Use Python 2.7 All Of A Sudden, Any Idea Why?

One way to circumvent this btw would be to also add/use the --python flag for virtualenv

2 years ago
0 For Some Reason The Agent Is Now Trying To Use Python 2.7 All Of A Sudden, Any Idea Why?

Still failing with 1.2.0rc3 šŸ˜ž AgitatedDove14 any thoughts on your end?

2 years ago
0 When Uploading An Artifact, Can I List It In Some Grouping (Like With Parameters, Having E.G.

So basically I'm wondering if it's possible to add some kind of small hierarchy in the artifacts, be it sections, groupings, tabs, folders, whatever.

2 years ago
0 Clearml Pipelines Can Be Build From Tasks, Functions, And Decorated Functions, According To The Examples In

I think -

  • Creating a pipeline from tasks is useful when you already ran some of these tasks in a given format, and you want to replicate the exact behaviour (ignoring any new code changes for example), while potentially changing some parameters.
  • From decorators - when the pipeline logic is very straightforward and you'd like to mostly leverage pipelines for parallel execution of computation graphs
  • From functions - as I described earlier :)
one year ago
Show more results compactanswers