Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Profile picture
SarcasticSparrow10
Moderator
13 Questions, 58 Answers
  Active since 10 January 2023
  Last activity 2 years ago

Reputation

0

Badges 1

58 × Eureka!
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Hi
Hi AgitatedDove14 , I put together some improvements for the parallel coordinate chart on this https://github.com/allegroai/trains/issues/259 . What are your...
4 years ago
0 Votes
14 Answers
1K Views
0 Votes 14 Answers 1K Views
I'm using trains-agent to run an experiment from a private git repo and I run into this error trains_agent: ERROR: Failed cloning repository. 1) Make sure yo...
4 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Is it possible to have clearml launch aws instances to run clearml-agent , execute jobs and then shutdown after the jobs have been completed? I have two rele...
aws
3 years ago
0 Votes
3 Answers
1K Views
0 Votes 3 Answers 1K Views
Where is the stdout of trains-agent stored in the service mode (running in background)?
4 years ago
0 Votes
4 Answers
1K Views
0 Votes 4 Answers 1K Views
Hello, I'm logging a plotly figure which contains subplots using Logger.report_plotly() method. However, many of the attributes (color, legend label) of the ...
4 years ago
0 Votes
2 Answers
1K Views
0 Votes 2 Answers 1K Views
Is there an upgrade guide from Trains to ClearML? I can't seem to find it.
4 years ago
0 Votes
8 Answers
1K Views
0 Votes 8 Answers 1K Views
Quick question on trains-agent and HPO. Say I have 10 experiments enqueued to a trains-agent . I understand the agent runs the experiment one-by-one. But can...
4 years ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
I am using reuse_last_task_id=True to overwrite an existing task (with same project and task name). But the experiment contains the torch model and therefore...
4 years ago
0 Votes
25 Answers
1K Views
0 Votes 25 Answers 1K Views
Hi ClearML, I tried to upgrade the clearml server following this https://clear.ml/docs/latest/docs/deploying_clearml/upgrade_server_aws_ec2_ami but it erased...
2 years ago
0 Votes
10 Answers
1K Views
0 Votes 10 Answers 1K Views
3 years ago
0 Votes
5 Answers
1K Views
0 Votes 5 Answers 1K Views
Hi, can we search tasks using wildcard in the webapp. Say I have task names exp_test1_i1 , exp_test2_i1 and I'd like to search for "exp" and "i1". Is that po...
4 years ago
0 Votes
7 Answers
1K Views
0 Votes 7 Answers 1K Views
Hi
Hi AgitatedDove14 , I'd appreciate your thoughts on trains-agent on the following topic. To run an experiment by a trains-agent , it must have already been r...
4 years ago
0 Votes
22 Answers
1K Views
0 Votes 22 Answers 1K Views
Hi
Hi AgitatedDove14 , I upgraded clearml from 0.17.4 to 0.17.5rc2 and the change broke my code as it seems like clearml has started using multiprocessing. I ge...
4 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

I had to manually create a dump for the mongo data and import it into 4.4. I was just referring to adding a note to the documentation for other users.

2 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

I cannot execute step 4 because I can't get past step 3. Does that make sense?

2 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

There was some complication during the upgrade so I had to resort to the manual process.

I have now been able to upgrade by dumping the mongodb data and restoring it independently.

2 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

The docker container in step 3 does not run because of the incompatibility

2 years ago
0 Quick Question On

Great, thanks!

4 years ago
0 Quick Question On

(Do notice that even though you can spin two agents on the same GPU, the nvidia drivers cannot share allocated GPU memory, so if one Task consumes too much memory the other will not have enough free GPU memory to run)

Basically the same restriction as manually launching two processes using the same GPU

That makes sense. Currently, I use python multiprocessing to launch multiple experiments on the sam GPU device. I'm guessing using trains-agent will be similar

4 years ago
0 I'M Using

Ok, So Git credentials are present at two locations - 1) outside the agent config and 2) inside it. I updated credentials at both locations and now I'm seeing agent.git_user = <username> in the dump, but I still have the same issue.
` # Set GIT user/pass credentials

leave blank for GIT SSH credentials ...

4 years ago
0 Quick Question On

Got it. I haven't tried setting up trains-agent yet so I don't know much about the overhead of launching the agent. I'd imagine if it has to create the full environment (installing requirements, etc), the overhead might not be that low. But as I'm reading, it looks like I can use a docker image with the full env. Is my understanding correct?

4 years ago
0 I'M Using

SuccessfulKoala55
For security reasons I don't want to have my password written out in a file. I'm trying to use https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token (PAT) from Github but I get authentication error. Is there an issue using PAT?

4 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

Yes, I tried to run steps 1,2,3,4 in order but got stuck at 3

2 years ago
0 Hi

Wait but that will skip all the assertion checks that I have in my code?!

4 years ago
0 Hi

Yes, I am using Pool. Here is what I think is happening. clearml launches a subprocess which I assume is a daemonic process. That process in-turn launches a subprocess for training which causes the error I mentioned

4 years ago
0 I'M Using

I'm using docker to run the experiment. Could it be that the config in the docker container doesn't have the git credentials?

4 years ago
0 I'M Using

That makes sense. The configuration file is located at ~/trains.conf which I believe is the default location.

No I can't see my username printed out in the dump

4 years ago
0 Hi

Haha.. that would be a problem then!

4 years ago
0 Hi

Thanks for the tip with the config file. I have reverted back to 0.17.4 but will try this.

4 years ago
0 Hi

You mean without manually executing it once ?

Yes. Just as it would be executed in trains

4 years ago
0 Hi

This is happening manually. I am not using agent yet

4 years ago
0 Hi

The second subprocess is by design. It becomes the primary process when clearml does not use multiprocessing. I hope I'm not confusing you further

4 years ago
0 I'M Using

fatal: could not read Username for ' ': terminal prompts disabled error: Could not fetch originWhy is trains-agent trying read from terminal prompt instead of trains.conf ?

4 years ago
0 I'M Using

but when I try to clone the repo directly, PAT works.

4 years ago
0 I'M Using

SuccessfulKoala55 Yes, I am using the --docker flag.

You are right about the Keyring. Once I make sure credentials are stored in a secure way, it works as expected. Thanks :)

4 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

Steps 1 and 2 on this https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_mongo44_migration/ say to backup opt/clearml/data/mongo and uncompress into /opt/clearml/data/mongo_4 . Isn't it just copying the old data files?

2 years ago
0 Hi

2. interesting error, maybe we can revert to "thread mode" if running under a daemon. (I have to admit, I'm not sure why python has this limitation, let me check it...)

Yes, I'm not sure either. I have banged my head against the wall in trying to have multiple level of subprocesses, but it gets too complicated with python. Let me know what you find out

4 years ago
0 Hi Clearml, I Tried To Upgrade The Clearml Server Following This

I was getting the error in step number 3

2 years ago
0 Hi, Which Database Services Are Used To Store The Logged Data Such As Scalar, Text, Matrix, Etc? How Can I Query These For A Downstream Process Programmatically Instead Of Just Within The Web Ui? If Scalar Data Is Stored In Mongodb, Can I Use Pymongo To R

Hi AgitatedDove14 Thanks, I'll check these out.

What is the exact use case you have in mind?

I want to store some additional data that is not relevant to training a model. For example, store inference results, explanations, etc and then use them in a different process. I currently use separate database for this.

Btw, I had been busy with another project and hadn't logged in here for some time. I see that you guys have made a lot of progress in the last two months! I'm excited to di...

3 years ago
Show more results compactanswers