Examples: query, "exact match", wildcard*, wild?ard, wild*rd
Fuzzy search: cake~ (finds cakes, bake)
Term boost: "red velvet"^4, chocolate^2
Field grouping: tags:(+work -"fun-stuff")
Escaping: Escape characters +-&|!(){}[]^"~*?:\ with \, e.g. \+
Range search: properties.timestamp:[1587729413488 TO *] (inclusive), properties.title:{A TO Z}(excluding A and Z)
Combinations: chocolate AND vanilla, chocolate OR vanilla, (chocolate OR vanilla) NOT "vanilla pudding"
Field search: properties.title:"The Title" AND text
Answered
Greetings Everyone, In The Course Of My Work, I Utilize A Particular Library That Necessitates More Than Just A Simple Clone And Dependency Installation Procedure. It Also Requires The Cloning Of An Additional Repository, Along With Its Installation, And

Greetings everyone,

In the course of my work, I utilize a particular library that necessitates more than just a simple clone and dependency installation procedure. It also requires the cloning of an additional repository, along with its installation, and the subsequent setting of a certain number of environmental variables.

Regrettably, ClearML, the platform I use for remote experiments, by default only logs git, git changes, and dependencies. Given this constraint, I am seeking guidance on how I could append specific commands that would be executed immediately after the main repository has been cloned and all the required dependencies have been installed.

Could anyone provide some insights or advice on how to accomplish this? I'd greatly appreciate it.

  
  
Posted one year ago
Votes Newest

Answers 8


@<1523701205467926528:profile|AgitatedDove14> Perhaps somewhere inside clear ml there is an order of actions for starting that can be changed?

  
  
Posted one year ago

Thanks a lot. I meant running a bash script after cloning the repository and setting the environment

Hmm that is currently not supported 😞
The main issue in adding support is where to store this bash script...

Perhaps somewhere inside clear ml there is an order of actions for starting that can be changed?

Not that I can think of,
but let's assume you could have such a thing, what would you have put in the bash script (basically I want to see maybe there is a workaround for you, based on your script)

  
  
Posted one year ago

see here the docker_setup_bash_script argument
None
It will be executed (no need for the #!/bin/bash btw) before starting to setup the env inside the container, so apt-get and the like can be executed if needed. Notice that if this is something that Always needs to be executed, you can put the same list of commands here: None

  
  
Posted one year ago

@<1523701205467926528:profile|AgitatedDove14> The bash script does the unloading of the necessary resources from aws and sets the environment variable
aws s3 cp ..... --recursive
export PYTHONPATH=" "
All commands can be added to the generated docker image, but you will have to change the project structure

  
  
Posted one year ago

Oh if this is the case, then by all means push it into your Task's docker_setup_bash_script
It does not seem to have to be done after the git clone, the only part the I can see is setting the PYTHONPATH to the additional repo you are pulling, and that should work.
The main hurdle might be passing credentials to git, but if you are using SSH it should be transparent
wdyt?

  
  
Posted one year ago

Thank you for your response @<1523701205467926528:profile|AgitatedDove14> . I will definitely try the solutions you described above. Could you please advise if it is possible to execute the "bash.sh" script directly before the environment setup stages for reproducing the experiment? The repository setup involves downloading resources from AWS. While creating a container that incorporates my requirements would help solve this problem, I am interested in finding a more flexible approach.

  
  
Posted one year ago

Hi @<1524560082761682944:profile|MammothParrot39>
The traditional solution is git submodules, basically main repo links to other repos. This way the agent can fully reproduce the full env.
Another option is to install the second repo as Python package with link to the repo and commit
And a third option is having the second repo as part of the docker.
Regrading env variables, you can add '-e env=val' as part if the docker arts section
Wdyt?

  
  
Posted one year ago

@<1523701205467926528:profile|AgitatedDove14> Thanks a lot. I meant running a bash script after cloning the repository and setting the environment

  
  
Posted one year ago
1K Views
8 Answers
one year ago
one year ago
Tags