
Reputation
Badges 1
25 × Eureka!maybe you can check alsoΒ
--version
Β that returns the helm menu
What do you mean? --version on cleaml-task ?
Yeah I think that for some reason the merge of the pbtxt raw file is not working.
Any chance you have an end to end example we could debug? (maybe just add a pbtxt for one of the examples?)
Hi ExasperatedCrocodile76
This is quite the hack, but doable π
`
file_path = task.connect_configuration(name = 'augmentations', configuration = 'augmentations.py')
import importlib
module_name = 'augmentations'
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) `
https://stackoverflow.com/a/54956419
BTW: the same hold for tagging multiple experiments at once
@<1569496075083976704:profile|SweetShells3> remove these from your pbtext:
name: "conformer_encoder"
platform: "onnxruntime_onnx"
default_model_filename: "model.bin"
Second, what do you have in your preprocess_encoder.py
?
And where are you getting the Error? (is it from the triton container? or from the Rest request?
Hi CrookedAlligator14
or is underlying data also accessible?
What do you mean by "underlying data" ?
WickedGoat98 did you setup a machine with trains-agent pulling from the "default" queue ?
In theory task.tags.remove(tag)
might also work, but I'm not sure of it will automatically be updated on the backend
No by definition the agent will only execute one Task at a time, you can spin a second agent on the same GPU :)
The other way around- "8011:8008"
MagnificentPig49 I was not aware of jsonargparse
from what I understand it's a nicer way to parse json configuration files, with argparser alike interface. Did I get that correctly?
Regrading the missing argparser, you are correct, the auto-magic is not working since jsonargparse
is calling an internal ArgParser function and not the external one (hence we miss it).
The quickest fix is adding the following line before you call parse_args()
:task.connect(parent_parser)
Checkout the trains-agent repo https://github.com/allegroai/trains-agent
It is fairly straight forward.
I simplified the code, just so I could test it, this one seems to work, feel free to add the missing argparser parts :)
` from argparse import ArgumentParser
from trains import Task
model_snapshots_path = 'mnt/trains'
task = Task.init(project_name='examples', task_name='test argparser', output_uri=model_snapshots_path)
logger = task.get_logger()
def main(args):
print('Got args: %s' % args)
if name == 'main':
parent_parser = ArgumentParser(add_help=False)
parent_parser....
It should actually work the same, if you find out it fails to properly register let me know (and then I guess a github issue is the next step)
Hi MagnificentPig49 unfortunately it's only in the trains-server docker, we are working on making it "presentable" and uploading it to it's repo.
It's written in Angular (v8 I think). Do you want to help out, it will definitely incentive the guys to tidy up the code and upload it :)
MagnificentPig49 that's a good question, I'll ask the guys π
BTW, I think the main issues is actually making sure there is enough documentation on how to compile it...
Anyhow I'll update here
MagnificentPig49 quick update, front-end guys updated me that with the next trains-server update they will have the web client code available on the repository , ETA probably mid May or so :)
Thanks MagnificentPig49 !
"what's the trains/trains-agent/trains-server versions ?" how can I check it?
trains/trains-agent are pip packages os,pip freeze | grep trains
trains-server you can check in the /profile page top left corner
HI PlainSquid19 could you add a bit more information? Are you running trains-agent ? is it in docker/venv mode ? what's the trains/trains-agent/trains-server versions ?
Is it vanilla pytorch ?
RoundMosquito25 this is a good point, I mean in theory it could be done, the question is the actual Bayesian optimization you are using.
Is it optuna (OptimizerOptuna) or OptimizerBOHB?
No worries, I'll see what I can do π
If you passed the correct path it should work (if it fails it would have failed right at the beginning).
BTW: I think it is clearml-agent --config-file <file here> daemon ...
RoundMosquito25 actually you can π# check the state every minute while an_optimizer.wait(timeout=1.0): running_tasks = an_optimizer.get_active_experiments() for task in running_tasks: task.get_last_scalar_metrics() # do something here
base line reference
https://github.com/allegroai/clearml/blob/f5700728837188d7d6005726c581c9d74fd91164/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py#L127
Hi ElegantCoyote26
sometimes the agents load an earlier version of one of my libraries.
I'm assuming some internal package that is installed from a wheel file not a direct git repo+commit link ?