Yes, passing custom object between steps should be possible. The only condition is for the objects to be pickleable. What are you returning exactly from init_experiment ?
That makes sense, yeah it would be nice to have a way to exclude some files when calling sync_folder
OutrageousSheep60 that is correct, each dataset is in a different subproject. That is why bug 2. happens as well
Regarding 1. , are you trying to delete the project from the UI? (I can't see an attached image in your message)
Hi @<1545216070686609408:profile|EnthusiasticCow4> ! Can't you just get the values of the hyperparameters and the losses, then plot them with something like mathplotlib then just report the plot to ClearML?
Could you please try with an older sdk version just to make sure there were no regressions?
Hi @<1554638160548335616:profile|AverageSealion33> ! We pull git repos to copy the directory your task is running in. Because you deleted .git , we can't do that anymore. I think that, to fix this, you could just run the agent in the directory .git previously existed.
@<1634001100262608896:profile|LazyAlligator31> it looks like the args get passed to a python thread. so the should be specified the same way as you would pass them to the args argument in a thread (so a tuple of positional arguments): func_args=("something", "else") . It looks like passing kwargs is not directly supported, but you could build a partial :
from functools import partial
scheduler.add_task(schedule_function=partial(clone_enqueue, arg_1="something", arg_2="else")...Hi @<1693795212020682752:profile|ClumsyChimpanzee88> ! Not sure I understand the question. If the commit ID does not exist remotely, then it can't be pulled. How would you pull the commit to another machine otherwise, is this possible using your current workflow?
Hi @<1523703107031142400:profile|FlatOctopus65> ! python3.9 introduced a breaking change for codebases that parse code containing slices. You can read more about it here: None . Notable:
* The code that produces a Python code from AST will need to handle indexing with tuples specially (see Tools/parser/unparse.py) because d[(a, b)] is valid syntax (although parenthesis are redundant), but d[(a, b:c)] is not.
What you could do is downgrade to...
@<1554638160548335616:profile|AverageSealion33> looks like hydra pulls the config relative to the scripts directory, and not the current working directory. The pipeline controller actually creates a temp file in /tmp when it pulls the step, so the script's directory will be /tmp and when searching for ../data , hydra will search in / . The .git likely caused your repository to be pulled, so your repo structure was created in /tmp , which caused the step to run correctly...
(We will deprecate continue_on_fail)
UnevenDolphin73 Yes it makes sense. At the moment, this is not possible. When using use_current_task=True the task gets attached to the dataset and moved under dataset_project/.datasets/dataset_name . Maybe we could make the task not disappear from its original project in the near future.
Hi @<1639799308809146368:profile|TritePigeon86> ! Please see continue_behaviour . You should be able to pass the parameter to your parent step. It is not documented yet, but it should be available in the latest version of clearml. See this for some documentation: None
Hi @<1654294828365647872:profile|GorgeousShrimp11> ! add_tags is an instance method, so you will need the controller instance to call it. To get the controller instance, you can do PipelineDecorator.get_current_pipeline() then call add_tags on the returned value. So: PipelineDecorator.get_current_pipeline().add_tags(tags=["tag1", "tag2"])
Hi @<1676400486225285120:profile|GracefulSquid84> ! Each step is indeed a clearml task. You could try using the step ID. Just make sure you pass the ID to the HPO step (you can do that by simply returning the Task.current_task().id
Hi @<1523701842515595264:profile|PleasantOwl46> ! This looks like a python problem. A useful SO thread: None
First, I would verify that I can access the api server without using the SDK. To do so, run this code after filling the credentials yourself (just login should be enough to verify that the api server is reachable)
api_server = ""
access_key = ""
secret_ke...Hi BoredHedgehog47 ! We tried to reproduce this, but failed. What we tried is running the attached main.py which Popen s sub.py .
Can you please run main.py as well and tell us if you still encounter the bug? If not, is there anything else you can think of that could trigger this bug besides creating a subprocess?
Thank you!
Hi SoreHorse95 ! I think that the way we interact with hydra doesn't account for overrides. We will need to look into this. In the meantime, do you also have somesort of stack trace or similar?
@<1675675705284759552:profile|NonsensicalAnt77> have you tried setting secure: true and host: storage.yandexcloud.net:443 ?
Hi @<1603198134261911552:profile|ColossalReindeer77> ! The usual workflow is that you modify the fields in your remoter run in either the Hyperparameters section or the configuration section, but not usually both (as in Hydra's case). When using CLI tools, people mostly modify the Hyperparameters section so we chose to set the allow_omegaconf_edit to False by default for parity.
We will add this to the SDK soon
Hi ShortElephant92 ! Random images, audio files, tables (trimmed to a few rows) are sent as Debug Samples for preview. By default, they are sent to our servers. Check this function if you wish to log the samples to another destination https://clear.ml/docs/latest/docs/references/sdk/logger/#set_default_upload_destination .
You could also add these entries in you clearml.conf to not send any samples for preview:
` sdk.dataset.preview.tabular.table_count: 0
sdk.dataset.preview.media.i...
Hi @<1635088270469632000:profile|LividReindeer58> ! parent_task.artifacts[artifact_name].get() should just work to get artifacts from the parent task (the artifact should be automatically unpickled). Are you getting any errors when you do this?
DangerousDragonfly8 you can try to start the pipeline like this:pipe.start(step_task_completed_callback=callback)where callback has the signature:def callback(pipeline, node, parameters): print(pipeline, node, parameters)Note that even tho the parameter name is step_task_completed_callback , it is actually ran before the task is started. This is actually a bug...
We will need to review the callbacks, but I think you can work with this for now...
Hi @<1691258557502066688:profile|RipeCrab75> ! You should make the code accessible to the agent somehow. You could either publish mypackage or push the project to a repository and use None to specify that repo if it is not automatically picked up.
Hi @<1694157594333024256:profile|DisturbedParrot38> ! We weren't able to reproduce, but you could find the source of the warning by appending the following code at the top of your script:
import traceback
import warnings
import sys
def warn_with_traceback(message, category, filename, lineno, file=None, line=None):
log = file if hasattr(file,'write') else sys.stderr
traceback.print_stack(file=log)
log.write(warnings.formatwarning(message, category, filename, lineno, line))
...