Reputation
Badges 1
606 × Eureka!I am going to try it again and send you the relevant part of the logs in a minute. Maybe I am interpreting something wrong.
Both, actually. So what I personally would find intuitive is something like this:
` class Task:
def load_statedict(self, state_dict):
pass
async def synchronize(self):
...
async def task_execute_remotely(self):
await self.synchronize()
...
def add_requirement(self, requirement):
...
@classmethod
async def init(task_name):
task = Task()
task.load_statedict(await Task.load_or_create(task_name))
await tas...
I am wondering where to put my experiment logic, so that it gets lazily executed and not at task definition time (i.e. in get_task_experiment()
how to get my experiment logic in there without running it)
I think I still don't get how clearml is supposed to work/be used. Why wouldn't the following work currently?
Example:
` task = Task.init(...)
if not running_remotely:
task_dict = task.export_task()
requirements = task_dict["script"]["requirements"]["pip"].splitlines()
requirement_torch = [r for r in requirements if r.startswith("torch==")]
requirements.remove(requirement_torch[0])
requirements.append("torch >= 1.8.1")
task_dict["script"]["requirements"]["pip"] = "\n"....
So with pipeline decorators can I implement this logic?
Maybe something like this is how it is intended to be used?
` # run_with_clearml.py
def get_main_task():
task = Task.create(project="my_project", name="my_experiment", script="main_script.py")
return task
def run_standalone(task_factory):
Task.enqueue(task_factory())
def run_in_pipeline(task_factory):
pipe = Pipelinecontroller()
pipe.add_step(preprocess, ...)
pipe.add_step(base_task_factory=task_factory, ...)
pipe.add_step(postprocess, ...)
pipe.start()
if...
Is there a way to capture uncommited changes with Task.create
just like Task.init
does? Actually, I would like to populate the repo, branch and packages automatically...
I will debug this myself a little more.
Also here is how I run my experiments right now, so I can execute them locally and remotely:
` # Initialize ClearML Task
task = (
Task.init(
project_name="examples",
task_name=args.name,
output_uri=True,
)
if track_remote or enqueue
else None
)
# Execute remotly via CLearML
if enqueue is not None and not running_remotely():
if enqueue == "None":
queue_name = None
task.reset()
...
By preexisting task I meant I have existing code that already uses Task.init
. I would like to use this code as my main task in my pipeline, i.e. after carla started.
I am also wondering how I integrate my (preexisting) main task in the pipeline. I start my main task like this: python my_script.py --myarg "myargs"
. How are the arguments captured? I am very confused, how one integrates this correctly...
To update the task.requirements before it actually creates it (the requirements are created in a background thread)
Why can't it be updated after creation?
What exactly does this mean? The environment is set after the script is started?
I just realized that I forgot again that I am using importlib and this is probably why everythings weird. I tried to reproduce the error was a smaller project and was not able to get the error again. Sorry for having wasted your time! 😐
Just tested it again. Here is my config:
https://gist.github.com/mctigger/086c5f8071a604605e9f7a172800b51d
In the Web UI under Configuration -> Hyper Parameters -> Environment
I can see the following:MUJOCO_GL osmesa
I forgot to add this:
` Here is my error:
Traceback (most recent call last):
File "src/run_gym.py", line 25, in <module>
print(os.environ["MUJOCO_GL"])
File "/home/tim/.clearml/venvs-builds/3.7/lib/python3.7/os.py", line 681, in getitem
raise KeyError(key) from None
KeyError: 'MUJOCO_GL' `
This is at the top of my script.
You can add and remove clearml-agents to/from the clearml-server anytime.
Nvm, I think its my mistake. I will investigate.
Also, is max_workers about compression threads or upload threads or both?
Tested with clearml-agent 1.0.1rc4/1.2.2 and clearml 1.3.2
Hi KindChimpanzee37 I was more asking about the general idea to make these settings task-specific, but thank you for the suggestion anyways, I will definitely apply it.
It seems like this is a bug however or is something like this to be expected? There shouldn't be files that are not shown in the WebUI..?
AgitatedDove14 Could you elaborate?
Thanks for answering, but I still do not get it. file_history_size
decides how many past files are shown? So if file_history_size=100
and I have 1 image/iteration and ran 1000 iterations, I will see images for iteration 900-1000?
I am referring to the UI. The default cleanup service should work with S3 with a correctly configured clearml service agent if I understand the workings correctly.