Here’s another place where /root/ is hardcoded https://github.com/allegroai/clearml-agent/blob/b196ab57931f3c67efcb561df0c8a2fe7c0e76f9/clearml_agent/commands/worker.py#L3338-L3341

That looks great!

Great! Thank you

I’m not sure but it seems like you get different kinds of flexibility depending on whether you enqueue the task yourself or whether you rely on execute_remotely . I think ideally if I could choose to get the benefit of auto-scanning provided by execute_remotely as well as more flexibility it would be great.

Is it possible to set that at task enqueueing SuccessfulKoala55 ?

For your second question, those are generated using custom tooling, it relies on the build system to be setup which is guaranteed by the docker image used. So I don’t think this is a case of supporting a specific env setup or build tool but just allowing custom script for env setup step / building code.
WDYT?

If venv works inside containers that’s even better. We actually have custom containers that build on master merges. I wonder if using our own containers which should have most the deps will work better than a simpler container.

I’ll try that!

OH! I was installing it on an env

AgitatedDove14 it was executed with Python 3 and I’m running in venv mode.

I do expect it to pip install though which doesn’t root access I think

EagerOtter28 I’m running into a similar situation as you.

I think you could use --standalone-mode and do the cloning yourself in the docker bash script that you can configure in the agent config.

Being able to create and remove queues as well as list their contents.

AgitatedDove14 when I try this I get
clearml.backend_interface.session.SendError: Action failed <400/110: tasks.enqueue/v1.0 (Invalid task status (Invalid status change): current_status=in_progress, new_status=queued)> (queue=e78d2fdf2d5140b6b5c6678338c532bb, task=95082c9174a04044b25253d724362ec1)

This is exactly what I was looking for. I thought once you call execute_remotely the task is sent and it’s too late to change anything.

I know this is not the default behavior so I’d be happy with having the option to override the repo when I call execute_remotely

Fixed it by adding this code block. Makes sense.
if clone: task = Task.clone(self) else: task = self # check if the server supports enqueueing aborted/stopped Tasks if Session.check_min_api_server_version('2.13'): self.mark_stopped(force=True) else: self.reset()

Yes

$ git remote -v fork git@github.com:salimmj/somerepo.git (fetch) fork git@github.com:salimmj/somerepo.git (push) origin git@github.com:mainuser/somerepo.git (fetch) origin git@github.com:mainuser/somerepo.git (push)I want to keep the above setup, the remote branch that will track my local will be on fork so it needs to pull from there. Currently it recognizes origin so it doesn’t work because the agent then can’t find the commit.

I already have that set to true and want that behavior. The issue is on the “committed” change set. When I push code to github I push to my fork and pull from the main/master repo (all changes go through PRs from fork to main).

Now when I use execute_remotely , whatever code does the git discovery, considers whatever repo I pull from the repo to use. But these changes haven’t necessarily been merged into main. The correct behavior would be to use the forked repo.

It is indeed autopopulated by init

The commit is valid for sure.

Is there a way to make it use ssh+git instead of git+git ? Maybe add a force_ssh_pip_install to the agent config?

Ok!

I think it works, I’m fixing something else that came up.

AgitatedDove14 can I specify a script to be run after pip install packages is done? I see that it’s possible in docker mode.

I hadn’t enabled that line when the failure happened.

The private_package can be installed by doing pip install git+ssh://git@github.com/user/private_package.git but the agent is trying to do pip install private_package which won’t work.

Our code is tightly integrated with protobuffers which needs to be re-compiled every now and then. We have a script to do that. If that’s not done, some imports end up failing.

... more-itertools==8.6.0 -e git+git@github.com:user/private_package.git@57f382f51d124299788544b3e7afa11c4cba2d1f#egg=private_package msgpack==1.0.2 msgpack-numpy==0.4.7.1 ...