Ah, that makes sense. What is supposed to be hidden changes depending on the section your in, which makes sense. Now there needs to a packman sprite easter egg hidden somewhere else.

Thanks again for the info. I might experiment with it to see first hand what the advantages are.

Oh, I get what's happening. That segment of the code is rerun when the task is enqueued remotely. So it's deleting itself. This also explains why it works fine locally. It's an ouroboros, the task is deleting itself.

Ah, I think I see the issue. In my head I was crossing ID with URL.

Yes.

In this case it's the ID of the "output" model from the first task.

On the original model.

I figured as much. This is basically what I was planning to do otherwise. I have questions around that.

• It appears that the 'extra' config is displayed in plain text on the web app and downloadable in json. I was just curious if this is best practices.
• I noticed in the AWS instance that's spun up when starting the autoscaler there's 3 settings in the config: use_credentials_chain: false, use_iam_instance_profile: false, use_owner_token: False are these strictly for the credentials t...

If I wanted to do this with the ID, how would I approach it?

I'm using pro. Sorry, for the delay, I didn't notice I never sent the response.

I'd like to provide the credentials to any ec2 instances that are spun up.

After some digging we found it was actually caused by the routers IPS protection. I thought it would be strange for github to be throttling things at this scale.

Since this could happen with a lot of services, maybe it would be worth a retry option? Especially if it's part of a pipeline.

It's a corporate one. We are also looking into options on Github's end.

Project 2:

2024-01-22 17:21:56
task 6518c3cd13394aa4abbc8f0dc34eb763 pulled from 8a69a982f5824762aeac7b000fbf2161 by worker bigbrother:10
2024-01-22 17:22:03
Current configuration (clearml_agent v1.7.0, location: /tmp/.clearml_agent.bojpliyx.cfg):
----------------------
agent.worker_id = bigbrother:10
agent.worker_name = bigbrother
agent.force_git_ssh_protocol = true
agent.python_binary = /home/natephysics/anaconda3/bin/python
agent.package_manager.type = pip
agent.package_manager.pip_v...

Hi Jake 👍 ,

Maybe the content is cached? The repo isn't big. I didn't realize the log was missing content. I believe I copied everything but I'll double check in a moment.

Actually, clearing the cache on the other project might have fixed it. I just tested it out and it seems to be working.

Looks good 😄

I'm not sure why the logs were incomplete. I think part of the reason it wasn't pulling from the repo was that it was pulling from cache. I cleared the clearml cache for that project and reran it. This should be the full log.

Unfortunately, that doesn't seem to have solved the problem. I tried the same thing with https and it seems to skip the lines with the @ symbol like it did before. Honestly, it seems more like it just isn't parsing those lines during the install.

Collecting darts==0.25.0
  Using cached darts-0.25.0-py3-none-any.whl (760 kB)
Collecting lightgbm
  Using cached lightgbm-4.1.0-py3-none-manylinux_2_28_x86_64.whl (3.1 MB)
Collecting prophet
  Using cached prophet-1.1.4-py3-none-manylinux_2_1...

Sorry I disappeared (went on a well deserved vacation). The problem is happening because of the ordering of the install. If I install using pip install -r ./requirements.txt then pip installs the packages in the order of the requirements file. However, during the installation process from ClearML, it installs the packages in order UNLESS there's a custom path provided, then it's saved for last. The reason this breaks my code is I have later packages that depend on the custom packages, as ...

Actually this is not how it works, pip will install in any way it sees fit, and it is not consistent between versions (it has to do with dependency resolving)

Oh I see. What a pain. 🤣

You can configure the agent to first install specific packages, and only then others, just add the package names here:

That's an interesting solution. I'll keep that in mind as I work more with ClearML.

Thanks for your help Martin!

It's even attempting to install omegaconf but not from the repo, likely because it's a dependency of hydra-colorlog.

Collecting omegaconf<2.4,>=2.2
  Using cached omegaconf-2.2.3-py3-none-any.whl (79 kB)
  Using cached omegaconf-2.2.2-py3-none-any.whl (79 kB)
  Using cached omegaconf-2.2.1-py3-none-any.whl (78 kB)

The git credentials are stored in the agent config and they work when I tested them on another project (not for setting up the environment but for downloading the repo of the task itself.)

@<1523701087100473344:profile|SuccessfulKoala55> You wouldn't happen to know what's going on here. :D

It's verbatim from requirements as I pass that into ClearML.

Oh, duh. I'll test that out. But I did have the agent.force_git_ssh_protocol: true

Thanks Martin. I read this method as "getting the data associated with the model training" not "getting metadata for the model". This is what I'm looking for.

Why? That's not how I authenticate. Also, if it was simply an issue with authentication wouldn't there be some error message in the log?

@<1523701070390366208:profile|CostlyOstrich36> Just pinging you 😄