But I do not have anything linked correctly since I rely in conda installing cuda/cudnn for me

From the log it installed:
cudatoolkit==11.1.1
based on the CUDA it found on the host machine: agent.cuda_version = 110
But for some reason it installed the pytorch from the conda "pytorch" repo without the cuda support.

Task.debug_simulate_remote_task simulates the Task being executed by the agent (basically same behaviour, only local). the argument it gets is the Task ID (string).
The to see how it works is to run the code once (no debug_simulate call), get the Task ID we created, then rerun with the debug_simulate_remote_task passing the previous Task.ID
Make sense ?

Basically, for a bit more context, this is part of an effort to incorporate ClearML Pipelines in a CI/CD framework.

@<1523701079223570432:profile|ReassuredOwl55> did you check these examples?
None
None
None

And I’d rather the testing/validation etc lived outside...

In the Task log itself it will say the version of all the packages, basically I wonder maybe it is using an older clearml version, and this is why I cannot reproduce it..

Does this require you run the pipeline locally (I see you have set default execution queue) or do any other specific set-up?

Yes this mean the pipeline Logic runs manually/locally (logic means launching components, not actually compute)
Please have a go at it, I'm sure some quirks in the psuedo code are missing but it should work, and I'll gladly help set it up

I set up the alert rule on this metric by defining a threshold to trigger the alert. Did I understand correctly?

Yes exactly!

Or the new metric should...

basically combining the two, yes looks good.

the other repos i have are constantly worked on and changing too

Not only it will be cloned automatically, the git diff of the sub-modules are stored as well 🙂

Hi WittyOwl57
I think what happens is it auto-logs the joblib load/save calls, these calls track models used/created by the code, and attach them to the model repository representing these model.
I'm assuming there are multiple load/save , and there are multiple model instances pointing to the same local file "file:///tmp/..." . The earning basically says it is re-registering existing models.
Make sense ?

My driver says "CUDA Version: 11.2" (I am not even sure this is correct, since I do not remember installing code in this machine, but idk) and there is no pytorch for 11.2, so maybe it fallbacks to cpu?

For some reason it detect CUDA 11.1 (I assume this is what you have installed, the driver CUDA version is the highest it will support not necessary what you have installed)

Hi @<1630377234361487360:profile|RoughSeaturtle43>

code from gitlab repo with ssl cert.

what do you mean by ssl secret? is it SSH or app-token ?

correct

I think the limit is in the free tier hosting ...

yeah. I am getting logs, but they are extremely puzzling to me. I would appreciate to actually have access to whole package structure..

Actual packages are updated back to "Installed Packages" section (under the execution tab).

indeed. can you maybe point where the docker command is composed.

https://github.com/allegroai/clearml-agent/blob/178af0dee84e22becb9eec8f81f343b9f2022630/clearml_agent/commands/worker.py#L3694
🙂
BTW: you can run/build the entire thing on your machin...

Ohh I see, so basically the ASG should check if the agent is Idle, rather than the Task is running ?

Hi SmallDeer34
The clearml-agent has its own cleaml.conf file , there you should put S3 credentials and they will be passed to any Task the agent executes:
https://github.com/allegroai/clearml-agent/blob/176b4a4cdec9c4303a946a82e22a579ae22c3355/docs/clearml.conf#L234

Using agent v1.01r1 in k8s glue.

I think a fix was recently committed, let me check it

Hi MuddySquid7
You can only add reports (scalars plots etc.) , though not to a published Task.
If you want to add an artifact, this should work.
prev_task = Task.get_task(task_id='112233') prev_task.mark_started(force=True) prev_task.reload() prev_task.upload_artifact(..., wait_for_upload=True) prev_task.mark_stopped(force=True)

is there a way to increase the size of the text input for fields or a better way to handle lists?

No 😞

Maybe an easier way to use connect_configuration instead ? it will take an entire dict and store it as text (format is hocon, which is YAML/Json compatible, which means it is hard to break when editing)

WittyOwl57 I think this is a great idea, can you open a feature issue on GitHub so this is not forgotten ?
BTW: regardless, if you have time to upgrade to the new the azure package upgrade, it will be great 🙂 this is on our to do list for a while, but since not a lot of users complained it got pushed ...

YEY 🙂

SubstantialElk6
Hmm do you have torch in the "installed packages" section of the Task ?
(This what the agent is using to setup the environment inside the docker, running as a pod)

Hi SmallDeer34
Can you try with the latest RC , I think we fixed something with the jupyter/colab/vscode support
!pip install clearml==1.0.3rc1

Think multiple hyper-paremter sections that we need to reference
(under the Tasks Configuration Tab, the Hyper parameters can have multiple sections)

Is task.parent something that could help?

Exactly 🙂 something like:
# my step is running here the_pipeline_task = Task.get_task(task_id=task.parent)

Hi ComfortableHorse5
Yes this is more of a suggestion that you should write them using the platform capabilities, the UI implementation is being worked on, as well as a few helpers classes, I thin you'll be able to see a few in the next release 🙂

For future readers, see discussion here:
https://clearml.slack.com/archives/CTK20V944/p1629840257158900?thread_ts=1629091260.446400&cid=CTK20V944

--docker or in clearml.conf https://github.com/allegroai/clearml-agent/blob/21c4857795e6392a848b296ceb5480aca5f98e4b/docs/clearml.conf#L153

overrides -> "kubectl run --overrides "
template -> "kubectl apply template.yaml"

Hi SkinnyPanda43
Are you trying to access the same Task or an external one ?

Oh what if the script is in the container already?

Hmm, the idea of clearml is that the container is a "base environment" and code is "injected", this makes sure it is easy to reuse it.
The easiest way is to add an "entry point" scripts that just calls the existing script inside the container.
You can have this python initial script on your local machine then when you call clearml-task it will upload the local "entry point" script directly to the Task, and then on the remote machin...