Ok thanks, we'll try it out on next availability.

Hi AgitatedDove14 , i was refering to
task.set_base_docker("nvcr.io/nvidia/tensorflow:19.11-tf2-py3 --env TRAINS_AGENT_GIT_USER=git_username_here --env TRAINS_AGENT_GIT_PASS=git_password_here")The above will give error
skipping docker argument TRAINS_AGENT_GIT_USER=git_username_here (only -e --env supported) TRAINS_AGENT_GIT_PASS=git_username_here (only -e --env supported)

Hi CostlyOstrich36 , That's correct.

Alright thanks, i will work on that.

Can i somehow perform an export or backup?

Ok, i guess i will have to kill the whole thing and refresh it.

It's a local deployment. I was only presented with username without a need to enter passwords. When I'm in, I don't see an option in my profile to set a password as well. Neither is there integration with ldap for example.

The problem is resolved by doing a git push. Somehow the git diff didn't capture the difference in requirements.txt in the project. I can't reproduce the same issue after this as well.

Hi AgitatedDove14 , that's what i am trying to figure out as well. The task has nothing to do with torch, and the requirements.txt doesn't have any torch packages as well.

Hi, i can't seem to find the source. What are the kind of situations where it will try to install torch outside of user requirements?

Hi.

We tried as advised above and it still didn't work.
Host: http://ecs.ai:443
output_uri = S3://ecs.ai:443/bucketname

This time round the client gave this error.
Botocore.exceptions.connectiinclosederror: connection was closed before we received a valid response from endpoint URL: ' http://ecs.ai/bucketname/.clearml.test '.

It's quite apparent that whatever clearml passed to boto3 ends up as a http call instead of https, which is wrong.

Hi. If we disable the API service, how will it affect the system? How do we disable?

Hi HelpfulDeer76 , I'm facing similar issues. Would you mind describing in detail how you deploy clearml-agent? Is it running as a pod on k8s?

Hi,
basically i run this block first and ended the script.
task = Task.init(project_name="afro-nmt", task_name=args.taskname, continue_last_task=args.taskid) Logger.current_logger().report_scalar(title="BLEU",series="JW300",value=args.jwbleu, iteration=args.lastiter)Then i run another script, with series different.
` task = Task.init(project_name="afro-nmt", task_name=args.taskname, continue_last_task=args.taskid)
Logger.current_logger().report_scalar(title="BLEU",series="SS900",value=arg...

It didn't work as expected.
` task init
task report iter 10

task init
task report iter 10

The second task pushed the reporting iteration to 20 instead. `

Hi TimelyPenguin76 , i am adding a debug sample to an existing task using the above method. What should i put for the iteration? I do not want to overwrite existing ones but i do not know what's the last count. This is for both scalar and media reporting.

It would make sense on a very large resource cluster. Unfortunately we only have less than 50 GPUs to share across. A multi-tenant SAAS would cut the resources into even more smaller clusters and not help with efficiency. Or would you have a suggestion?

Ok, let me check this out first thing on Monday, thanks AgitatedDove14 .

thanks, let me try that.

Ok that works. thanks.

Can this issue be solved with vault? It doesn't make sense to expose secrets like that.

Previously we had similar issues when we switched images used in agent. Might want to check on that.

What type of pipeline steps are you running? From task, decorator or function?

We were trying with 'from task' at the moment. But the question apply to all methods.

If they're all running on the same container why not make them the same task and do things in parallel?

The tasks were created by different teams and their tasks content is rather independent and modular. Usage of them is usually optional. For example, task1 performs 'image whitening', task2 performs 'image resize'.

Thanks. This appears to be solely for web UI and API, What if i want to orchestrate on K8S?

So these (PIP_INDEX_URL) weren't used when clearml starts running pip.

Thanks TimelyPenguin76 , let me try it out now.

Hi, when i tried ip:port, it references the right host and bucket....BUT... the file is not found on the ECS S3 even though i can see from the logs that it states Completed model upload to s3://ecs.ai:80/clearml-models/artifacts/ ...

I did notice that in the tmp folder, .clearml_agent.xxxxx.cfg does not exists.

and out of curiosity, what did you think we were talking about? cos i didn't see anywhere else that might print the secrets.

Thought this looked familiar.
https://clearml.slack.com/archives/CTK20V944/p1635323823155700?thread_ts=1635323823.155700&cid=CTK20V944