and they don't know how to write code, is this still possible?

well this means there is some standard of the data, right? what is that standard? unfortunately in our space there is no standard fort data, it's just too generic, so everyone always end with custom parsing of a sort.
Does that make sense ?

well cudnn is actually missing from the base image...

Hi @<1691258563357315072:profile|ColorfulKitten60>
I think we need some context for this question 🙂

Do you think this is better ? (the API documentation is coming directly from the python doc-string, so the code will always have the latest documentation)
https://github.com/allegroai/clearml/blob/c58e8a4c6a1294f8acec6ed9cba81c3b91aa2abd/clearml/datasets/dataset.py#L633

If you set the TMP env variable you can control the tmp folder. Would that work?

and curl http://localhost:8080 works?

Hi @<1597399925723762688:profile|IrritableStork32>
I think that if you have clearml installed an configured on your machine it should just work:
None

, I need to understand it what happens when I press "Enqueue" In web UI and set it to default queue

The Task ID is pushed into the execution queue (from the UI / backend that is it), Then you have clearml-agent running on Your machine, the agent listens on queue/s and pulls jobs from queue.
It will pull the Task ID from the queue, setup the environment according to the Task (i.e. either inside a docker container or in a new virtual-env), clone the code/apply uncommitted changes ...

IrateBee40
Check the first steps here:
https://clear.ml/docs/latest/docs/getting_started/ds/ds_first_steps
(Basically you have to generate credentials / configure you machine so it knows where the server is and how to access it)

Make sense ?

Thanks GreasyPenguin14 !

Ohh I see, could you copy paste what you put there (instead of the secret and key *** will do 🙂 )

ElegantCoyote26
parser = get_parser() args_ = vars(parser.parse_args()) task.connect(args_)There is no need to connect args_ Task.init will automatically catch the argparser.

FYI:
ssh -R 8080:localhost:8080 -R 8008:localhost:8008 -R 8081:localhost:8081 replace_with_username@ubuntu_ip_heresolved the issue 🙂

EnviousStarfish54 whats your matplotlib version ?

😞 anything that can be done?

Martin, thank you very much for your time and dedication, I really appreciate it

My pleasure 🙂

Yes, I have latest 1.0.5 version now and it gives same result in UI as previous version that I used

Hmm are you saying the auto hydra connection doesn't work ? is it the folder structure ?
When is the Task.init is called ?
See example here:
https://github.com/allegroai/clearml/blob/master/examples/frameworks/hydra/hydra_example.py

Hi @<1727497172041076736:profile|TightSheep99>
Yes it can, it will upload the meta-data as well as the files (it will also do de-dup and will not upload files that already exist in the dataset based on the hash of teh file content)

(with matplotlib 3.2+ I get no warning, let me check with 3.1)

but instead, they cannot be run if the files they produce, were not committed.

The thing with git, if you have new files and you did not add them, they will not appear in the git diff, hence missing when running from the agent. Does that sound like your case?

How do I reproduce it ? (all the processes are on the same machine?)

Hi @<1523701079223570432:profile|ReassuredOwl55>

I want to kick off the pipeline and then check completion

outside

of the pipeline task. (edited)

Basically the pipeline is a Task (of a certain type).
You do the "standard" thing, you clone the pipeline Task, you enqueue it, and you wait for it's status

task = Task.clone(source_task="<pipeline ID here>")
Task.enqueue(task, queue_name=services)
task.wait_for_status(...)

wdyt?

SparklingHedgehong28 this is actually quite cool! Still not sure why not just use the built in autoscaler https://github.com/allegroai/clearml/tree/master/examples/services/aws-autoscaler , but it is a really cool usage of ASG 🤩

So are you saying why do we need to install a specific pip version ?
You can "disable it" by selecting a very high version
pip_version: "<40"https://github.com/allegroai/clearml-agent/blob/077148be00ead21084d63a14bf89d13d049cf7db/docs/clearml.conf#L67

Hmm let me check something

Hi @<1658281093108862976:profile|EncouragingPenguin15>
Should work, I'm assuming multiple nodes are running agents ? or are you saying Ray spins the jobs and clearml logs them ?

Hi UnsightlyShark53 apologies for this delayed reply, slack doesn't alert users unless you add @ , so things sometimes get lost :(
I think you pointed at the correct culprit...
Did you manage to overcome the circular include?
BTW , how could I reproduce it? It will be nice if we could solve it

As long as you import clearml on the main script, it should work. Regarding the Nvidia container, it should not interfere with any running processes, the only issue is memory limit. BTW any reason not to spin an agent on a dedicated machine? What is the gpu used for in the ckearml server machine?

I don't think so. it is solved by installing openssh-client to the docker image or by adding deploy token to the cloning url in web ui

You can also have the token (token==password) configured as the defauylt user/pass in your agent's clearml.conf
https://github.com/allegroai/clearml-agent/blob/73625bf00fc7b4506554c1df9abd393b49b2a8ed/docs/clearml.conf#L19

EnviousStarfish54 thanks again for the reproducible code, it seems this is a Web UI bug, I'll keep you updated.

Hmm so if I understand what's going on, convert_test.py needs to have the test.json , since it creates the test.json but it does not call git add on it, the test.json will not be part of the git diff hence missing when executing remotely by the agent.
If test.json is relatively small (i.e. not 10s of MB) you could store it as configuration on the Task. for example:
` local_copy_of_test_json = task.connect_configuration('/path/to/test.json', name='test config')
print(...