woot woot, glad to hear that!

Lately I've heard of groups that do slices of datasets for distributed training, or who "stream" data.

Hmm so maybe a "glob" alike parameter for get_local_copy(select_filter='subfolder/*') ?

https://github.com/huggingface/transformers/blob/040283170cd559b59b8eb37fe9fe8e99ff7edcbc/src/transformers/trainer_pt_utils.py#L954
specifically called here:
https://github.com/huggingface/transformers/blob/040283170cd559b59b8eb37fe9fe8e99ff7edcbc/examples/pytorch/language-modeling/run_mlm.py#L480
Maybe after this line add:
Task.current_task().upload_artifact('trainer_state.json, name='state') `wdyt?

CLI? Code ?

Okay, so I think it doesn't find the correct Task, otherwise it wouldn't print the warning,
How do you setup the HPO class ? Could you copy paste the code?

Yes, the same will work with artifacts, use pass the full url to the artifact_object it should just register it as is.

How do I reproduce it ? (all the processes are on the same machine?)

JuicyDog96 Yes please!
Let me check what's the status with the docs repository, and I'll get back to you soon 🙂

It should work 🙂 as long as the versions match, if they don't the venv will install the version you need (which is great, only penalty is the install, download wise it will be cached)

I think so (you can also comment out the Task.init() just to verify this is not a clearml issue)

Seems like passing the Task object is not working as expected (I'll make sure it is fixed).
Try:
dataset._task.set_parent(Task.current_task().id)

The other order (with custom decorator above pipeline fails - just for you info

)

This is on "purpose" the pipeline decorator has to be the top decorator.
Glad it works!

Hi @<1523702868694011904:profile|AbruptCow41>
Check what are you getting when running git status inside the working directory, this is essentially how it works. Are you expecting to later run it with an agent?

Should work in all cases, plotly/matplotlib/scalar_rerport

Exactly, that’s my problem: I want to remove it to make sure it is reinstalled (because the version can change)

JitteryCoyote63 yes, this is definitely a pip bug... can you test with the latest pip version, maybe it was fixed? (i.e. git+https:// link)

MelancholyChicken65 found it ! thank you for finding this issue.
I'm hoping to get an update soon 🙂

Hi TrickyFox41

Hey since Hydra does not work with

clearml-task

I should shouldn't it? what does not work ?

The other way will not work, as if you start with "pip" you cannot fail ... (if you fail it's in run time which is too late)

Hmmm maybe 

 I thought that was expected behavior from poetry side actually

I think this is the expected behavior, hence bug?!

Alternatively I understand I can also run the agent using...

No you should not if you are running the agent inside a container it cannot work in docker mode and spin its own containers
Bottom line use clearml-agent daemon

Hi SuperiorDucks36

you have such a great and clear GUI

😊

I personally would love to do it with a CLI

Actually a lot of stuff are harder to get from UI (like current state of your local repository etc.) But I think your point stands 🙂 We will start with CLI, because it is faster to deploy/iterate, then when you guys say this is a winner we will have a wizard in the UI.
What do you think?

what does mount -a return ?

Is it also possible to specify different user/api_token for different hosts? For example I have a github and a private gitlab that I both want to be able to access.

ReassuredTiger98 my apologies I just realize you can use ~/.git-credentials for that. The agent will automatically map the host .git-credentials into the docker :)

Does this file look familiar to you?
file not found: archive/constants.pkl

Yes exactly like a Task (pipeline is a type of task)
'''
clonedpipeline=Task.clone(pipeline_uid_here)
Task.enqueue(...)
'''

For running the pipeline remotely I want the path to be like /Users/adityachaudhry/.clearml/cache/......

I'm not sure I follow, if you are getting a path with all your folders from get_local_copy , that's exactly what you are looking for, no?

@<1523705099182936064:profile|GrievingDeer61> if you see them in your Tensorboard they will be captured in ClearML as well.
wdyt?

EDIT
examples (tensorboard or tensorboardX):
None
None

Thanks!

Hi @<1610083503607648256:profile|DiminutiveToad80>
You mean the pipeline logic? It should autodetect the imports of the logic function (like any Task.init call)
You can however call Task.force_requirements_env_freeze and pass a local requiremenst.txt
Make sure to call it before create the Pipeline object
None

Thank you!