Sure, is it reproducible ?

Runtime, every time the add_step needs to create a New Task to be enqueued

Are you aware of any other way then (other than the 

secure: false

 flag?

Actually self -signing and providing certificate file is already supported with boto (and thus clearml)
AWS_CA_BUNDLE
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

I'm going to follow your suggestion and just put the extra effort into distributing a pre-built image.

That sounds good 🙂
If you feel the need is important, I do have a hack in mind, it will be doable once we have support for entrypoint "-c python_code_here". But since this is still not available I believe you are right and build an image would be the easiest.

A note on the docker image, remember that when running inside the docker we inherit the system packages installed on the d...

Hi @<1619867971730018304:profile|WhimsicalGorilla67>
No 😞 only the "admin" (owner) of the workspace has access to it

As long as the ~/.aws is configured, I "think" it should work. (I'm assuming you are referring IAM roles?)
I would also suggest using the latest aws_autoscaler (basically it adds a CLI wizard, I think the functionality is very much the same)

Hi DepressedFox45
Basically move the import into the function, it will automatically detect the package.
@PipelineDecorator.component(...) def step_one(...): import sklearn import pandas as pd # stuffMake sense ?

The file itslef is csv.gz compressed, it's actually sending from the file-server back that messes things
(you can test with output_uri=/tmp/folder )

right now I can't figure out how to get the session in order to get the notebook path

you mean the code that fires "HTTPConnectionPool" ?

Hmm it should probably be pinned

Yey @ https://app.slack.com/team/U01CJ43KX2N this one does not work!
Give me a minute I'll

yep 🙂

https://github.com/allegroai/clearml/commit/56825f9e7af26a43da0f2f12454e23a43e995a25

I am symlinking the .clearml directory to a NAS server and this is perhaps part of the problem.

Yep, that sounds about right, it uses Posix file system for internal lock mechanisms (multi process locks), and my guess is that the NAS for some reason does not support it...

Thanks IntriguedRat44 !
I'll follow up on GitHub 🙂

And having a pdf is easier/better than sharing a link to the results page ?

https://github.com/allegroai/clearml/commit/737ca91d2a6f42b5f1ef19815fac93e183ba5398

task = Task.get_task('task_id_here') task.mark_started(force=True) task.upload_artifact(..., wait_on_upload=True) task.mark_completed()

Hi @<1546665638829756416:profile|LovelyChimpanzee39>

anyone know what params I need to pass in order to enable it?

we feel you there 😞 this is an actual plotly feature that we really want to add, but kind of out of our hands: None
feel free to support as there 🤞

: For artifacts already registered, returns simply the entry and for artifacts not existing, contact server to retrieve them

This is the current state.
Downloading the artifacts is done only when actually calling get()/get_local_copy()

That's the theory, I still see it is not there

p.s. StraightCoral86 I might be missing something here, please feel free to describe the entire execution scenario and what you are trying to achieve 🙂

Okay that seems to explain it. Now the question is why it installed it in the wrong place.

It's the same but done from outside, you want the same and "offline" as well right?

Hi MortifiedCrow63
I finally got GS credentials, there is something weird going on. I can verify the issue, with model upload I get timeout error while upload_artifacts just works.
Just updating here that we are looking into it.

Hi TightElk12
One option will be to call task.close() at the end of each step and task.init at the beginning of another.
Will that do?

Hi @<1547028031053238272:profile|MassiveGoldfish6>
hmm yeah you need to remove the "hidden" system_tag from the project

from clearml.backend_api.session.client import APIClient

c = APIClient()

print(c.projects.get_by_id("PROJECT_ID_HERE").to_dict())

c.projects.update(project="PROJECT_ID_HERE", system_tags=["test"])

print(c.projects.get_by_id("PROJECT_ID_HERE").to_dict())

Notice you can get the project ID from the URL
`/projects/1974af8ccdac454b836c47349c4e826e/experiments/84...

sdk.storage.cache.size.cleanup_margin_percent

Hi ReassuredTiger98
This is actually future proofing the cache mechanism and allowing it be "smarter" i.e. clean based on cache folder size instead of cache folder entries, this is currently not available

sdk.storage.cache

 parameters for the agent?

For both local execution and with an agent

When are datasets deleted if I run local execution?

When you hit the cache entry limit (100 if I recall). This can a...

Hi SubstantialElk6

but in terms of data provenance, its not clear how i can associate the data versions with the processes that created it.

I think DeliciousBluewhale87 ’s approach is what we are aiming for, but with code.
So using clearml-data from CLI is basically storing/versioning of files (with differentiable based storage etc, but still).
What ou are after (I think) is in your preprocessing code using the programtic Dataset class, to create the Dataset from code, this a...

Thanks GrievingTurkey78
Sure just PR (should work with any Python/Hydra version):
kwargs['config']=config kwargs['task_function']=partial(PatchHydra._patched_task_function, task_function,) result = PatchHydra._original_run_job(*args, **kwargs)