Hi SubstantialElk6
ClearML-Data doesn't actually "load" the data, it brings it locally and returns a folder with all your data files, from that point onward, it's up to your code to load it to the framework. Make sense ?

Hi @<1541954607595393024:profile|BattyCrocodile47> and @<1523701225533476864:profile|ObedientDolphin41>

"we're already on AWS, why not use SageMaker?"

TBH, I've never gone through the ML workflow with SageMaker.

LOL I'm assuming this is why you are asking 🙂

• First, you can use SageMaker and still log everything to ClearML (2 lines integration). At least you will have visibility to everything that is running/failing 🙂
• SageMaker job is a container, which means for ...

Could it be it defaulted to the demo server instead of your own server?

Interesting use case, do you already have the connect_configuration in the code? or do we need to somehow create it ?

That wasn't scheduled by ClearML).

This means that from Clearml perspective they are "manual" i.e the job it self (by calling Task.init) create the experiment in the system, and fills in all the fields.

But for a k8s job, I'm still unsuccessful.

HelpfulDeer76 When you say "unsuccessful" what exactly do you mean ?
Could it be they are reported to the clearml demo server (the default server if no configuration is found) ?

Interesting...
We could followup the .env configuration, and allow the clearml-task to add configuration files from cmd line. This will be relatively easy to add. We could expand the Environment support (that somewhat exists), and add the ability to read variables from .emv and Add them to an "hyperparemeter" section, named Environment. wdyt?

That makes sense to me, what do you think about the following:
` from clearml import PipelineDecorator

class AbstractPipeline(object):
def init():
pass

@PipelineDecorator.pipeline(...)
def run(self, run_arg):
data = self.step1(run_arg)
final_model = self.step2(data)
self.upload_model(final_model)

@PipelineDecorator.component(...)
def step1(self, arg_a):
# do something
return value

@PipelineDecorator.component(...)
def step2(self, arg_b):
# do ...

re-running this code produces the same printoutsJust to be clear, you are saying the "random" results are consistent over runs ?

If I don't specify the type for N in the component I get an error because N is interpreted as a string.

Yes the default value is used for proper casting, In the next version we will use the type hints for that as well 🙂
If I un-comment the last two lines and rerun this script, the second pipeline call results in an error:I think that If you need multiple p...

is some explanation of how functions become pipelines and components.

That is a good point! I'll make sure we mention it in the pipeline section of the docs

This whole experiment with random numbers started as my attempt at verifying that code in clearml.pipeline

Correct 🙂 BTW: this is why we added PipelineDecorator.run_locally() so it is easier to debug, you can also use PipelineDecorator.debug_pipeline() to run the entire pipeline in a single process as python ...

The second run prints out the same (non) "random" numbers as the first run

ClearML sets the initial random seed for you, basically trying to help with reproducibility. That said inside the function you can always do:
import random import time random.seed(time.time())

do you have a video showing the use case for clearml-session

I totally think we should, I'll pass it along 🙂

what is the difference between vscode via clearml-session and vscode via remote ssh extension ?

Nice! remote vscode is usually thought of as SSH, basically you have your vscode running on your machine, and using SSH vscode automatically connects to the remote machine.

Clearml-Session also ads a new capability VSCode inside your browser, where the VSCode itself as well...

Yeah I think using voxel for forensics makes sense. What's your use case ?

There are also "completed, aborted, queued" .
Archived is actually a tag (system tag, not user tag). There is a "state machines" of moving from one state to the other. The special case is "published" that we probably should have called "locked". The idea is that if a Task/Model is published, you cannot reset it (and even deleting requires force flag).
I would use additional user tags (or even system-tags) to mark "deployed" state, wdyt?

Yes 🙂 documentation is being worked on ... Anyhow we will be uploading a new documentation site soon (hopefully in a week or so), putting it all on GitHub so it will be easier for the community to edit and add more

LOL AlertBlackbird30 had a PR and pulled it 🙂
Major release due next week after that we will put a a roadmap on the main GitHub page.
Anything specific you have in mind ?

I would recommend reading this blog post, it should give you a glimpse of what can be built 🙂
https://medium.com/pytorch/how-trigo-built-a-scalable-ai-development-deployment-pipeline-for-frictionless-retail-b583d25d0dd

WackyRabbit7 basically yes 🙂

First let's verify with the manual change, but yes

ReassuredTiger98 there is an open issue on supporting bash script as pre run inside a docker (which will be supported in the next major release)
BTW: if you already have a docker file the fastest way would just to build the docker file and push it once, then you just specify the docker image:tag, this can be done a Task specific level.

Hi JumpyDragonfly13

1. is "10.19.20.15" accessible from your machine (i.e. can you ping to it)?
2. Can you manually SSH to 10.19.20.15 on port 10022 ?

What do you mean by "modules first and find a way to install that package" ?
Are those modules already in wheels ? are they part a git repository?
(the pipeline component can also start inside a git repository it clones)

Programmatically before , importing the package, set os.environ['TRAINS_CONFIG_FILE']='~/my_new_trains.conf'
BTW: What's the use case for doing so?

thanks for helping again

My pleasure :)

The problem is, the configuration is loaded at import time, so there is no "time" to pass anything other than environment variable.
That said if the only difference is server config you can use
Task.set_credentials

Great ascii tree 🙂
GrittyKangaroo27 assuming you are doing:
@PipelineDecorator.component(..., repo='.') def my_component(): ...The function my_component will be running in the repository root, so in thoery it could access the packages 1/2
(I'm assuming here directory "project" is the repository root)
Does that make sense ?
BTW: when you pass repo='.' to @PipelineDecorator.component it takes the current repository that exists on the local machine running the pipel...

Hi SlipperyDove40
plotly is about 4Mb... trains about 0.5MB what'd the breakdown of the packages ? This seems far away from 250Mb limit

SlipperyDove40 I just installed a fresh copy py3.6 and plotly on ubuntu. the entire venv dir is ~86MB

Hi @<1730033904972206080:profile|FantasticSeaurchin8>
Is this only relates to this
https://github.com/coqui-ai/Trainer/issues/7
Or is it a clearml sdk issue?

https://allegro.ai/docs/webapp/webapp_project/#customizing-the-experiments-table

Hmm @<1523701083040387072:profile|UnevenDolphin73> I think this is the reason, None
and this means that even without a full lock file poetry can still build an environment

Regrading check, we can add one here
https://github.com/allegroai/clearml/blob/12fa7c92aaf8770d770c8ed05094e924b9099c16/clearml/storage/helper.py#L713