Was going crazy for a short amount of time yelling to myself: I just installed clear-agent init!

oh noooooooooooooooooo
I can relate so much, happens to me too often that copy pasting into bash just uses the unicode character instead of the regular ascii one
I'll let the front-end guys know, so we do not make ppl go crazy 😉

Hi ShinyWhale52
Luigi's approach is basically an extension of a functional dag, where each node is a single function. Let's think of Kedro as extension of this approach.
With both the assumption is that a node is a single function (sometimes it really is) and we just want to create a meta execution path (i.e. the execution dag, quite similar to TF v1).
ClearML pipelines are a different story (in a way).
The main difference is that with ClearML each node is a Task, not a function. That mean...

it knows it’s a notebook and automatically adds the notebook as an artifact right?

correct

and the uncommited changes becomes the nottebook converted to a script?

correct

In one case I am seeing actual git diff coming in instead of the notebook.

it might be there is both a git repository and a notebook and the git diff will show before the notebook is detected and shown instead ? (there is a watchdog refreshing the notebook every 30sec or so)

remote repository

's lock file.

Which file is that? the poetry lock of the internal VCS lock (the agent itself)

The driver script (the one initializes models and initializes a training sequence) was not at git repo and besides that one, everything is.

Yes there is an issue when you have both git repo and totally uncommitted file, since clearml can store either standalone script or a git repository, the mix of the two is not actually supported. Does that make sense ?

RoughTiger69 how did you end up with a Task with just "origin" in the repo field ?

Hi JumpyDragonfly13 , just making sure, do you have an agent running on a remote machine ?
Can you have a direct TCP connection to the remote machine (the default port it will use is 10022)

JitteryCoyote63 that makes total sense!!
The reporting subprocess is not being updated with the new value! Let me check how we can pass it along...

Hopefully once things calm down at work I will find more time.

Sounds good 🙂

Ephemeral Dataset, I like that! Is this like splitting a dataset for example, then training/testing, when done deleting. Making sure the entire pipeline is reproducible, but without storing the data long term?

IdealPanda97 Hmm I see...
Well, unfortunately, Trains is all about free access to all 🙂
That said, the Enterprise edition does add permissions and data management on top of Trains. You can get in touch through the https://allegro.ai/enterprise/#contact , I'm sure someone will get back to you soon.

is it possible to perform debugging operations with pycharm integration using remote session?

Sure, use clearml-session it will open an ssh connection to the remote machine, then you can use pycharm

and I've made a script to edit it to our needs as part of the installation process 

 Thanks Martin!

My pleasure, btw: there is no actual need to configure all the clearml.conf values. It will actually take the defaults from the clearml package itself. This means you only need something like:
` api {

server config here

}
sdk.aws.s3{

minio config here

} `

Hi SubstantialElk6
You are uploading an artifact, a good use case for numpy artifact would be a feature table.
If you want to upload an image use either report_media or report_image or upload PIL image as artifact.
What do you think?

Okay, let me see...

HarebrainedBear62 this is what I have.
clearml-data will store all the files for you, and version the entire thing, make is a breeze to abstract the dataset from the code. Querying data is available using Apache Drill (though currently it is still not built into the platform, but we are planning to get there soon) Since this is Image based data/meta-data, I know the paid tier of ClearML, has n additional dedicated data management solution specifically for images, with full ability to query m...

could be nice to have a direct "task comparison" link in the UI somewhere,

you mean like a "cart" for comparison ? or just to "save the state" so you can move between projects ?

JitteryCoyote63 in the UI what's the value of "config" ? Is it empty, it a string?
Also, could you check if removing the 'type=str' from the add_argument changes the behavior?

MuddySquid7 I might have found something, and this is very very odd, it seems it will Not upload any new images post the history size, which is very odd considering the number of users actively using this feature...
Do you want to try a hack to see if it solved your issue ?

Hi ElegantKangaroo44 ,
This is basically the number of average number of experiments running, and the number of projects, and number of users. I think this is about it. nothing like google-analytics stuff. It is mainly aimed at giving some idea on how large is the usage. Sounds reasonable?

Hi AgitatedTurtle16

My question is how to use it to manage my experiments (docker containers). Simply put, let's say:

So basically once you see an experiment in the UI, it means you can launch it on an agent.
There is No need to containerize your experiment (actually that's kind of the idea, removing the need to always containerize everything).
The agent will clone the code, apply uncommitted changes & install the packages in the base-container-image at runtime.
This allows you to u...

Yes, the webserver doesn't know where the api server is, it will access /api and then the nginx running the webapp will do the routing (reverse proxy)
I think that for some reason it is failing to do that (actually similarly to the stackoverflow you linked)

FYI all the git pulls are cached even in docker mode so there is no "tax" to pay for pulling the sub-modules (only the first time of course)

No worries 🙂

Hi @<1603198134261911552:profile|ColossalReindeer77>

Hello! does anyone know how to do

HPO

when your parameters are in a

Hydra

Basically hydra parameters are overridden with "Hydra/param"
(this is equivalent to the "override" option of hydra in CLI)

And what is exactly missing from the "installed packages" ? Is "help_models" an additional wheel you have to install ?
Just making sure here, but remember that if your original code did not have a git repo, the only thing that is "copied" to the trains-server is the initial script, so any accompanying scripts will be missing in the trains-agent environment

Can i log new lines to an old dataframe plot? any other suggestions?

Hi ChubbyLouse32
you mean to an already reported Table? or an artifact ? or a dataset ?

Three options:
In your code: Task.init(..., output_uri='s3://.../'2. Configure a default output_uri to be used by all tasks: https://github.com/allegroai/clearml/blob/64042f6c4fdaaf15b6c5f816f2fbf50f89c313e2/docs/clearml.conf#L156
3. In the UI after you clone a Task under Execution tab, "output" "destination"

In all cases output_uri can be:
/mnt/share/folder (if you have a shared folder between all machines. http://trains-server:8081/ gs://bucket azure://bucket/

"what's the trains/trains-agent/trains-server versions ?" how can I check it?

trains/trains-agent are pip packages os,
pip freeze | grep trains
trains-server you can check in the /profile page top left corner

Hmm maybe this is the issue, :

Conda error: UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (cudatoolkit):

  - pytorch~=1.8.0 -> cudatoolkit[version='>=10.1,<10.2|>=10.2,<10.3']

This makes no sense, conda is saying pytorch=1.8 needs cudatoolkit <10.2/10.3 but actually it needs cudatoolkit 11.1