Pycharm does get confused sometimes

This is what I just used:
` import os
from argparse import ArgumentParser

from tensorflow.keras import utils as np_utils
from tensorflow.keras.datasets import mnist
from tensorflow.keras.layers import Activation, Dense, Softmax
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import ModelCheckpoint

from clearml import Task

parser = ArgumentParser()
parser.add_argument('--output-uri', type=str, required=False)
args =...

Hi JumpyPig73 , I think it was synced to github. You can already test with: git install git+ https://github.com/allegroai/clearml.git

OutrageousSheep60 so if this is the case I think you need to add "external links" i.e. upload the individual files to GCS, then register the links to GCS, does that make sense ?

If this is the case why not have the stream process call the rest api, then move forward with the result? This way it scales out of the box, the main "conceptual" difference is that the restapi is used internally, and the upside is the event streaming processing becomes part of the application layer, not tied with the compute cost of the model , wdyt?

PungentLouse55 I'm checking something here, you might stumbled on a bug in parameter overriding. Updating here soon ...

Okay good news, there is a fix, bad news, sync to GitHub will only be tomorrow

Hi ScantChimpanzee51
In order to get it to work:
conf_file = "options.yml" conf_file = task.connect_configuration(conf_file, "Yaml options") with open(conf_file, "r") as f: ...The reason is it will not overwrite the local file but return a temp file for you to read.
And come to think about it, maybe we should add an argument saying, it should allow it to overwrite the local file, wdyt?

https://github.com/allegroai/clearml/commit/56825f9e7af26a43da0f2f12454e23a43e995a25

Did you meant 

--detached

 ?

Oops yes sorry you are correct should be --detached 🙂

Hi @<1838387863251587072:profile|JealousCrocodile85>
I'm assuming this is with clearml-serving, notice that it cannot install the correct scikit learn package per endpoint, you have to specify it in the docker compose or k8s helm, see example here
Example
[https://github.com/clearml/clearml-serving/blob/5c7077537ad46439f864f24e99e2ea5d4d5b35b3/docker/docker-compose.yml#L103](https://github.com/clearml/clearml-serving/blob/5c7077537ad46439f864f24e99e2ea5d4d5b35b3/docker/docker-compose.yml#...

Woo, what a doozy.

yeah those "broken" pip versions are making our life hard ...

What's strange is that the remote jobs, as soon as they are launched, if I compare their configs while in state pending, they have the right all different configs, but later, while running,

Wait I think I found it, since usuallyu the case with hydra you configure everything from overrides / config, when launched remotely it looks at it by default. But with the launch plugin it should be overwritten with the Task
` task = Task.init(...)
task.set_parameter(name="Hydra/_allow_omegaconf_ed...

Hi StrangePelican34
What exactly I not working? Are you getting any TB reports?

How come the second one is one line?

i’m working on creating a custom config with istio

That is awesome! let me know if we could help 🙂
Also please consider PRing it, I'm sure other users will appreciate the option

im helping train my friend

on clearml to assist with his astrophysics research,

if that's the case, what you can do is use the agent inside your sbatch script,
(full open source). This means the sbatch becomes " clearml-agent execute --id <task_id_here> " this will set up the environment and monitor the job and still allow you to launch it from slurm, wdyt?

Would this be equivalent to an automated job submission from clearml to the cluster?

yes exactly

I am looking for a setup which allows me to essentially create the workers and start the tasks from a slurm script

hmm I see, basically the slurm Admins are afraid you will create a script the clogs the SLURM cluster, hence no automated job submission, so you want to use slurm as a "time on cluster" and then when your time is allocated, use clearml for the job submission, is that cor...

Hi ChubbyLouse32
If I understand correctly you can relatively easy take a clearml Task and launch it on LSF, an integration would be something like:

` from clearml import Task
from clearml.backend_api.session.client import APIClient

while True:
result = client.queues.get_next_task(queue=q_id)
if not result or not result.entry:
sleep(5)
continue
task_id = result.entry.task

here is where we create the LSF job, this is just a pseudo code

os.system("lsf-launch-cmd 'clearml...

is the model overridden or its version is automatically increased?

You will have another model, with the same name (assuming the second Task has the same name), but a new ID. So if I understand you correctly, we have auto-versioning :)

(Not sure it actually has that information)

Where again does clearml place the venv?

Usually ~/.clearml/venvs-builds/<python version>/
Multiple agents will be venvs-builds.1 and so on

` @PipelineDecorator.component(
name="my step", return_values=['data_frame'], cache=True, task_type=TaskTypes.data_processing)
def step_one(pickle_data_url: str, extra: int = 43):

stuff here `This seemed to work for me

is there a way to visualize the pipeline such that this step is “stuck” in executing?

Yes there is, the pipelline plot (see plots section on the Pipeline Task, will show the current state of the pipeline.
But I have a feeling you have something else in mind?
Maybe add Tag on the pipeline Task itself (then remove it when it continues) ?
I'm assuming you need something that is quite prominent in the UI, so someone knows ?
(BTW I would think of integrating it with the slack monitor, to p...

@<1558624430622511104:profile|PanickyBee11> how are you launching the code on multiple machines ?
are they all reporting to the same Task?

And these ?
https://github.com/allegroai/clearml-helm-charts/blob/19a6785a03b780c2d22da1e79bcd69ac9ffcd839/charts/clearml-agent/values.yaml#L48

SubstantialElk6 I just executed it , and everything seems okay on my machine.
Could you pull the latest clearml-agent from the github and try again ?

EDIT:
just try to run:
git clone cd clearml-agent python examples/k8s_glue_example.py

Hi CheerfulGorilla72 ,
Sure there are:
https://github.com/allegroai/clearml/tree/master/examples/frameworks/pytorch-lightning

If I install using

pip install -r ./requirements.txt

then pip installs the packages in the order of the requirements file.

Actually this is not how it works, pip will install in any way it sees fit, and it is not consistent between versions (it has to do with dependency resolving)

However, during the installation process from ClearML, it installs the packages in order UNLESS there's a custom path provided, then it's saved for last

Correct because the custom (I...

Hi @<1523706266315132928:profile|DefiantHippopotamus88>
The idea is that clearml-server acts as a control plane and can sit on a different machine, obviously you can run both on the same machine for testing. Specifically it looks like the clearml-sering is not configured correctly as the error points to issue with initial handshake/login between the triton containers and the clearml-server. How did you configure the clearml-serving docker compose?