Hi BeefyHippopotamus73

. I checked the template task and the list of “Installed Packages” indeed does not have one of my required packages in the list.

Basically the "installed packages" is auto populated based on the directly imported packages n your code base.
Could it be you do not have import snowflake-connector-python and this is a derivative package (i.e. required from a different package)

BTW: when you clone your Task in the UI you can edit and add the missing packages,...

default is clearml data server

Yes the default is the clearml files server, what did you configure it to ? (e.g. should be something like None )

LOL, if you can get it to run any python code, I can help with the rest. We just need to make sure we can capture the output, and then start the VScode remote debugging feature directly from the extension.

No after, do you see the poetry lock removed in the uncommitted changes?

Thanks GiganticTurtle0
So the bug is "mock_step" is storing "NUMBER_2" argument value in the second instance?

Sure thing 🙂

I'm assuming some package imports absl (the TF define package) and that's the reason you see the TF defines). Does that make sense?

I just think that the create function should expect

dataset_name

to be None in the case of

use_current_task=True

(or allow the dataset name to differ from the task name)

I think you are correct, at least we should output a warning that it is ignored ... I'll make sure we do 🙂

Hi DrabCockroach54
Notice the free GPU memory is global hence (low), but the memory (at least with new nvidia drivers) is per process. I'm assuming that the processes using the memory is not a sub process? could that be ? whats the OS you are running on?

Everything seems correct...
Let's try to set it manually.
create a file ~/trains.conf , then copy paste the credentials section from the UI, it should look something like:
api { web_server: http:127.0.0.1:8080 api_server: http:127.0.0.1:8008 files_server: http:127.0.0.1:8081 credentials { "access_key" = "access" "secret_key" = "secret" } }Let's see if that works

Hi RipeGoose2
I just test the hydra example, seems to work when you add the offline right after the import:
` from clearml import Task

Task.set_offline(True) `

Hi ConvolutedSealion94
Yes 🙂
Task.set_random_seed(my_seed=123) # disable setting random number generators by passing None task = Task.init(...)

Let me take a look, what's the clearml-server version and clearml python version?

Ohh sorry you will also need to fix the

def _patched_task_function

The parameter order is important as the partial call relies on it.

My bad no need for that 🙂

DeliciousBluewhale87 great we have progress, this look slike it is inheriting from the system packages:
For example you can see in the log,
Requirement already satisfied: future>=0.16.0 in /usr/local/lib/python3.6/dist-packagesNow the question is which docker it is running, because as you can see at the bottom of the log, tensorflow is not listed as installed, but other packages installed inside the docker are listed.
wdyt?

Hi FunnyTurkey96
Let me check what's the status here
(BTW: Is this for a specific Task or for a specific Project?)

Nice catch AverageBee39 🙂

and those env variables are credentials for ClearML. Since they are taken from k8s secrets, they are the same for every user.

Oh ...

I can create secrets for every new user and set env variables accordingly, but perhaps you see a better way out?

So the thing is, if a User spins the k8s job, the user needs to pass their credentials (so the system knows who it is)... You could just pass the user's key/secret (not nice, but probably not a big issue, as everyone is an Admin anyhow,...

hmm interesting use case, why do you need to add the "--no-binary"

Hi RoundMosquito25
This is a bit old but probably a good start:
https://clear.ml/blog/stacking-up-against-the-competition/
tl;dr
ClearML advantages (at least a few I can think of)
Scales way better Enables out of the box experiment orchestration (i.e. remote execution etc) Data management Nicer UI Full RestAPI Full MLops platform Model serving Query-able model repositoryProbably more 🙂

I think my question is more about design, is a ModelPipeline class a self contained pipeline? (i.e. containing all the different steps or is it a single step in a pipeline)

You can always log it manually:
from clearml import InputModel input_model = InputModel.import_model(weights_url='/tmp/keras_example/weight.6.hdf5')

Really what I need is for A and B to be separate tasks, but guarantee they will be assigned to the same machine so that the clearml dataset cache on that machine will be warm.

I think that what you are looking for is multi-machine cache (which is fully supported). Basically mount an NFS/SMB folder from a NAS to any of those machines, configure the cache folder to point to it, and not you do not need to worry about affinity ?
no?

Is there a way to group A and B into a sub-pipeline, h...

WickedGoat98 I suspect the main difference is with GitHub your are cloning with https (i.e. not credentials needed) , but with gitlab you are using SSH authentication to clone the repository .If on the machine running the trains-agent you can "git clone" your repository (i.e. from command line), the trains-agent should be able to do the same (basically make sure you have the SSH keys in your ~/.ssh folder.

Are you testing the trains-agent service from (i.e. from the docker compose) o...

can we also put the path to the CA?

Yes :)

I think it would be nicer if the CLI had a subcommand to show the content of 

~/.clearml_data.json

 .

Actually, it only stores the last dataset id at the moment, no not much 🙂
But maybe we should have a cmd line that just outputs the current datasetid, this means it will be easier to grab and pipe
WDYT?

requirements specified with git repo

you mean the reuqirements.txt is inside the gir repo? or do you mean a link to the git-repo as part of the requirements?
Can you also provide an example of the content, I think I have an idea

Merged 🙂

Thanks! I think I was able to locate the issue, but I wanted to verify 🙂

yes I'm with you, we need to fix it asap