No worries, it's always good to know what can be built later.
I would start with a static .env file (i.e. the same for everyone), or start with hacking the python code to load the .env at the beginning 🤞

I have made a PR request.

Thanks you!!! 🎉 we will merge shortly 🙂

I see, give me a minute to check what would be the easiest

Sure, thing, I'll fix the "create_draft" docstring to suggest it

Hi RoughTiger69
A. Yes makes total sense . Basically you can use Task.export Task.import to do achieve this process (notice we assume the dataset artifacts links are available on both, usually this is the case)

B. The easiest way would be to use Process , then one subprocess is exporting from dev , where the credentials and configuration is passed with os environment. The another subprocess imports it to the prod server (again with os environment pointing to the prod server). Make sense?

yeah. I am getting logs, but they are extremely puzzling to me. I would appreciate to actually have access to whole package structure..

Actual packages are updated back to "Installed Packages" section (under the execution tab).

indeed. can you maybe point where the docker command is composed.

https://github.com/allegroai/clearml-agent/blob/178af0dee84e22becb9eec8f81f343b9f2022630/clearml_agent/commands/worker.py#L3694
🙂
BTW: you can run/build the entire thing on your machin...

Hi DeliciousBluewhale87
Hmm, good question.
Basically the idea is that if you have ingestion service on the pods (i.e. as part of the yaml template used by the k8s glue) you can specify to the glue what are the exposed ports, so it knows (1) what's the maximum of instances it can spin, e.g. one per port (2) it will set the external port number on the Task, so that the running agent/code will be aware of the exposed port.
A use case for it would be combing the clearml-session with the k8s gl...

CooperativeFox72 btw, are you guys running those 20 experiments manually or through trains-agent ?

RoundMosquito25 actually you can 🙂
# check the state every minute while an_optimizer.wait(timeout=1.0): running_tasks = an_optimizer.get_active_experiments() for task in running_tasks: task.get_last_scalar_metrics() # do something herebase line reference
https://github.com/allegroai/clearml/blob/f5700728837188d7d6005726c581c9d74fd91164/examples/optimization/hyper-parameter-optimization/hyper_parameter_optimizer.py#L127

Yey! nice!

That makes sense...
Basically in the open-source version the approach is everyone sees everything for maximum transparency (and also ease of use). I know there are access-roles in the paid tier and vault for exactly these types of things...

Where do you currently save them? and how do you pass them to the remote machine ?

I'm assuming these are the Only packages that are imported directly (i.e. pandas requires other packages but the code imports pandas so this is what listed).
The way ClearML detect packages, it first tries to understand if this is a "standalone" scrip, if it does, than only imports in the main script are logged. Then if it "thinks" this is not a standalone script, then it will analyze the entire repository.
make sense ?

I think there was an issue with the entire .ml domain name (at least for some dns providers)

can we use a currently setup virtualenv by any chance?

You mean, if the cleamrl-agent needs to setup a new venv each time? are you running in docker mode ?
(by default it is caching the venv so the second time it is using a precached full venv, installing nothing)

Hmm you mean how long it takes for the server to timeout on registered worker? I'm not sure this is easily configured

if i put pipe.start earlier in the code, the pipeline fails to execute the actual steps.

pipe.start should be called after the pipeline was constructed and should be the "last" call of the script.
Not sure I follow what is "before" the code?

if I want to run the experiment the first time without creating the 

template

?

You mean without manually executing it once ?

BTW: I think an easy fix could be:
if running_remotely(): pipeline.start() else: pipeline.create_draft()

CrookedWalrus33 I found the issue, this is only failing with Python 3.6.
Let me check something

Hi BurlyPig26
I think you can easily change the Web port, but not the API (8008) or files (8081) port
How are you deploying it?

Task.running_localy()
Should do the trick

oh sorry my bad, then you probably need to define all OS environment variable for python temp folder for the agent (the Task process itself is a child process so it will inherit it)

TMPDIR/new/tmp TMP=/new/tmp TEMP=/new/tmp clearml-agent daemon ...

Hi @<1532532498972545024:profile|LittleReindeer37>
Does Hydra support notebooks ? If it does, can you point to an exapmle?

@<1560074028276781056:profile|HealthyDove84> if you want you can PR a fix, it should be very simple basically:
None

        elif np_dtype == str:
            return "STRING"
        elif np_dtype == np.object_ or np_dtype.type == np.bytes_:
            return "BYTES"
        return None

Thanks ElegantCoyote26 I'll look into it. Seems like someone liked our automagical approach 🙂

I suppose the same would need to be done for any 

client 

PC running 

clearml

 such that you are submitting dataset upload jobs?

Correct

That is, the dataset is perhaps local to my laptop, or on a development VM that is not in the 

clearml

 system, but I from there I want to submit a copy of a dataset, then I would need to configure the storage section in the same way as well?

Correct

Hi TrickyFox41

is there a way to cache the docker containers used by the agents

You mean for the apt get install part? or the venv?
(the apt packages themselves are cached on the host machine)
for the venv I would recommend turning on cache here:
https://github.com/allegroai/clearml-agent/blob/76c533a2e8e8e3403bfd25c94ba8000ae98857c1/docs/clearml.conf#L131

Regrading the missing packages, you might want to test with:
force_analyze_entire_repo: falsehttps://github.com/allegroai/trains/blob/c3fd3ed7c681e92e2fb2c3f6fd3493854803d781/docs/trains.conf#L162

Or if you have a full venv you like to store instead:
https://github.com/allegroai/trains/blob/c3fd3ed7c681e92e2fb2c3f6fd3493854803d781/docs/trains.conf#L169

BTW:
What is the missed package?

oh dear 😞 if that's the case I think you should open an Issue on pypa/pip , I'm not sure what we can do other than that ...

check the latest RC, it solved an issue with dataset uploading,
Let me check if it also solved this issue