And task = Task.init(project_name=conf.get("project_name"), ...) is basically a no-op in remote execution so it does not matter if conf is empty, right?

I think I may have brought this up multiple times in different ways :D
When dealing with long and complicated configurations (whether config objects, yaml, or otherwise), it's often useful to break them down into relevant chunks (think hydra, maybe).

In our case, we have a custom YAML instruction !include , i.e.
` # foo.yaml
bar: baz

bar.yaml

obj: !include foo.yaml
maybe_another_obj: !include foo.yaml `

I dunno :man-shrugging: but Task.init is clearly incompatible with pytest and friends

Any thoughts @<1523701070390366208:profile|CostlyOstrich36> ?
I wouldn’t want to run the entire notebook, just a specific part of it.

I guess in theory I could write a run_step.py , similarly to how the pipeline in ClearML works… 🤔 And then use Task.create() etc?

Not really - it will just show the string. A preview would be more like a low-res version of the uploaded image or similar.

We're wondering how many on-premise machines we'd like to deprecate. For that, we want to see how often our "on premise" queue is used (how often a task is submitted and run), for how long, how many resources it consumes (on average), etc.

Is it currently broken? 🤔

Hm, just a small update - I just verified and it does indeed work on linux:

` import clearml
import dotenv

if name == "main":
dotenv.load_dotenv()
config = clearml.backend_api.Config.load() # Success, parsed with environment variables `

Interesting, why won’t it be possible? Quite easy to get the source code using e.g. dill .

It can also log generate a log file with this method, it does not have to output it to CONSOLE tab.

I wouldn't mind going the requests route if I could find the API end point from the SDK?

~ is a bit weird since it's not part of the package (might as well let the user go through clearml-init ), but using ${PWD} works! 👍 👍

(Though I still had to add the CLEARML_API_HOST and CLEARML_WEB_HOST ofc, or define them in the clearml.conf)

In the Profile section, yes, they are well defined (bucket, secret, key, and endpoint)

@<1523701070390366208:profile|CostlyOstrich36> I added None btw

Always great to find a bug! I'll make relevant SDK updates then.

Feels like we've been over this 😄 Has there been new developments perhaps?
It's essentially that this - https://clear.ml/docs/latest/docs/guides/advanced/multiple_tasks_single_process cannot work in a remote execution.

Either one would be nice to have. I kinda like the instant search option, but could live with an ENTER to search.

I opened this meanwhile - https://github.com/allegroai/clearml-server/issues/138

Generally, it would also be good if the pop-up presented some hints about what went wrong with fetching the experiments. Here, I know the pattern is incomplete and invalid. A less advanced user might not understand what's up.

I have seen this quite frequently as well tbh!

I'll kill the agent and try again but with the detached mode 🤔

Adding bucket = clearml in aws.s3.credentials did not help either

Hm, that seems less than ideal. I was hoping I could pass some CSV locations. I'll try and find a workaround for that. Thanks!

I just used this to create the dual_gpu queue:
clearml-agent daemon --queue dual_gpu --create-queue --gpus 0,1 --detached

Hah. Now it worked.

yes, a lot of moving pieces here as we're trying to migrate to AWS and set up autoscaler and more 😅

Yeah I managed to work around those former two, mostly by using Task.create instead of Task.init . It's actually the whole bunch of daemons running in the background that takes a long time, not the zipping.

Regarding the second - I'm not doing anything per se. I'm running in offline mode and I'm trying to create a dataset, and this is the error I get...
There is a data object it, but there is no script object attached to it (presumably again because of pytest?)

The key/secret is also shared internally so that sounds like a nice mitigation actually!
Which environment variable am I looking for? I couldn't spot anything specifically in that environment variables page

There's not much (or anything) in the log to provide...

` (.venv) 15:42 [0:user@server$~] CLEARML_CONFIG_FILE=~/agent_clearml.conf clearml-agent daemon --queue default on_prem --detached --order-fairness
Environment variables set from configuration: ['AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY', 'AWS_DEFAULT_REGION']
...

We have the following, works fine (we also use internal zip packaging for our models):

model = OutputModel(task=self.task, name=self.job_name, tags=kwargs.get('tags', self.task.get_tags()), framework=framework)
model.connect(task=self.task, name=self.job_name)
model.update_weights(weights_filename=cc_model.save())

Okay so the new functionality is maintained; thanks!