Hi @<1523701868901961728:profile|ReassuredTiger98> when you get to it...
please download the wheel, then install it with

pip3 install -U clearml_agent-0.17.3rc0-py3-none-any.whl

Then run the daemon with the additional --debug argument, basically:

clearml-agent --debug daemon --foreground ...

Once the agent is running please send the Task's log from your console 🙂

Let me know if it solved it, if it did I'll make sure we push the RC

ElegantCoyote26 can you browse to http://localhost:8080 on the machine that was running the trains-init ?

Task.set_offline(True)
https://clear.ml/docs/latest/docs/references/sdk/task#taskset_offline

Hi LovelyHamster1
As you noted, passing overrides in Args/overrides , for example ['training.max_epochs=1000']
should work when running with the agent.

Could you verify with the latest RC, there was a fix to support the latest hydra version
pip install clearml==0.17.5rc5

Will using Model.remove, completely delete from storage as well? (edited)

correct see argument delete_weights_file=True

You mean the entire organization already has Kubeflow, or to better organize something (if this is the second, what are we organizing, pipelines?)

Ohh so you are saying you can store it properly, but only editing in the UI is limited ? (Maybe this is just a UI thing)

We should probably change it so it is more human readable 🙂

I want to be able to access the data just avoid reporting the experiment results

Yes, you are correct 😞
If you just want to skip the logging you can always add an if to the Tasl.init call ?!

I’ll check if I could wrap the code in something that calls the Task.delete if debugging

Whatever you think works best for you, I was genuinely curious 🙂
To me (personally) it is helpful to have a log even while debugging (comparing to previous runs etc, trying to see what went wrong even on a console output level). When I'm done I just search for everything I worked on select all, and archive them. Then a cleanup service in the background clears all the archived Tasks once they ar...

BTW: Basically just call Task.init(...) the rest is magic 🙂

So obviously that is the problem

Correct.
ShaggyHare67 how come the "installed packages" are now empty ?
They should be automatically filled when executing locally?!
Any chance someone mistakenly deleted them?
Regrading the python environment, trains-agent is creating a new clean venv for every experiment, if you need you can set in your trains.conf :
agent.package_manager.system_site_packages: true
https://github.com/allegroai/trains-agent/blob/de332b9e6b66a2e7c67...

Hi @<1727497172041076736:profile|TightSheep99>
Yes it can, it will upload the meta-data as well as the files (it will also do de-dup and will not upload files that already exist in the dataset based on the hash of teh file content)

For future readers, see discussion here:
https://clearml.slack.com/archives/CTK20V944/p1629840257158900?thread_ts=1629091260.446400&cid=CTK20V944

AbruptWorm50 can you send full image (X axis is missing from the graph)

DefeatedOstrich93 many thanks I was able to reproduce it (basically newly added files caused git apply to fail)
Fix will be part of the next clearml-agent RC

Yes this seems like it is stuck, could you test with the demo server ?
(basically remove the clearml.conf it will connect automatically)

1e876021bbef49a291d66ac9a2270705 just make sure you reset it 🙂

Hi JitteryCoyote63

I change the project.default_output_destination? I tried setting it to None but it is not updated

How did yo try to change it? and where do you see the effect ?

I can definitely see your point from the "DevOps" perspective, but from the user perspective it put the "liability" on me to "optimize" the resource, which to me sounds a bit much to put on my tiny shoulders, I just have a general knowledge on what I need. For example lots of CPUs (because I know my process scales well with more cpus), or large memory (because I have an entire dataset in memory). Personally (and really only my personal perspective), I'd rather have the option to select from a...

and those env variables are credentials for ClearML. Since they are taken from k8s secrets, they are the same for every user.

Oh ...

I can create secrets for every new user and set env variables accordingly, but perhaps you see a better way out?

So the thing is, if a User spins the k8s job, the user needs to pass their credentials (so the system knows who it is)... You could just pass the user's key/secret (not nice, but probably not a big issue, as everyone is an Admin anyhow,...

NaughtyFish36

No module named 'leap.learn.data_tools.merge_data.merge_data'

This seems to be the error but I cannot see leap in the installed packages , Notice that if the Task has "Installed Packages" section then the agent will use that Not the "requirements.txt" , Only if this section is Empty it will revert to the "requirements.txt" in the repo.
How did you create the Task in the first place?
I see that you added "leap" into the initial bashscript, actually you should add i...

might be my folder permissions hmm

That actually makes sense, also notice that if you are running under a diff user, the ~ (home folder) is different

I'm thinking of a few plots in my current in-house tooling which are slightly different than the standard charts we look at. For example a custom parallel coordinate chart that can use aggregations, categorical variables, etc.

This can be done by comparing experiments, then check the Hyper-Parameters tab, and select graph from the drop down at the top

So my question in general is pertaining to if I would need to get better at Javascript if I were to make those changes. My guess is ...

Hi @<1695969549783928832:profile|ObedientTurkey46>
Use --services-mode in the agent , it will run many Tasks on the same machine, this is usually associated with the services queue, but can be run on any queue. This way you could have the same machine easily running those multiple "control" tasks.
wdyt?

Hurray ?

Task.running_localy()
Should do the trick

I guess. or pipelines that you can compose after running experiments to see that experiments are connected to each other

hmm what do you mean by "compose after running experiments" ? like a way to group them? what is the relation between one "item" to another ?
If this is a sequence of Tasks , are they executed by a controller ?

ok, I will do a simple workaround for this (use an additional parameter that I can update using parameter_override and then check if it exists and update the configuration in python myself)

Yep sounds good, something like this?
from clearml.utilities.dicts import ReadOnlyDict, merge_dicts overrides = {} task.connect(overrides) configuration = {#stuff here} task.connect_configuration(configuration) merge_dicts configuration.update(overrides)BTW: this will allow you to override any s...