Thanks EnviousStarfish54
Let me check if I can reproduce it

And is Task.init called on all processes ?

Thanks EnviousStarfish54 we are working on moving them there!
BTW, in the mean time, please feel free to open GitHub issue under train, at least until they are moved (hopefully end of Sept).

Hi @<1533619725983027200:profile|BattyHedgehong22>
Can you elaborate ? what do you mean params file ?
Is this something like:

Task.current_task().connect_configuration('my_conf.json', name="my conf file")

Hi @<1896369926088691712:profile|AdorableCrab88>
Something like:
None
None
None

I was trying to do exactly as you mentioned setting the environment variable 

before

 any trains import but it didn't work

In your entry point script, (even if you do not call trains/ Task.init ) add:
import os os.environ['TRAINS_CONFIG_FILE']='~/my_new_trains.conf' import trainsThen when you actually import trains, everything is already set and it will not read the configuration again.
Make sense ?

No worries 🙂

hmmm How do I reproduce it ?

Hi @<1523701295830011904:profile|CluelessFlamingo93>
What do you mean? what's the difference between ClearML server and self hosted? both are self hosted no?

Hi @<1523701304709353472:profile|OddShrimp85>

there anywhere I could get a charr that can work with lower version of k8s? Or any other methods?

I think the solution is to install it manually from the helm chart (basically take it out and build a Job YAML, wdyt?

Hmm interesting, will pass it along to FE 🙂 3. That is nice! I wonder if this is built into the graph library

Hmm so the Task.init should be called on the main process, this way the subprocess knows the Task is already created (you can call Task.init twice to get the task object). I wonder if we somehow can communicate between the sub processes without initializing in the main one...

Hi VexedElephant56
Yes it is:
Define CLEARML_AGENT_SKIP_PYTHON_ENV_INSTALL=1
(if running in doecker mode add -e CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 as container args)
https://clear.ml/docs/latest/docs/clearml_agent/clearml_agent_env_var

Hi GracefulDog98

Any guess why the password is "incorrect" for me?

Basically the clearml-session CLI needs to be able to access (SSH) into the host (cleaml-agent) machine,
is that possible?

Yep, basically this will query the Task and get the last one:
https://github.com/allegroai/clearml/blob/ca70f0a6f6d52054a095672dc087390fabf2870d/clearml/task.py#L729
Notice task_filter allows you do do all sorts of filtering
https://github.com/allegroai/clearml/blob/ca70f0a6f6d52054a095672dc087390fabf2870d/clearml/task.py#L781

Thanks @<1630377234361487360:profile|RoughSeaturtle43>

server certificate verification failed. CAfile: none CRLfile: none

Oh I see this is an https issue inside the container, you need to mount your self signed certificate
add something like that to your agent.conf:

extra_docker_arguments: ["-v", "/path/to/cert.pem:/etc/ssl/certs/myca.pem"]

None

GiddyTurkey39
BTW: you can always add the missing package via code:
Task.add_requirements('torch', optional_version)

WARNING:root:Could not lock cache folder /home/ronslos/.clearml/venvs-cache: [Errno 11] Resource temporarily unavailable

Hi @<1549927125220331520:profile|ZealousHare78>
could it be you are also working on the same machine ? are you running the agent in docker mode or venv mode ?

no, at least not yet, someone definitely needs to do that though haha

Currently all the unit tests are internal (the hardest part is providing server they can run against and verify the results, hence the challange)

For example, if ClearML would offer a

TestSession

that is local and does not communicate to any backend

Offline mode? it stores everything into a folder, then zips it, you can access the target folder or the zip file and verify all the data/states

• In a notebook, create a method and decorate it by fastai.script’s @call_parse .Any chance you have a very simple code/notebook to reference (this will really help in fixing the issue)?

Sorry @<1524922424720625664:profile|TartLeopard58> 😞 we probably missed it
clearml-session is still being developed 🙂
Which issue are you referring to ?

And when exactly are you getting the "user aborted" message)?
How do you start the process (are you manually running it, or is it an agent, or maybe pycharm?)
Can you provide the full log ?

Wow, those guys....

Maybe we should do that automatically ? wdyt?

Hi SkinnyPanda43

cannot schedule new futures after interpreter shutdown

This seems like a strange exception...
What's the setup here ? jupyter notebook ? how is the interpreter down ?

Is there any progress made on the clearml-serving repo?

Hi JitteryCoyote63
yes, things are progressing slower than expected, I'm expecting actual work will be pushed in early Jan. On the bright side we are trying to work closely with TorchServing team and Nvidia Triton to expand capabilities.
Currently it seems the setup will be "proxy server container" for per-post processing, then serving engine container (Triton/Torch), with monitoring container as control plan (i.e. collecting s...

My pleasure :)

Hmm BitterStarfish58 what's the error you are getting ?
Any chance you are over the free tier quota ?

SubstantialElk6 if you call Task.init with continue_last_task=<task_id> it will automatically add the last_iteration of the previous run, to any logging/report so you never overwrite the previous reports 🙂

Thanks you for noticing the issue!