(probably sync from CLI 🙂 )

tell me please, does the agent always create a virtual environment?

Yes, but it inherits from the container preinstalled system environment

is it possible to make the agent run the script in an already prepared docker container without creating a virtual environment in the container?

You can set the CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 environment variable

As I installed ClearML using pip,

Where is the clearml-serving runs ? usually your configuration file is in ~/clearml.conf
Notice if it is not there it means it is using the defaults so just create a new one and add that line

@<1541954607595393024:profile|BattyCrocodile47> not restarting the docker, restarting the Docker service (on Mac it's an app, I think there is an option on the Docker app to do that)

Okay, I think this might be a bit of an overkill, but I'll entertain the idea 🙂
Try passing the user as key, and password as secret?

The goal is to be able to run

docker-compose up

in CI, which starts a clearml-server. And then make several API calls to the started ClearML server to prove that the VS Code extension code is working.

Oh I see, if this is CI workflow, why not run in offline mode ?
None

Hi @<1661542579272945664:profile|SaltySpider22>
Basically you need to put all of these files into a repository , which is always a good practice.
The reason is that the pipeline (and for that matter any Task on the system) can store wither a single script or a git reference, but not multiple scripts.

a CI for the vscode extension ? So spin server + agent & connect to it as part of the CI?

yes, or (because I deployed clearml using helm in kubernetes) from the same machine, but multiple pods (tasks).

Oh now I see, long story short, no 😞 the correct way of doing that is every node/pod creates it's own dataset,
then when you are done, you create a new version with the X datasets that you created as parents, the newly created version is just "meta" it basically tells the system how to combine the previously generated datasets (i.e. no data is actually re-uploa...

Are you inheriting from their docker file ?

The easiest if export_task / update_task:
https://allegro.ai/docs/task.html#trains.task.Task.export_task
https://allegro.ai/docs/task.html#trains.task.Task.update_task
Check the structure returned by export_task, you'll find the entire configuration test there,
then, you can use that to update back the Task.

BTW:
Partial update is also supported...

Is this reproducible ?

SteepDeer88
Try the following:
` Task.add_requirements("pycocotools-windows", "; platform_system == "Windows"")
Task.add_requirements("pycocotools", "; platform_system != "Windows"")

Task.init(...) You should see in your "installed packages" something like: pycocotools-windows ; platform_system == "Windows"
pycocotools ; platform_system != "Windows" `

I see, that means xarray is not an actual package but a folder add to the python path.
This explains why Task.add_requirements fails, as it is supposed to add python packages to the equivalent of "requirements.txt" ...
Is the folder part of the git repository ? How would you pass it to the remote machine the cleamrl-agent is running on?

When are those keys used?

They are the default keys for internal access, basically just make up something, otherwise someoune could access the server with the default keys

Hi MagnificentSeaurchin79
Yes this is a bit confusing 🙂
Datasets are stored as delta changes from parent versions.

A dataset contains a list of files and list of artifacts where these files exist. This means that if we add a new file to a dataset we create a new dataset from a parent dataset and want to add a file, we have to add a link to the file, and have a new artifact containing just the delta (i.e. the new file) from the parent version When you delete a file you just remove the li...

we made two tb versions of / task and wrote in parallel.
And I wanted to know if it is possible here as well.

Basically you will have different series (based on the TB log file) on the same graph so you can compare 🙂 all automatically

JitteryCoyote63 This is odd you have both python3.9 and python3.8 on the container, and since it says (probably) ob the task the agent should run python3.9 it's trying to use it for creating the enthronement (it does not matter that agent is using python3.8).
1673431344706 agent-1 DEBUG /usr/bin/python3.9 /usr/bin/python3.9: No module named pipThis is the main issue, pip is missing for python3.9 and this is why it reverted to python 3.8 when it was setting the environment.
It should prob...

JitteryCoyote63

I agree that its name is not search-engine friendly,

LOL 😄
It was an internal joke the guys decided to call it "trains" cause you know it trains...
It was unstoppable, we should probably do a line of merch with AI 🚆 😉
Anyhow, this one definitely backfired...

Probably less secure though :)

Hi @<1561885921379356672:profile|GorgeousPuppy74>

• Could you copy the 3 messages here into your original message, it helps keeping things tidy and nice (press on the 3 dot menu and select edit)
• what do you mean by "currently its not executing in queue-01", you changed it so it should be pushed to queue-02, no? Also notice that you can run the enire pipeline as sub-processes for debugging,
  just call pipe.start_locally(run_pipeline_steps_locally=True)
  You also need an agent on the ser...

is it consistent ? (the error), meaning it happens on other integer values ?

Awesome! any way to hear the talk w/o/ registering for the whole conference?

CloudySwallow27 Anyway we will make sure we upload the talk to the clearml youtube channel after the Talk

I think you can watch it after GTC on the nvidia website, and a week after that we will be able to upload it to the youtube channel 🙂

Yep 🙂
Basically:
` task = Task.get_task(task_id='aaaa')
while task.status not in ('completed', 'stopped',):

do something ?

sleep(15) `(Notice task.status / task.get_status() will refresh the Task status on every call)

Different question. How can I pass PYTHONPATH env variable to a task, run by agent (so python can find classes inside m subdirectories)?

Hi HelpfulHare30
By default the working directory will be added to the python path, this means if I have under execution:
Working Dir: "." Script: "src/script.py"The root git repo will be added to the python path.
BTW: next RC you could add a flag to the agent to always add the git repo

Mmm well, I can think of a pipeline that could save its state in the instant before the error occurred.

This is already the case, if you clone the pipeline Task change the Args/_continue_pipeline_ to True and enqueue

Linking to my answer here:
https://clearml.slack.com/archives/C028LNAAR5H/p1636989891013200?thread_ts=1636963279.012500&cid=C028LNAAR5H

TrickySheep9 Yes, let's do that!
How do you PR a change ?

that's the entire repo link ? not something like https://github.com/ ... ?