task.set_script(working_dir=dir, entry_point="my_script.py")Why do you have this part? isn't it the same code, the script entry point is auto detected ?

... or when I run my_script.py locally (in order to create and enqueue the task)?

the latter, When the script is running locally

So something like

os.path.join(os.path.dirname(file), "requirements.txt")

is the right way?

Sure this will work 🙂

where is it running? could you restart all the dockers ? Is it running on your machine?

(This is why we recommend using pip, because it is stable and clearml-agent takes care of pytorch/cuda verions)

Hi EagerOtter28

Let's say we query another time and get 60k images. Now it is not trivial to create a new dataset B but only upload the diff: ...

Use Dataset.sync (or clearml-data sync) to check which files where changed/added.

All files are already hashed, right? I wonder why 

clearml-data

 does not keep files in a semi-flat hierarchy and groups them together to datasets?

It kind of does, it has a full listing of all the files with their hash (SHA2) values, ...

Then the type hints are not removed from helper and the code immediately crashes when being run

Oh yes I see your point, that does make sense (btw removing the type hints will solve the issue)
regardless let me make sure this is solved

FreshParrot56 we could add this capability, but the main caveat is that f your version depends on multiple parent versions you still need to download and extract all the parent versions, which means that when you clear them you might hurt later performance. Does that make sense? What is the use-case / scenario for you?

I think EmbarrassedSpider34 is correct.
When you pass the requirements to clearml-task, actually the agent depending on how it was configured (conda / pip) will do the installation.
That said, maybe it is worth adding support to provide the env.yml in the CLI ?
(Notice that adding specific channels needs to be configured on the agent, they are not stored per Task)
AlertCamel57 wdyt?

The import process actually creates a new Task every import, that said if you take a look here:
https://github.com/allegroai/trains/blob/10ec4d56fb4a1f933128b35d68c727189310aae8/trains/task.py#L1733
you can pass a pre-existing Task ID to "import_task" https://github.com/allegroai/trains/blob/10ec4d56fb4a1f933128b35d68c727189310aae8/trains/task.py#L1653

@<1671689437261598720:profile|FranticWhale40> this one: None

Hmm that is odd. Let me take a look and ask the guys. Thank you for quickly testing the RC! I'm hoping a new RC with a fix will be there tomorrow, if we can quickly replicate

I'm guessing this is done through code-server?

correct

I'm currently rolling a JupyterHub instance (multiuser, with codeserver inside) on the same machine as clearml-server. That’s where tasks are executed etc. so, all browser dev env.

Yeah, the idea with clearml-session each user can self serve themselves the container that works best for them. With a jupyterhub they start to step on each other's toes very quickly ...

Done HandsomeCrow5 +1 added 🙂
btw: if you feel you can share how your reports looks like (screen shot is great), this will greatly help in supporting this feature , thanks

Hi @<1643060801088524288:profile|HarebrainedOstrich43>
try this RC let me know if it works 🙂

pip install clearml==1.13.3rc1

Is there a helper function option at all that means you can flush the clearml-agent working space automatically, or by command?

Every Task execution the agent clears the venv (packages are cached locally, but the actual venv is cleared). If you want you can turn on the venv cache, but there is no need to manually clear the agent's cache.

is it normal that it's slower than my device even though the agent is much more powerful than my device? or because it is just a simple code

Could be the agent is not using the GPU for some reason?

Hi ConvincingSwan15
A few background questions:

Where is the code that we want to optimize? Do you already have a Task of that code executed?

"find my learning script"

Could you elaborate ? is this connect to the first question ?

No -- that section is blank,

This is the main issue, it should be filled with the requirement being auto detected.
The entire script was executed from within vscode, and the Task was created but it was not prefilled with anything ?
Just making sure, you called Task.init inside your code ?

Are you saying you had that odd script entry-point created by calling Task.init? (To clarify this is the problem)
Btw after you clone the experiment you can always manually edit both entry point and working dir, which based on what you said should be "script.py" and "folder"

i would like to have it also save on the bucket

oh if this is the casse, you can just change the clearml file server to point to GS bucket, everything will be stored there.
Just change your clearml.conf:
files_server: " "https://github.com/allegroai/clearml/blob/d45ec5d3e2caf1af477b37fcb36a81595fb9759f/docs/clearml.conf#L10

BroadMole98 as one can expect long answer as well 🙂

I have a workflow with 19000 job nodes in it.

wow, 19k job nodes? as in a single pipeline 19k steps?

The main idea of the trains-agent is to allow multi-node workloads, and creating pipelines on top of a scheduler without worrying about docker packaging (done automatically for you), and to have a proper scheduler with priority (that is missing from k8s)

If the first step is just "logging" all the steps, you can easily add "Task...

Hi TrickySheep9
Long story short, clearml-session fully supports k8s (using k8s glue)
The --remote-gateway along side ports mode will basically allow you to setup a k8s service so that every session will register with a specific port so k8s does ingest foe you and route the SSH connection to the pod itslef, everything else is tunneled over the original SSH connection.
Make sense ?

print(requests.get(url='

print(requests.get(url='

Wait even without the pipeline decorator this function creates the warning?

I can reproduce 😞 give me a moment to verify

@<1523706266315132928:profile|DefiantHippopotamus88> seems like you are missing the ports 🙂

CLEARML_WEB_HOST="

"
CLEARML_API_HOST="

"
CLEARML_FILES_HOST="

"

PlainSquid19 Trains will analyze the entire repository if this is a git repo code, and a single script file if there is no repository found.

It will not analyze an entire folder if it is not in a git repository, because it will not be able to recreate this folder anyhow. Make sense ?

Hi ExcitedCat13
Sure, download the plugin from the git repo (Install instructions in the repo).
Regarding remote debugging, are referring to ssh ?
The plugin itself is designed to make sure that when you work on a remote machine with pycharm clearml will log the local git repo and changes (as the .git folder is not synced to the remote machine)

Hi ProudChicken98
task.connect(input) preserves the types based on the "input" dict types, on the flip side get_parameters returns the string representation (as stored on the clearml-server).
Is there a specific reason for using get_parameters over connect ?

Hi UpsetTurkey67

repository discovery stores github repo in the form:

...

while for others

git@github.com:...

Yes that depends on how they locally cloned the repo (via SSH or user/pass/token)

Interestingly in the former case the ssh config is ignored and cloning repository breaks on the worker

If you have passed git user/pass to the agent it should use them not SSH, how did you configure the agent ?

Hi @<1578193384537853952:profile|MoodyOx45>

I have a task A that creates another task B via subprocess.

So the thing about the agent, when it runs the code, there is only One task to rule them all. basically any fork/spawn of subprocess will automatically be logged as the parent Task
I think that what you want is to build a pipeline from those Tasks? Or create a Task and enqueue it manually directly from Task A?

(btw: you can forcefully cause the subprocess to create it's own Task b...