Of course. Here it is
https://github.com/allegroai/clearml/issues/684
I'll keep you updated

Hi NonsensicalWoodpecker96
you can you the SDK 🙂

task = Task.init(project_name=project_name, task_name=task_name)
task.set_comment('Hi there')

Hi SteepDeer88
I wrote this script to try to reproduce the error. I am passing there +50 parameters and so far everything works fine. Could you please give me some more details about your issue, so that we could reproduce it ?

from clearml import Task
import argparse

'''
COMMAND LINE:
python -m my_script --project_name my_project --task_name my_task --execute_remotely true --remote_queue default --param_1 parameter...

Hi PanickyMoth78
There is indeed a versioning mechanism available for the open source version 🎉

The datasets keep track of their "genealogy" so you can easily access the version that you need through its ID

In order to create a child dataset, you simply have to use the parameter "parent_datasets" when you create your dataset : have a look at
https://clear.ml/docs/latest/docs/clearml_data/clearml_data_sdk#datasetcreate

You also alternatively squash datasets together to create a c...

i dont know if it will help but here is what i would test :
remove temporary the task init in the controller use name and project parameters that dont have spaces in their name dont use services as a default queue

hey SmugSnake6
Can you give some more precisions on your configuration please ? (clearml, agent, server versions)
Also, if you have some example code to share it could help us reproduce the issue and thus help you a lot faster 🙂 (script, command line for firing your agent)

I see some points that you should fix
in the train step, you return 2 items but you have only one in its decorator: add mock do you really need to init a task in the pipeline controller ? you will automatically get one when executing the pipeline

you are in a regular execution - i mean not a local one. So the different pipeline tasks has been enqueued. You simply need to fire an agent to pull the enqueued tasks. I would advice you to specify the queue in the steps (parameter execution_queue ).
You then fire your agent :
clearml-agent daemon --queue my_queue

when you spin a container , you map a host port with a container port using -p parameter
docker run -v ~/clearml.conf:/root/clearml.conf -p 8080:8080 -e CLEARML_SERVING_TASK_ID =<service_id> -e CLEARML_SERVING_POLL_FREQ =5 clearml-serving-inference:latest
Here you map your computer's port 8080 with the container port 8080. If your 8080 port is already used, you can use another, using for example -p 8081:8080

can you share your logs ?

can you do docker ps to check if there are running containers that already bind the port ?

i managed to import a custom package using the same way you did : i have added the current dir path to my system
i have a 2 steps pipeline :

1. Run a function from a custom package. This function returns a Dataloader (built from torchvision.MNIST) 2) This step receives the dataloader built in the first step as a parameter ; it shows random samples from itthere has been no error to return the dataloader at the end of step1 and to import it at step2. Here is my code :

` from clearml import Pi...

Do you think that you could send us a bit of code in order to better understand how to reproduce the bug ? In particular about how you use dotenv...
So far, something like that is working normally. with both clearml 1.3.2 & 1.4.0

`
task = Task.init(project_name=project_name, task_name=task_name)

img_path = os.path.normpath("**/Images")
img_path = os.path.join(img_path, "*.png")

print("==> Uploading to Azure")
remote_url = "azure://****.blob.core.windows.net/*****/"
StorageManager.uplo...

but in the other hand, when you parse your minio console, you have all the buckets shown as directories right ? there is no file in the root dir. So we used the same logic and decided to reproduce that very same structure. Thus when you will parse the local_folder, you will have the same structure as shown in the console

Hi,
We are going to try to reproduce this issue and will update you asap

Interesting. We are opening a discussion to weight the pros and cons of those different approaches - i ll of course keep you updated>
Could you please open a github issue abot that topic ? 🙏
http://github.com/allegroai/clearml/issues

check that your task are enqueued in the queue the agent is listening to.
from the webUI, in your step's task, check the default_queue in the configuration section.
when you fire the agent you should have a log that also specifies which queue the agentis ssigned to
finally, in the webApp, you can check the Workers & Queues section. There you could see the agent(s), the queue they are listening to, and what tasks are enqueued in what queue

i am also speaking with another user this morning, who has the very same issue
can you give me some more details about your config, and share your error logs please ?

One agent is assigned to one queue ; so he will execute one task at the time, sequentially, according to their rank in the queue. But you can create as many queues as you want, and assign an agent for each one. You simply fire each agent from a terminal, with a command like :
clearml-agent daemon --queue my_queue_i
if you have more than one gpu, you can also choose for each agent which gpu(s) to allocate

You can initiate your task as usual. When some dataset will be used in it - for example it could start by retrieving it using Dataset.get - then the dataset will be registered in the Info section (check in the UI) 😊

hey WhoppingMole85 good morning !
try to pip it !
pip install clearml -U
and then check with
pip show clearml

yes it could worth it, i will submit, thanks. This is the same for Task.get_task() : either id or project_name/task_name
🙂

hey WhoppingMole85
Do you want to initiate a task and link it to a dataset, or simply create a dataset ?

you can freeze your local env and thus get all the packages installed. With pip (on linux) it would be something like that :
pip freeze > requirements.txt
(doc here https://pip.pypa.io/en/stable/cli/pip_freeze/ )

yes it is 🙂 do you manage to upgrade ?
We also brought a lot of new features in the datasets in 1.6.2 version !

what bother me is that it worked until yesterday, and you didnt changed your code. So the only thing i can think of is a bug introduced with the new sdk version, that was released yesterday. I am inverstigating with the sdk team, i will keep you updated asap ! 🙂

is it a task you are trying to access to or a dataset ? if you need to retrieve a task, you should use Task.get_task()

if i do that :
ds=Dataset.create(dataset_project='datasets',dataset_name='dataset_0')
it will result in the creation of 2 experiments :
results page: the task that corresponds to the script that launched the dataset creation - it will be in PROJECTS/datasets/.datasets/dataset_0 dataset page: the dataset itself : would be in DATASETS/dataset_0

i am not sure i get you here.
when pip installing clearml-agent, it doesnt fire any agent. the procedure is that after having installed the package, if there isnt any config file, you do clearml-agent init and you enter the credentials, which are stored in clearml.conf. If there is a conf file, you simply edit it and manually enter the credentials. so i dont understand what you mean by "remove it"

can you also check that you can access the servers ?
try to do curl http://<my server>:port for your different servers ? and share the results 🙂