hi VexedKoala41
Your agent is running into a docker container that may have a different version of python installed. It tries to install a version of the package that doesn't exist for this python version.
Try to specify the latest matching version Task.add_requirements( ‘ipython’ , '7.16.3')

Concerning how to use ParameterSet :
I first declare the set
my_param_set = ParameterSet([ {'General/batch_size': 32, 'General/epochs': 30}, {'General/batch_size': 64, 'General/epochs': 20}, {'General/batch_size': 128, 'General/epochs': 10} ])This is a very basic example, it is also possible to use more complex things into the set (see https://clear.ml/docs/latest/docs/references/sdk/hpo_parameters_parameterset/ for UniformParameter Range usage in ParameterSet).

Then i do ...

Yes I think that it would be great if you could try to run it with a simpler configuration

what do you mean ? the average time that the tasks are waiting before being executed by an agent ? that is to say the average difference between enqueue time and beginning time ?

JuicyFox94

i guess so. make your tests and please keep us updated if you still encounter issues 🙂

Hey ReassuredTiger98
Is there any update from your side ?
I confirm that you need to put your key and secret in the credentials section of the configuration file . As Idan, I let my policy configuration untouched

ok so here is the example.
the idea is to use the API, and finally reproduce what the WebUI does.

` from clearml.backend_api.session.client import APIClient
from clearml import Task

task = Task.get_task(task_id=xxxx)
#or Task.get_task(project_name=xxx, task_name=xxx)

client = APIClient()
my_data = client.tasks.get_by_id(task.id).to_dict()

with open('./my_data.csv', 'w') as f:
for key in my_data.keys():
f.write("%s, %s\n" % (key, my_data[key]) ) `

hi PanickyMoth78
from within your function my_pipeline_function here is how to access the project and task names :

task = Task.current_task()
task_name = task.name
full_project_path = task.get_project_name()
project_name = full_project_path.split('/')[0]

Note that you could also use the full_project_path to get both project and task name
task_name = full_project_name.split('/')[-1]

🙂
here is a bit of code that seems to do the job. have a look
`
wrapper = Task.get_task(project_name="***", task_name="***")

req_obj = events.DownloadTaskLogRequest(wrapper.id)
res = wrapper.session.send_request(
service=req_obj._service,
action=req_obj._action,
version=req_obj._version,
json=req_obj.to_dict(),
method=req_obj._method,
async_enable=False,
headers=None,
)
print(res.json()) `

hey RoughTiger69
Can you describe me how you are setting up the environment variable please ?

Setting up that flag will skip the virtual env installation : the agent will use your environment and the packages installed into it.

Using Task.add_requirements(requirements.txt) allows to add specific packages at will. Note that this function will be executed even with the flag CLEARML_AGENT_SKIP_PIP_VENV_INSTALL set

If the data is updated into the same local / network folder structure, which serves as a dataset's single point of truth, you can schedule a script which uses the dataset sync functionality which will update the dataset based on the modifications made to the folder.

You can then modify precisely what you need in that structure, and get a new updated dataset version

thanks ! we have added quite a lot of new features on datasets on our last releases. I would encourage you to update your clearml packages 🙂

can you describe the issue ?

can you share the logs please ?

hey UnevenDolphin73
did you manage to get rid of this issue ?

Hey UnevenDolphin73
I have tried to reproduce the issue but with no success. I manage to auto report any graph designed according to your description - values between [0,50] and sudden extreme values. So far everything seems to be ok on my side. have you found something new reguarding this issue ? Could you send me more details on the graph which reporting hangs ?
Thanks

Hi SmugTurtle78
We currently don't support GitHub deploy keys, but there might be a way to make the task use SSH (and not HTTPS), so that you could put the SSH key on the AWS machine. Please let me check if I can find such a solution, and come back to you after.

hi DistressedGoat23
can you send me an example of you report your histogram ?
i managed to transpose a histogram using something like this :

` t = Task.init(project_name=project_name, task_name=task_name)

df = px.data.tips()
figV = px.histogram(df, x="total_bill", y="tip", orientation='v')
figH = px.histogram(df, x="total_bill", y="tip", orientation='h')

log = t.get_logger()

log.report_plotly(
title='Histo',
series='plotlyH',
iteration=0,
figure=figH)

log.report_plotly(...

Hi AverageRabbit65
You are using Pipeline from Task.
The steps in this case are existing clearml tasks, thus the task you specify when you add each step ( parameters base_task_project and base_task_name ) are attributes of pre existing tasks.
To make this example work, you have first to create them
project_name = 'rudolf' Task.init(project_name=project_name, task_name="Pipeline step 1 process dataset") Task.init(project_name=project_name, task_name="Pipeline step 2 train model")
You co...

good to know. we will try to enquire that. thanks

hi SparklingElephant70
i was asking myself about this datasets / .datasets / None
this None i weird : if you look at the example i sent, you should see the dataset name here. Just to be sure, can you confirm me than when you fire the command line you pass both dataset_project AND dataset_name ?

hum interesting. have you updated your clearml to the latest version ? we released new versions those days

Hope it will help 🤞 . Do not hesitate to ask if the error persists

Hi SparklingElephant70
The function doesn't seem to find any datasets which project_name matches your request.
Some more detailed code on how you create your dataset, and how you try to retrieve it, could help me to better understand the issue 🙂

but that i still not explaining why it was working 2 days ago and now it is not !
i am investigating, and will keep you updated

is it a task you are trying to access to or a dataset ? if you need to retrieve a task, you should use Task.get_task()

if i do that :
ds=Dataset.create(dataset_project='datasets',dataset_name='dataset_0')
it will result in the creation of 2 experiments :
results page: the task that corresponds to the script that launched the dataset creation - it will be in PROJECTS/datasets/.datasets/dataset_0 dataset page: the dataset itself : would be in DATASETS/dataset_0

what bother me is that it worked until yesterday, and you didnt changed your code. So the only thing i can think of is a bug introduced with the new sdk version, that was released yesterday. I am inverstigating with the sdk team, i will keep you updated asap ! 🙂

can you share with me an example or part from your code ? I might miss something in wht you intend to achieve

SubstantialElk6
Can you provide us a screenshot with a better resolution, to check the ratio between the total workers/active workers ?