JitteryCoyote63 Great to hear 🙂
BTW:

Would it be possible to extend 

Task.init

  with a 

force_reuse

 that would enforce reusing these tasks

You can pass continue_last_task=True I think it should be equivalent to what you suggest

ohh, not really 😞 this is really low level editing the DB.
You might be able to forcefully edit the links (i.e. artifacts) on the Dataset (task)
Check if this works

from clearml.backend_api.session.client import APIClient
c = APIClient()
t = c.tasks.get_by_id("DATASET_UUID_HERE")
# you might need to loop over the artifacts
t.data.execution.artifacts[0].uri = "

"
c.tasks.edit(task=t.id, execution=t.data.execution, force=True)

there is a bug wherein both

Task.current_task()

and

Logger.current_logger()

return

None

.

This is not a bug this means something broke, the environment variable CLEARML_TASK_ID Has to be set inside the agent's process
How are you running it? (also log 🙂 , you can DM so it is not public here)

Thanks TrickyRaccoon92
I think it's about time we remove the survey link anyhow 🙂
I'll make sure it happens ..,

I double checked with the guys this issue was fixed in 1.14 (of clearml server). It should be released tomorrow / weekend

I think the limit is in the free tier hosting ...

Hi FiercePenguin76
It seems it fails detecting the notebook server and thinks this is a "script running".
What is exactly your setup?
docker image ?
jupyter-lab version ?
clearml version?
Also are you getting any warning when calling Task.init ?

Could you please add it, I really do not want to miss it 🙂

This is already part of the docker-compose file,
https://github.com/allegroai/clearml-server/blob/master/docker/docker-compose.yml

I'll try to create a more classic image.

That is always better, though I remember we have some flag to allow that, you can try with:
CLEARML_AGENT_SKIP_PIP_VENV_INSTALL=1 clearml-agent ...

Hi StickyBlackbird93
Yes, this agent version is rather old ( clearml_agent v1.0.0 )
it had a bug where pytorch wheel aaarch broke the agent (by default the agent in docker mode, will use the latest stable version, but not in venv mode)
Basically upgrade to the latest clearml-agent version it should solve the issue:
pip3 install -U clearml-agemnt==1.2.3BTW for future debugging, this is the interesting part of the log (Notice it is looking for the correct pytorch based on the auto de...

I think this is the main issue, is this reproducible ? How can we test that?

Hmm I just noticed:

'--rm', '', 'bash'

This is odd this is an extra argument passed as "empty text" how did that end up there? could it be you did not provide any docker image or default docker container?

Yea I know, I reported this

LOL, apologies these days it a miracle I still remember my login passwords 😉

JitteryCoyote63 Not sure how/why the X-Pack feature was on (it is not used by the system), but you can disable it with an environment variable in the docker-compose
xpack.security.enabled=falseShould solve the problem ...

From here:
https://medium.com/@ospaarmann/tidbits-solving-the-elasticsearch-x-pack-license-issue-in-docker-d15bb22d82fd

@<1720249421582569472:profile|NonchalantSeaanemone34>

dso = Dataset.create(
        dataset_project= project_name,
        dataset_name= dataset_name,
        parent_datasets=[parent_datasets_id],
)
dso = Dataset.get(
        dataset_project= project_name,
        dataset_name= dataset_name,
        only_completed=True,
        only_published=False,
        alias='latest',
)

why are you creating a dataset then getting a dataset on the same object?
it seems you are trying to upload...

Hi ShinyWhale52
This is just a suggestion, but this is what I would do:

1. use clearml-data and create a dataset from the local CSV file
   clearml-data create ... clearml-data sync --folder (where the csv file is)2. Write a python code that takes the csv file from the dataset and creates a new dataset of the preprocessed data
   ` from clearml import Dataset

original_csv_folder = Dataset.get(dataset_id=args.dataset).get_local_copy()

process csv file -> generate a new csv

preproces...

SubstantialElk6
The ~<package name with first name dropped> == a.b.c is a known conda/pip temporary install issue. (Some left over from previous package install)
The easiest way is to find the site-packages folder and delete the package, or create a new virtual environment
BTW:
pip freeze will also list these broken packages

We should probably add (set_task_type :))

JitteryCoyote63 is this still an issue?

👍

I have a process that cleans the 

/tmp

 each day,

WackyRabbit7 the files (configuration etc.) that are mapped into the containers are stored there.
They should clean themselves, that said, we have noticed that the services-mode skips this cleanup, and it will be solved on the next RC of clearml-agent.
Make sense ?

I was thinking such limitations will exist only for published

Published Task could not me "marked started" even when with force flag

Hi @<1578193378640662528:profile|MoodySeaurchin4>

but is it possible to log some metrics too, like rmse or the likes? If so, how would you do it?

Sure, I'm assuming this is part of the output ? if not, this means this is part of your code, and if this is the case then yes you should use collect_custom_statistics_fn
None
`collect_custom_statistics_fn({'rmse'...

But what I get with 

get_local_copy()

 is the following path: ...

Get local path will return an immutable copy of the dataset, by definition this will not be the "source" storing the data.
(Also notice that the dataset itself is stored in zip files, and when you get the "local-copy" you get the extracted files)
Make sense ?

Hi SubstantialElk6

but in terms of data provenance, its not clear how i can associate the data versions with the processes that created it.

I think DeliciousBluewhale87 ’s approach is what we are aiming for, but with code.
So using clearml-data from CLI is basically storing/versioning of files (with differentiable based storage etc, but still).
What ou are after (I think) is in your preprocessing code using the programtic Dataset class, to create the Dataset from code, this a...

Yes, which looks like a lot, but you only need to d that once.
Auto scheduler would make (1) redundant (as it would spin the instance up/down based on the jobs in the queue)

I figured out the problem...

Nice!

Unfortunately, the hyperparameters in configuration object seems to be superior to the hyperparameters in Hyperparameter section

Hmm what do you mean by that ? how did you construct the code itself? (you should be able to "prioritize" one over the over)

Hi PanickyMoth78

it was uploading fine for most of the day but now it is not uploading metrics and at the end

Where are you uploading metrics to (i.e. where is the clearml-server) ?
Are you seeing any retry logging on your console ?
packages/clearml/backend_interface/metrics/reporter.py", line 124, in wait_for_eventsThis seems to be consistent with waiting for metrics to be flushed to the backend, but usually you will see retry messages on your console when that happens

Hi GrittyCormorant73

When I archive the pipeline and go into the archive and delete the pipeline, the artifacts are not deleted.

Which clearml-server version are you using? The artifact delete was only recently added