and also take a look into development.apply_environment

I see, I suspect you might need to do some migrations

And when you run it again under exactly the same circumstances it works fine?

SmallDeer34 , and they still have the same colors when you maximize the graph?

Also, what happens when you try to access the dataset through the code?

@<1559711623147425792:profile|PlainPelican41> , I think you should switch to using pipelines from decorators for easier management of your tasks/code. Creating pipelines from pipelines according to your scenario (difficulty managing code) would only lead to more difficulty tracking the code executions - How would you show pipelines of pipelines results? How many depths would be enabled? How should data aggregation work? This introduces many problems without any benefit.

I think restructuring...

@<1544853721739956224:profile|QuizzicalFox36> , yes 🙂

Can you provide a snippet to try and reproduce?

EcstaticBaldeagle77 , please lower all the dockers then run the following commands and then try raising them again.
` sudo mkdir -p /opt/clearml/data/elastic
sudo mkdir -p /opt/clearml/data/mongo/db
sudo mkdir -p /opt/clearml/data/mongo/configdb
sudo mkdir -p /opt/clearml/data/redis
sudo mkdir -p /opt/clearml/logs
sudo mkdir -p /opt/clearml/config
sudo mkdir -p /opt/clearml/data/fileserver

sudo chown -R 1000:1000 /opt/clearml `
Also, where did you find the instructions with 'trains' in it?

If it works on two computers and one computer is having problems then I'll be suspecting some issue with the computer itself. Maybe permissions or network issues

My guess other agents are sitting on different machines, did you verify that the credentials are the same between the different clearml.conf files? Maybe @<1523701087100473344:profile|SuccessfulKoala55> might have an idea

https://clear.ml/docs/latest/docs/references/sdk/automation_controller_pipelinecontroller#get_processed_nodes

WackyRabbit7 ,I am noticing that the files are saved locally, is there any chance that the files are over-written during the run or get deleted at some point and then replaced?

Also, is there a reason the files are being saved locally and not at the fileserver?

I couldn't manage to reproduce it on my end. But also in my cases it always saves the files to the fileserver. So I'm curious what's making it save locally in your case

StickyCoyote36 , I think that is the solution. Is there a reason you want to ignore the "installed packages"? After all those are the packages that the task was ran with.

Hi @<1546303254386708480:profile|DisgustedBear75> , there are a few reasons remote execution can fail. Can you please describe what you were trying to do and please add logs?

Backup what you currently have

And for future reference - Always a good thing to do a backup before upgrading elastic versions (or upgrades in general). Periodic upgrades are also advised 🙂

I'm not sure, maybe @<1523701087100473344:profile|SuccessfulKoala55> might have an idea 🙂

Hi @<1575294289515122688:profile|JoyousMole49> , it looks like you are over your usage quota. Check in the settings page to see your uages

I think this is what you're looking for

Also, are you maybe somehow tunneled into the EC2 instance and therefor may be seeing it as localhost?

Hi SubstantialElk6 , maybe SuccessfulKoala55 might have more input on this 🙂

Is it possible to do something so that the change of the server address is supported and the pictures are pulled up on the new server from the new server?

Do the links point to a bucket or the fileserver?

I'm not sure pipeline of pipelines would work out of the box without some hacks. What is your use case though, why do you need a pipeline of pipelines? In the end it would abstract into a single large pipeline anyways

Can you look in the UI if the execution parameters were logged?

ScaryLeopard77 , Hi! Is there a specific reason to the aversion from pipelines? What is the use case?

"continue with this already created pipeline and add the currently run task to it"

I'm not sure I understand, can you please elaborate? (I'm pretty sure it's a pipelines feature)

Hi @<1556812486840160256:profile|SuccessfulRaven86> , just to make things easier, can you comment out these 3 lines in the config file? This will cause the sdk to have default behavior. Afterwards try with store_code_diff_from_remote: false

What do you see in the uncommitted changes section of the experiment?

UnevenDolphin73 , Hi!

I would avoid using cache_dir since it's only a cache. I think using S3 or the fileserver with Task.upload_artifact() is a nice solution

Also what do you mean by 'augment' arguments?

SwankySeaurchin41 , I think you don't need to connect pipelines. Think of pipeline as a DAG execution. You can build it anyway you want 🙂

So in which scenario do you want to keep those folders as artifacts and where would you like to store them?