Hi EcstaticMouse10 ,
On what OS are you running?
CluelessElephant89 , did you run the vm.max_map_count
command for elastic? Also what amount of RAM memory do you have on the machine you're running on?
CluelessElephant89 try the elastic search logs clearml-elastic
I'll clarify - on the server you have two parts - the clearml folders where all the mongo/elastic/redis data sits and you have the dockers. So, downgrading would mean using previous version dockers. However if you don't have a backup of your data I don't suggest you do this since data might become corrupt (mismatching elastic versions is bad for elastic)
Hi SoggyBeetle95 , did you try rolling back and re-trying the upgrade? Do you have a backup of the data?
And for future reference - Always a good thing to do a backup before upgrading elastic versions (or upgrades in general). Periodic upgrades are also advised 🙂
SoggyBeetle95 , in the ClearML UI you should see a small notification at the top when there is a new version available
Backup what you currently have
It looks like a connection issue, it's failing to pull the docker image
Hi @<1523701295830011904:profile|CluelessFlamingo93> , I think you can also control the agent sampling rate (to sample queue every 10 or 20 seconds instead of 5 for example)
@<1719524641879363584:profile|ThankfulClams64> , are logs showing up without issue on the 'problematic' machine?
Hi @<1639799308809146368:profile|TritePigeon86> , I'm not sure I understand, can you please elaborate on exactly what you're doing?
Hi @<1670964687132430336:profile|SpicyFrog56> , can you please add the full log?
Hi @<1578918150261444608:profile|RoundJellyfish71> , you can certainly add models directly to the registry.
I think this is what you're looking for 🙂
None
My bad, if you set auto_connect_streams to false, you basically disable the console logging... Please see the documentation:
auto_connect_streams (Union[bool, Mapping[str, bool]]) – Control the automatic logging of stdout and stderr.
Hi @<1562973083189383168:profile|GrievingDuck15> , do you have a standalone snippet that reproduces this error?
Hi @<1751777178984386560:profile|ConfusedGoat3> , I think you might need to run some migration script on the database, basically changing the paths of the artifacts registered to the new IP
Hi @<1570220858075516928:profile|SlipperySheep79> , you can set various cache limitations in clearml.conf
. The issue you encountered is specifically regarding Datasets? If that is the case, I think this is the section that you're looking for - None
I think you'd have to write it yourself. Basically, the artifact paths in experiments are saved in mongo. You would need to write a script that would modify those values in Mongo directly
Hi @<1671689448707854336:profile|GiganticBee35> , the suggested method would be triggering it via the REST API. If you open developer tools (F12) while in the web UI, you will see everything the web UI is sending to the backend so launching jobs/pipelines via the web UI will give you full visibility into what you need to automate
Hi @<1555362936292118528:profile|AdventurousElephant3> , if you clone/reset the task, you can change the logging level to 'debug'
Hi @<1545216070686609408:profile|EnthusiasticCow4> , what version of clearml
& server are you using?
I suggest you take a look at the examples here and play with them a bit 🙂
None
Hi @<1545216070686609408:profile|EnthusiasticCow4> , generally speaking, pipelines are a special type of task. When you write steps using decorators you don't have to add the task init. However you can also build pipelines using existing tasks in the system, where those were created with task.init
Hi @<1702130048917573632:profile|BlushingHedgehong95> , where did you see this?
I'm not aware of an ability to inject directly from an sql database. You can automate it by dumping from sql to csv, I assume
What exactly are you looking to set up?
I was referring to the SDK python package (ClearML) and ClearML-Agent
Hi SweetHippopotamus84 , what version of ClearML are you using? Also, do you have a small snippet to play with?
I'll take a large snippet too 😛
Do you have any idea what's the source of this?TypeError: __init__() got an unexpected keyword argument 'configurations'