For example--packages "tqdm>=2.1" "scikit-learn"
Do you see any errors in the ES container?
I might be wrong. Did you try 1.9.1?
SucculentBeetle7 , can you please give an example of the pathing for an artifact?
Is there a specific reason you would want them executed on the same machine? Cache?
I think it depends on your code and the pipeline setup. You can also cache steps - avoiding the entire need to worry about artifacts.
I see, thanks for the input!
Hi @<1523701132025663488:profile|SlimyElephant79> , to answer your questions:
-
Does ClearML store any dataset outside of this S3 storage (and local storage) for preview or compression purposes?Some preview data might be stored inside mongodb (if it's a table for example). This of course can be disabled so no data/previews are exposed to the ClearML backend.
-
Are any data packets related to model or data versioning routed through the API server? (We are pretty confident it shouldn't, but j...
Hi @<1774245260931633152:profile|GloriousGoldfish63> , this feature is waiting enablement on clearml-serving side and will be supported in the next release
GrievingTurkey78 , what timeout did you set? Please note that it's in seconds so it needs to be a fairly large number
Hi @<1539417873305309184:profile|DangerousMole43> , I'm afraid this is not configurable currently. What is your use case?
Hi @<1523711002288328704:profile|YummyLion54> , it hasn't been added to the PRO yet 🙂
Hi @<1724960464275771392:profile|DepravedBee82> , I believe this is not currently supported
Hi @<1570583227918192640:profile|FloppySwallow46> , please don't @ the entire channel for help 🙂
If a task is in pending it means that no agent picked it up yet. Maybe the agent is unavailable or the process crashed. Check in that direction
Hi @<1523702251011444736:profile|ScaryBluewhale66> , I think the only port you need is the one that is allocated to the apiserver
@<1806135353375985664:profile|ColossalHawk50> , you basically run the agent outside of a docker container and then use the --docker flag to run it in docker mode. Makes sense?
ShinyLobster84 , can you please elaborate? I'm guessing your jupyter notebook? How are you running it? Did you run the experiment from it?
The "template" task
Hi @<1523704157695905792:profile|VivaciousBadger56> , can you add a screenshot of what you're talking about?
Hi WittySeal28 , can you please paste here the entire console of what you did and what you pasted?
This is very strange and unrelated to ClearML:
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '/run/user/1003/vscode-git-7ea59f7d55.sock'
}
Missing or invalid credentials.
Error: connect ECONNREFUSED /run/user/1003/vscode-git-7ea59f7d55.sock
at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
errno:...
VexedCat68 , what errors are you getting? What exactly is not working, the webserver or apiserver? Are you trying to access the server from the machine you set it up on or remotely?
Hi @<1547028074090991616:profile|ShaggySwan64> , so the issue is when writing to the files server? Is it possible that the machine itself is having a hard time to write the data?
Hi @<1632913939241111552:profile|HighRaccoon77> , the most 'basic' solution would be adding a piece of code at the end of your script to shut down the machine but obviously it would be unpleasant to run locally without Task.execute_remotely() - None
Are you specifically using Sagemaker? Do you have any api interface you could work with to manipulate shutdown of machines?
Hi @<1610083503607648256:profile|DiminutiveToad80> , can you please add a full log of the run?
Ok, will do once I get back to the office, thanks for the heads up! 🙂
Did anything change in your configurations? In the previous version there was no such issue? Is the agent version the only change?