Exactly, it should have auto-detected the package.
Or well, because it's not geared for tests, I'm just encountering weird shit. Just calling task.close() takes a long time
Thanks SuccessfulKoala55 ! Is this listed anywhere in the documentation?
Could I set an environment variable there and then refer to it internally in the config with the ${...} notation?
I see https://github.com/allegroai/clearml-agent/blob/d2f3614ab06be763ca145bd6e4ba50d4799a1bb2/clearml_agent/backend_config/utils.py#L23 but not where it's called 🤔
Maybe @<1523701827080556544:profile|JuicyFox94> can answer some questions then…
For example, what’s the difference between agentk8sglue.nodeSelector and agentk8sglue.basePodTemplate.nodeSelector ?
Am I correct in understanding that the former decides the node type that runs the “scaler” (listening to the given agentk8sglue.queue ), and the latter for any new booted instance/pod, that will actually run the agent and the task?
Read: The former can be kept lightweight, as it does no...
Yeah that works fine 😄 I just fetch it once to map argparse users to their IDs for later filtering.
The Slack Monitoring example should be updated btw, as they now use slack_sdk instead of slack (in the import statements)
I think so, it was just missing from the official documentation 🙂 Thanks!
yes, a lot of moving pieces here as we're trying to migrate to AWS and set up autoscaler and more 😅
I opened a GH issue shortly after posting here. @<1523701312477204480:profile|FrothyDog40> replied (hoping I tagged the right person).
We need to close the task. This is part of our unittests for a framework built on top of ClearML, so every test creates and closes a task.
We have a read-only user with personal access token for these things, works seamlessly throughout and in our current on premise servers... So perhaps something missing in the autoscaler definitions?
I dunno :man-shrugging: but Task.init is clearly incompatible with pytest and friends
One must then ask, of course, what to do if e.g. a text refers to a dictionary configuration object? 🤔
Latest (1.5.1 I believe?), full log incoming, but it's like I've posted elsewhere already 🤔
It just sets up the environment and immediately crashes when trying to run the code.
The setup itself is done correctly.
It failed on some missing files in my remote_execution, but otherwise seems fine now
Could you provide a more complete set of instructions, for the less inclined?
How would I backup the data in future times etc?
At any case, if we were upgrading from e.g. 4.0.0 to 4.1.2, this shouldn't have happened?
I have no idea what’s the difference, but it does not log the internal repository 😞 If I knew why, I would be able to solve it myself… hehe
The key/secret is also shared internally so that sounds like a nice mitigation actually!
Which environment variable am I looking for? I couldn't spot anything specifically in that environment variables page
Anything specific we should look into TimelyPenguin76 ?
That's up and running and is perfectly fine.
But there's nothing of that sort happening. The process where it's failing is on getting tasks for a project.
For the former (static-ish environment variables), just add:
environment {
VAR1: value1
VAR2: value2
}
to the agent’s clearml.conf
Because setting env vars and ensuring they exist on the remote machine during execution etc is more complicated 😁
There are always ways around, I was just wondering what is the expected flow 🙂
Yes it would be 🙂
Visualization is always a difficult topic... I'm not sure about that, but a callback would be nice.
One idea that comes to mind (this is of course limited to DataFrames), but think the git diff , where I imagine 3 independent section:
Removed columns (+ truncated preview of removed values) (see below) Added columns (+ truncated preview of removed values)
The middle column is then a bit complicated, but I would see some kind of "shared columns" dataframe, where each ...
AgitatedDove14 Basically the fact that this happens without user control is very frustrating - https://github.com/allegroai/clearml/blob/447714eaa4ac09b4d44a41bfa31da3b1a23c52fe/clearml/datasets/dataset.py#L191
Yup, latest version of ClearML SDK, and we're deployed on AWS using K8s helm
Also I can't select any tasks from the dashboard search results 😞