Depending on where the agent is, the value of DATA_DIR
might change
so in my code, I'll use this environment variable to read from disk
In my use case I'm using an agent on the same mahcine I'm developing, so pointing this env var to the same venv I'm using for development, will skip the venv creation process from teh task requirements?
Especially coming from the standpoint of a team leader or other kind of supervision (or anyone who wants to view the experiment which is not the code author), when looking at an experiment you want to see the actual code
but the task pending says its in the queue
moreover I think I found a bug
but nowhere in the docs does it say anything about the permissions for the IAM
which permissions should it have? I would like to avoid full EC2 access if possible, and only choose the necessary permissions
the level of configurability in this thing is one of the best I've seen
AgitatedDove14 , I followed the instructions for updating the ClearML server, and the visualization stays the same
(it works now, with 20 GB)
So once I enqueue it is up? Docs says I can configure the queues that the auto scaler listens to in order to spin up instances, inside the auto scale task - I wanted to make sure that this config has nothing to do to where the auto scale task was enqueued to
pgrep -af trains
shows that there is nothing running with that name
Yes, I have a metric I want to monitor so I will be able to sort my experiments by it. It is logged in this manner
logger.report_scalar(title='Mean Top 4 Accuracy', series=ARGS.model, iteration=0, value=results['top_4_acc'].mean())
When looking at my dashboard this is how it looks
doesn't contain the number 4
AgitatedDove14 all I did was to cerate this metric as "last" and then turned on the "max" and "min" and then turned them off
I can't reproduce it now but:
I restarted the services and it didn't help I deleted the columns, and created them again after a while and it helped
FriendlySquid61
Just updating, I still haven't touched this.... I did not consider the time it would take me to set up the auto scaling, so I must attend other issues now, I hope to get back to this soon and make it work
I have a single IAM, my question is what kind of permissions I should associate with the IAM so that the autoscaler task will work
Does that mean that teh AWS autoscaler in trains, manages EC2 auto scaling directly without using the AWS built in EC2 auto scaler?
If the credentials don't have access tothe autoscale service obviously it won't work