Reputation
Badges 1
61 × Eureka!This was with using one task in a multiprocessing.pool and the next one in the main process. I switched to have all tasks in a separate process via ProcessPoolExecutor and now it runs fine 👍 (version 0.17.5)
I restarted the cleanup service. Now I get some messages like this:
2021-07-16 12:39:46,736 - clearml.storage - ERROR - Failed creating storage object file:// Reason: 'NoneType' object has no attribute 'replace'
2021-07-16 12:39:46,736 - clearml.Task - ERROR - Failed deleting None: 'NoneType' object has no attribute 'delete'
WARNING:root:Could not delete Task ID=eb11c92928af477e9e732d0cad47a57e, sequence item 0: expected str instance, NoneType found
any idea?
Some of the experiments are done on a GCP instance instead of the local server on which we also run ClearML. The experiments running on GCP report to the same local clearml server, but the IP address for clearml configured on the GCP instance is different (and then forwarded). Is this the problem?
Is there a way to test this? It seems my git user and token are correct. I can do git clone https://<NAME_TOKEN >:<ACCESSTOKEN> gitlab.com/mycompany/repo.git
However when starting the service it fails with:
cloning: git@gitlab.com:mycompany/repo.git
Using user/pass credentials - replacing ssh url
:mycompany/repo.git' with https url '
'
2021-01-18 20:04:08
User aborted: stopping task (3)
I dont see that option in my ~/clearml.conf?
I was looking for all the metric names, similar as what you get when clicking the '+ metric' in customize columns. But turns out I will implement it in a different way, not needed anymore
Is there also a method to get all the project names?
And is there an easy way to get all the metrics associated with a project?
I am using gitlab, I can create an access token. From the gitlab page:
"Personal Access Tokens
You can generate a personal access token for each application you use that needs access to the GitLab API."
However, now I have an access token, not an username/password. Is there also an option to authenticate with the access token?
Started clearml server again and now everything seems to work fine
With a name of 98 characters errors like 'munmap_chunk(): invalid pointer',
'double free or corruption (!prev)' or 'free(): invalid next size (normal) occur later in my script. But maybe this is not related to clearml but to the filesystem. Just wondering if there was a maximum length from the clearml side
yes, I wanted the confirmation that this is also a good solution for datasets with medical images
using auto_connect_frameworks={'pytorch': False} now
Yes, it did I think that makes sense
I created it with clearml-init, nothing special. It looks like
# ClearML SDK configuration file api { # Notice: 'host' is the api server (default port 8008), not the web server. api_server:
web_server:
files_server:
# Credentials are generated using the webapp,
`
# Override with os environment: CLEARML_API_ACCESS_KEY / CLEARML_API_SECRET_KEY
credentials {"access_key": "NMWOE5C3RGTX473M9D3M", "secret_key": "L@G50jO+TJ23#8Eerp1E$4y=elUt11P!BL...
btw the same happens when I try this on localhost
# ClearML SDK configuration file api { # Notice: 'host' is the api server (default port 8008), not the web server. api_server:
web_server:
files_server:
# Credentials are generated using the webapp,
# Override with os environment: CLEARML_API_ACCESS_KEY / CLEARML_API_SECRET_KEY
upgrading to? 2020.3.3 is the latest version? https://www.jetbrains.com/pycharm/download/other.html
This code snippet does reproduce:
` import os
from clearml import Task
parameters = {
'experiment': {
'project_name': 'test',
'experiment_name': 'test_exp',
'tags': []
}
}
Task.set_offline(True)
if not Task.is_offline():
os.environ['CLEARML_NO_DEFAULT_SERVER'] = '1'
task = Task.init(
project_name=parameters['experiment']['project_name'],
task_name=parameters['experiment']['experiment_name'],
task_type='testing',
tags=parameters['experiment...
` # Never save to clearml demo server
Task.set_offline(parameters['experiment'].get('offline', False))
if not Task.is_offline():
os.environ['CLEARML_NO_DEFAULT_SERVER'] = '1'
task = Task.init(
project_name=parameters['experiment']['project_name'],
task_name=parameters['experiment']['experiment_name'],
task_type=task_type,
tags=parameters['experiment']['tags'],
auto_connect_arg_parser=True,
auto_connect_streams=True,
auto_connect_frameworks=True,
auto_resour...
I enqueue to service to the services queue, not done anything myself with agents
Yes other then the link it generates it works fine
Yes, I add these metrics as extra columns and then I sort them. I want to know which experiments performs best in daylight for example or which during night. Therefore I think a is not the right choice