Hello, I Am Also Having Issues When Migrating Our Clearml Install, Specifically From 0.17 To 1.0.0 (I Was Doing It For 1.0.2 At First But I'Ve Seen In Some Messages That To Avoid Issues We Should First Upgrade To 1.0.0, Though It Didn'T Change Anything In

Unanswered

This is the log I get by using kubectl logs <api-server-pod-name> , not sure if that's the "proper" way to get it but here are the last 30 lines or so after clicking a few experiments:

[2021-06-28 13:58:03,358] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_next_task in 1072ms [2021-06-28 13:58:04,079] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 8ms [2021-06-28 13:58:04,173] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_task_tags in 1ms [2021-06-28 13:58:04,182] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_task_parents in 6ms [2021-06-28 13:58:04,184] [8] [INFO] [clearml.service_repo] Returned 200 for users.get_all_ex in 18ms [2021-06-28 13:58:04,218] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_types in 4ms [2021-06-28 13:58:04,227] [8] [WARNING] [clearml.service_repo] Returned 400 for tasks.get_all_ex in 0ms, msg=Validation error (invalid task field): path=hyperparams.General.n_estimators.value [2021-06-28 13:58:04,262] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_all in 2ms [2021-06-28 13:58:05,461] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_next_task in 1193ms [2021-06-28 13:58:06,205] [8] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms [2021-06-28 13:58:06,544] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 10ms [2021-06-28 13:58:06,566] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 28ms [2021-06-28 13:58:07,230] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 10ms [2021-06-28 13:58:07,230] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 6ms [2021-06-28 13:58:07,249] [8] [INFO] [clearml.service_repo] Returned 200 for users.get_all_ex in 7ms [2021-06-28 13:58:07,257] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_task_parents in 3ms [2021-06-28 13:58:07,262] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_task_tags in 1ms [2021-06-28 13:58:07,270] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_types in 3ms [2021-06-28 13:58:07,315] [8] [WARNING] [clearml.service_repo] Returned 400 for tasks.get_all_ex in 1ms, msg=Validation error (invalid task field): path=hyperparams.General.n_estimators.value [2021-06-28 13:58:07,917] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 4ms [2021-06-28 13:58:07,934] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 15ms [2021-06-28 13:58:08,429] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_all in 3ms [2021-06-28 13:58:09,057] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 11ms [2021-06-28 13:58:09,078] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 26ms [2021-06-28 13:58:09,541] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 9ms [2021-06-28 13:58:09,550] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 23ms [2021-06-28 13:58:09,561] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_next_task in 1052ms [2021-06-28 13:58:10,096] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 11ms [2021-06-28 13:58:10,103] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 28ms [2021-06-28 13:58:10,476] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_all in 2ms [2021-06-28 13:58:11,014] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 28ms [2021-06-28 13:58:11,027] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_all_ex in 37ms [2021-06-28 13:58:11,161] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 14ms [2021-06-28 13:58:11,172] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 5ms [2021-06-28 13:58:11,176] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_task_parents in 6ms [2021-06-28 13:58:11,179] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_types in 4ms [2021-06-28 13:58:11,186] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_task_tags in 2ms [2021-06-28 13:58:11,200] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_all_ex in 12ms [2021-06-28 13:58:11,201] [8] [INFO] [clearml.service_repo] Returned 200 for users.get_all_ex in 21ms [2021-06-28 13:58:11,623] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_next_task in 1140ms [2021-06-28 13:58:12,126] [8] [ERROR] [clearml.service_repo] Returned 500 for tasks.get_by_id_ex in 10ms, msg=Inconsistent data encountered in document: document=Output, field=model [2021-06-28 13:58:12,129] [8] [INFO] [clearml.service_repo] Returned 200 for tasks.get_by_id_ex in 6ms [2021-06-28 13:58:12,337] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 18ms [2021-06-28 13:58:12,369] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 17ms [2021-06-28 13:58:13,347] [8] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms [2021-06-28 13:58:14,656] [8] [INFO] [clearml.service_repo] Returned 200 for queues.get_all in 3ms [2021-06-28 13:58:15,025] [8] [INFO] [clearml.service_repo] Returned 200 for projects.get_all_ex in 9ms

  				
Posted 
	3 years ago

					More
				  		
  Report
		
					CleanSheep2
				
					0
					 × 1

191 Views

0 Answers

3 years ago

one year ago