Reputation
Badges 1
72 × Eureka!i attach train.py here,
and to run it i do python src/train.py
it seems if i access with my dns cannot see
and if access with ip address can see
my case more like there is a task/process that running but somehow its takes too long to completed. it can be because connection issue forgot to put connection timeout, a problem connection database, etc that makes status still running, but its traped in a situation like that.
so i want to force shutdown a task to failed if that happen
Thanks @<1523701205467926528:profile|AgitatedDove14> , right now i just use trigger to send notification and do it manually. ClearML Superb!
hi @<1523701087100473344:profile|SuccessfulKoala55> , it solved! thanks for information CLEARML_ENV
! I just accidently write environment varible CLEARML_ENV on every clearml-agent.conf. π
yes, so far i know, if we want to upload dataset on clearml, we need provide local_path to data, then clearml will upload to the platform.
my data not on local, but s3 bucket.
is there a way to point s3 url ? my currently workflow is download my data from s3 bucket to local, then upload to clearml.
Hi AgitatedDove14 , is the Dataset.get
will take all child too?
yup,
example just need choosing between SGD, Adam, AdamW on optimizer field
maybe accidently install my custom solution on this https://github.com/muhammadAgfian96/clearml/commit/01db9aa40537a6c2f83977220423556a48614c3a at that time. so i said the test is passed.
Hi @<1523701070390366208:profile|CostlyOstrich36>
i attach for complete log
here my structure:
.
βββ app
β βββ backend
β βββ frontend
βββ assets
β βββ demo-app-sample.png
β βββ workflow.png
βββ config
β βββ clearml.conf
β βββ list_models.py
β βββ list_optimizer.py
β βββ __pycache__
β βββ train_config.py
βββ docker
β βββ Dockerfile
β βββ Dockerfile.app
β βββ requirements.prod.txt
β βββ requirements.train.txt
β βββ requirements.txt
βββ lightning_logs
βββ Mak...
Hi, @<1523701070390366208:profile|CostlyOstrich36> ,yes! correct! how to achive that? it will save my storage.
https://github.com/mert-kurttutan/torchview
maybe can try this one, and can send to logger clearml at the end.
my config is same like issue #763
` import clearml
from clearml import StorageManager, Dataset
from rich import print
version_clearml = clearml.version
manager = StorageManager()
print(f'clearml: {version_clearml}')
try:
minio_s3_url = 'x/x/x/x/x/x/x'
print('\n-------------download folder-------------')
target_folder = manager.download_folder(
local_folder='tmp',
remote_url=f' '
)
except Exception as e:
print(e)
print('FAILED: download fold...
Hi SmugDolphin23 , i have try 1.8.4rc1, and yeah its working! Thanks!
Hi @<1523701205467926528:profile|AgitatedDove14> , Thanks for rresponse!
this my simple code to test scheduler
import datetime
from clearml.automation import TaskScheduler
def test_make():
print('test running', datetime.datetime.now())
if __name__ == '__main__':
task_scheduler = TaskScheduler(
sync_frequency_minutes=30,
force_create_task_name='controller_feedback',
force_create_task_project='Automation/Controller',
)
print('\n[utc_timestamp]...
oh okay, so i need to set that to path ssd, yeah?
is it this one? or there is
docker_internal_mounts {
sdk_cache: "/clearml_agent_cache"
apt_cache: "path/to/ssd/apt-cache"
ssh_folder: "/root/.ssh"
pip_cache: "path/to/ssd/clearml-cache/pip"
poetry_cache: "/mnt/hdd_2/clearml-cache/pypoetry"
vcs_cache: "path/to/ssd/clearml-cache/vcs-cache"
venv_build: "path/to/ssd/clearml-cache/venvs-builds"
pip_download: "path/to/ssd/cle...
wow, okay, i think will move all logs/plot/artifacs to my storage s3. Thanks! really helpful!
i see thanks for the answer, i will read that reference.
still i need do this?dataset.upload() dataset.finalize()
i have another question,
if we have uploaded data clearml, how we add data?
this is my way right now.
dataset = Dataset.create( dataset_project=metadata[2], dataset_name=metadata[3], description=description, output_uri=f"
", parent_datasets=[id_dataset_latest] )
Hi @<1523701070390366208:profile|CostlyOstrich36> , i think can try this to run it as standalone:
i see okay thanks
Hi @<1523701070390366208:profile|CostlyOstrich36> ,
i use vpn and set location in US still cannot access too.
when i set location to Germany, it can access.
any idea to solve this from user side?
remove this params will solve use_current_task=True,
Hi @<1523701070390366208:profile|CostlyOstrich36> , thanks for response, sorry for late replay,
this is my configuration in yaml, i facing difficulty when there is params in list. somehow, form to display bunch list not easy to see. do you have suggestion? Thanks!
download-data:
dataset_train:
-
-
-
dataset_test:
-
-
-
train:
data:
batch: 4
input_size: 224
split:
t...
i am using dictionary, more convinient for me and can categorize each params.
Thanks, hope that feature will ready soon!
i see,
thanks for clarify. i just want to find other solutions to storing secret value. rightnow i just storing secret value on env in clearml.conf in my workers. but it will complicated if there is new value, i need update workers conf and redeploy workers.
Iβm running the agent in βpipβ mode. I need to fetch certain secret values, which would be best done using Python code. If I incorporate it into the script (repository), others could deduce the path to retrieve the environment or secret value. Storing the environment variables in the clearml.config isnβt very flexible either.
the current my solution is upload my config to s3, and the pipeline will download it and read it when execute. but its decrase flexiblity.