I guess. maybe I'll just stash.
Not a big fan of notebooks X git so I attempt to push these less frequently.
SuccessfulKoala55 when running a changed script without adding committing and pushing the changes are not applied. I do have '# WARNING! git diff too large to store, clear this section to execute without it.' due to unsaved notebook changes.
I guess that's the issue?
If it's just a UI issue, than this is still unsolved.
The execution uses an older version that is committed.
TimelyPenguin76 good morning,
From the CLI. Yes, I see it.
This specific issue doesn't concern the output URI (although I did a run both with it and without) as I'm trying to load a config file that's being saved locally using manager.upload_file.
SuccessfulKoala55 Is this example correct:
https://allegro.ai/docs/examples/examples_storagehelper/#uploading-a-file
manager.upload_file(local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder")
I'm not specifying a filename under remote URL, just like in the example.
Were you able to replicate the issue with task?
Locally I had no issues finding loading etc., didn't try to load it into memory remotely. The thing is, since this is stuck at training (with msg ‘TRAINS Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start’) I will probably have no idea whether it was loaded or not.
manager.upload_file(local_file=str(output_dir/"config.json"), remote_url=remote_url+'bucket/')This is my specific upload. I wanted to make sure the example in the documentation is accurate.
SuccessfulKoala55 Conclusions:
The example in the documentation is missing a filename at the end of remote URL (an error I gotten locally when I tried to upload). When using a trains task to track my run AND changing my scripts output directory, I get: 'TRAINS Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start' - and that's it, it's stuck on running and no error. When running the same script without the trains task it works.
https://github.com/ThilinaRajapakse/simpletransformers#minimal-start-for-multilabel-classification that's what I'm using (just with DistilBERT)
The error doesn't appear when not using the storage manager.
The files were crearted, and the one I needed uploaded to storage.
yes, when I comment out the storage manager no error.
You can probably replicate this yourself. I'm using simple transformers library for quick benchmarking.
Just run one his examples (I'm using multilabel clasification): https://github.com/ThilinaRajapakse/simpletransformers but change the output_dir to something else. when you don't track this works (no task.init), if you track this get stuck.
I'll check.
I do however expect to see an error message when something isn't working, this just got stuck.
No error.
I didn't check the contents on the remote machine. However, when you run it locally it creates a bunch of files (text, model etc.)
manager.upload_file(local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder")