
Reputation
Badges 1
131 × Eureka!You need to set a specific port for your clearml server and then just set a rule in your reverse proxy (eg. nginx) to make point the specific route you want toward that port
@<1523701087100473344:profile|SuccessfulKoala55> here you go
Hey, I'm a SaaS user in PRO tier and I was wondering if it was a feature available on the auto-scaler apps so I could improve the cost-efficiency of my provisionned GCP A100 instances
alias
: "Alias of the dataset. If set, the ‘alias : dataset ID’ key-value pair will be set under the hyperparameters section ‘Datasets’"
I have to concede that I found that description a bit vague at first but if you check that https://clear.ml/docs/latest/docs/clearml_data/best_practices#organize-datasets-for-easier-access you see that:
"In cases where you use a dataset in a task (e.g. consuming a dataset), you can easily track which dataset the task is using by using ` Dataset.get...
Yup, so if I understand this is strictly an Enterprise feature and is not planned to be available in the Pro version ?
I would like instead of having to:
Fetch latest dataset to get the current latest version Increment the version number Create and upload a new version of the datasetTo be able to:
Select a dataset project by name Create a new version of the dataset by choosing what increment in SEMVER standard I would like to add for this version number (major/minor/patch) and upload
CostlyOstrich36 Having the same issue running on a remote worker, even tho the line works correctly on python interpreter and the component run correctly in local debug mode (but not standard local mode):
` File "/root/.clearml/venvs-builds/3.10/code/generate_dataset.py", line 18, in generate_dataset
time_range = pd.date_range(start=start_date, end=end_date, freq='D').to_pydatetime().tolist()
File "/root/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/pandas/core/indexes/date...
Thus the main difference of behavior must be coming from the _debug_execute_step_function
property in the Controller
class, currently skimming through it to try to identify a cause, did I provide you enough info btw CostlyOstrich36 ?
Oh, that's nice, if I import a model using InputModel do I still need to specify a OutputModel ?
There is a gap in the GPU offer on GCP and there is no modern middle-ground for a TPU with more than 16GB GRAM and less than 40GB, so sometime we need to provision a A100 to get the training speed we want but we don't use all the GRAM so I figured out if we could batch 2 training tasks on the same A100 instance we would still be on the winning side in term of CUDA cores and getting the most of the GPU-time we're paying.
The train.py
is the default YOLOv5 training file, I initiated the task outside the call, should I go edit their training command-line file ?
But the task appeared with the correct name and outputs in the pipeline and the experiment manager
Okay I confirm having default parameters fixes that issue, but kinda sad to have lost 3 days into that super weird behavior
I'm looking for the github issue so I can check the PR fixing that, do you have any info on that ?
Wll idk the scalars are not reported and I get this message ClearML Monitor: Could not detect iteration reporting, falling back to iterations as seconds-from-start
, i'll go open the pull request for that right away
And by extension is there a way to upsert a dataset by automatically creating an entry wich a incremented version or create it if it does not exists ? Or am I forced to do a get, check if the latest version is fainallyzed, then increment de version of that version and create my new version ?
When running with PipelineDecorator.run_locally()
I get the legitimate pandas error that I fixed by specifying the freq
param in the pd.date_range(....
line in the component:Launching step [generate_dataset] ClearML results page:
`
[STEP 1/4] Generating dataset from autocut logs...
Traceback (most recent call last):
File "/tmp/tmp2jgq29nl.py", line 137, in <module>
results = generate_dataset(**kwargs)
File "/tmp/tmp2jgq29nl.py", line 18, in generate_dataset
...
ClearML package version used: 1.9.1
ClearML Server: SaaS - Pro Tier
AnxiousSeal95 Okay it seems to work with a compute optimized c2-standard-4
instance
Ah no i cant since the pipeline is in its own dummy model and you cannot reattach pipelines to real projects so I must instanciate a dummy task just to attach the output model to the correct project
Hey SuccessfulKoala55 currently using the clearml
package version 1.7.1
and my server is a PRO SaaS deployment
Seems pretty critical tho, SuccessfulKoala55 is the 1.7.2
releasing soon ? ( I see the rc2 has been built 3 days ago)
So it seems to be an issue with the component parameter called in:
` @PipelineDecorator.pipeline(
name="VINZ Auto-Retrain",
project="VINZ",
version="0.0.1",
pipeline_execution_queue="Quad_VCPU_16GB"
)
def executing_pipeline(start_date, end_date):
print("Starting VINZ Auto-Retrain pipeline...")
print(f"Start date: {start_date}")
print(f"End date: {end_date}")
window_dataset_id = generate_dataset(start_date, end_date)
if name == 'main':
PipelineDec...
Okay, thank you for the explanations!
And additionally does the When executing a Task (experiment) remotely, this method has no effect).
part means that if it is executed in a remote worker inside a pipeline without the dataset downloaded the method will have no effect ?
I got some credentials issues to in some pipelines steps and I solved it using
task = Task.current_task()
task.setup_aws_upload(...)
It can allows you to explicitaly specify credentials
Hum, must be more arcane then, I guess the official support would be able to provide an answer, they usually answer within 24 hours