yes I did finally resolve this. If my memory serves, I think I needed to do some double checking that the clearml.conf file was in the proper location. It wasn’t where I thought I put it and I had to set all the proper environment variables in the container.

sounds good I’ll give that a shot real quick

clearml==1.4.1

Thanks! I think you were right on all counts there. That was my work around.

ah, I just ran:
helper = StorageHelper(url=s3_uri, base_url=s3_uri) context.log.info(f"helper: {helper}")and I got:
ValueError: Missing key and secret for S3 storage access ( ` )
File "/opt/venv/lib/python3.7/site-packages/dagster/core/execution/plan/utils.py", line 44, in solid_execution_error_boundary
yield
File "/opt/venv/lib/python3.7/site-packages/dagster/utils/init.py", line 383, in iterate_with_context
next_output = next(iterator)
File "/opt/venv/lib/python3.7...

I think I can set AWS_PROFILE . I have theoretically placed the clearml.conf in the correct place in the container: $HOME/clearml.conf . But I get an error that’s actually not very helpful, I still think the issue is getting boto3 credentials but I’m not sure I can prove it at this point. I posted the error in the community channel https://clearml.slack.com/archives/CTK20V944/p1653507863528969

SuccessfulKoala55 and AgitatedDove14 So setting my environment variable worked for my local runs, but it doesn’t work when running it from a container (in AWS ECS, for example)

I just checked for white space at the beginning or end of the url but there are none

well, I can get this exact object from my machine, from the same script through boto3, so I’m not sure where the disconnect is AgitatedDove14 SuccessfulKoala55

I suppose in the end I’m going to want to log the inference values back to snowflake as well … haven’t gotten to that part yet

I’m happy to use AWS’s ALB I was just even less sure about how to set that up at the time. I assume I need to setup a Target Group Attachement to the clearml-webserver ip address, correct?

That’s very helpful, thanks!. BTW when I enter that command I get No resources found in clearml namespace. Similarly if I run:
kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services clearml
I get:
Error from server (NotFound): services "clearml" not found I

I wonder if there’s an exception being caught by the final try/except block (below) but because I’m running it in dagster I’m not getting the logger output like I would normally:
` # Don't canonize URL since we already did it
try:
instance = cls(base_url=base_url, url=url, logger=logger, canonize_url=False, **kwargs)
except (StorageError, UsageError) as ex:
cls._get_logger().error(str(ex))
return None
except Exception as ex:
cls._get_logger().error("Failed creating stora...

looks like I accidentally placed my clearml.conf file in a non-standard place so I had to set the CLEARML_CONFIG_FILE environment variable. Thanks for your help AgitatedDove14 !!

I have already successfully printed model.name though I didn’t try the other ones.

For that reason I suspect there is a silent error trying to grab boto3 credentials.

in case it’s helpful, the url is an S3 URI

I had to modify the code a bit because I’m running this in a dagster pipeline but that print line put out: helper None

also, I can run this same code (same model) in a separate script (not in a docker container) with the AWS_PROFILE env variable set and load the model properly with joblib

s3://<bucket>/<foo>/local/<env>/<project-name>/v0-0-1/2022-05-12-30-9-rocketclassifier.7b7c02c4dac946518bf6955e83128bc2/models/2022-05-12-30-9-rocketclassifier.pkl.gz

so a little more clarity. I get nearly this same error when downloading the same object from boto3 if I try to get at it from an s3 resource like so:
` bad_s3_resource = boto3.resource("s3")
obj = bad_s3_resource.Object(bucket, key)
obj.get()["Body"]

botocore.exceptions.NoCredentialsError: Unable to locate credentials Only when I set the boto3 session with the appropriate profile does it work: sess = boto3.session.Session(profile_name="foo-profile")
s3_resource = sess.resource("s3")
...

Don’t know why I didn’t think of it earlier, but I set my env variable AWS_PROFILE=<foo-profile> and it copied it successfully.

no, I’m not

very nice, thanks!

That’s how I generate my raw input data is from a Snowflake query, then I do all the feature encoding/building etc.