This image is used for the task pods, not the agent pod.

The task pod (experiment) started reaching out to an IP associated with malicious activity. The IP was associated with 1000+ domain names. The activity was identified in AWS guard duty with a high severity level.

BoredHedgehog47 What is the pod container itself ?
EDIT:
Are you suggesting the default "ubuntu:18.04" is somehow contaminated ?
https://hub.docker.com/layers/library/ubuntu/18.04/images/sha256-d5c260797a173fe5852953656a15a9e58ba14c5306c175305b3a05e0303416db?context=explore

The domain names were bogus sounding businesses

Yes

Would using 22.04 Ubuntu still work in the task execution?

This is where we saw the malicious activity

Important part here: Malicious IP: 165.160.15.20

What malicious activity?

For task execution? It would work, yes, however it would take longer to install, I assume

The task pod (experiment) started reaching out to an IP associated with malicious activity. The IP was associated with 1000+ domain names. The activity was identified in AWS guard duty with a high severity level.

"title": "Unusual outbound communication seen from EC2 instance i-<> on server port 80.",

"ipAddressV4": "165.160.15.20", "organization": { "asn": "19574", "asnOrg": "CSC", "isp": "Corporation Service Company", "org": "Corporation Service Company" }, "country": { "countryName": "United States" }, "city": { "cityName": "" }, "geoLocation": { "lat": 37.751, "lon": -97.822 } }, "remotePortDetails": { "port": 80, "portName": "HTTP" }, "localPortDetails": { "port": 9134, "portName": "Unknown" }, "protocol": "TCP", "blocked": false, "localIpDetails": { "ipAddressV4": "10.32.2.13" }

Are you suggesting the default "ubuntu:18.04" is somehow contaminated ?

This is an official Ubuntu container (nothing to do with ClearML), this is Very Very odd...

yes

"additionalInfo": { "inBytes": "438", "localPort": "9134", "outBytes": "401", "unusual": "80", "value": "{\"inBytes\":\"438\",\"localPort\":\"9134\",\"outBytes\":\"401\",\"unusual\":\"80\"}", "type": "default" },

During the task pod runtime

The task pod (experiment) started reaching out to an IP associated with malicious activity. The IP was associated with 1000+ domain names. The activity was identified in AWS guard duty with a high severity level.

Can you share some details? I would really like to get to the bottom of that...