Are you suggesting the default "ubuntu:18.04" is somehow contaminated ?

This is an official Ubuntu container (nothing to do with ClearML), this is Very Very odd...

Yes

The task pod (experiment) started reaching out to an IP associated with malicious activity. The IP was associated with 1000+ domain names. The activity was identified in AWS guard duty with a high severity level.

BoredHedgehog47 What is the pod container itself ?
EDIT:
Are you suggesting the default "ubuntu:18.04" is somehow contaminated ?
https://hub.docker.com/layers/library/ubuntu/18.04/images/sha256-d5c260797a173fe5852953656a15a9e58ba14c5306c175305b3a05e0303416db?context=explore

Important part here: Malicious IP: 165.160.15.20

"ipAddressV4": "165.160.15.20", "organization": { "asn": "19574", "asnOrg": "CSC", "isp": "Corporation Service Company", "org": "Corporation Service Company" }, "country": { "countryName": "United States" }, "city": { "cityName": "" }, "geoLocation": { "lat": 37.751, "lon": -97.822 } }, "remotePortDetails": { "port": 80, "portName": "HTTP" }, "localPortDetails": { "port": 9134, "portName": "Unknown" }, "protocol": "TCP", "blocked": false, "localIpDetails": { "ipAddressV4": "10.32.2.13" }

"additionalInfo": { "inBytes": "438", "localPort": "9134", "outBytes": "401", "unusual": "80", "value": "{\"inBytes\":\"438\",\"localPort\":\"9134\",\"outBytes\":\"401\",\"unusual\":\"80\"}", "type": "default" },

"title": "Unusual outbound communication seen from EC2 instance i-<> on server port 80.",

yes

The task pod (experiment) started reaching out to an IP associated with malicious activity. The IP was associated with 1000+ domain names. The activity was identified in AWS guard duty with a high severity level.

Can you share some details? I would really like to get to the bottom of that...

For task execution? It would work, yes, however it would take longer to install, I assume

Would using 22.04 Ubuntu still work in the task execution?

The domain names were bogus sounding businesses

The task pod (experiment) started reaching out to an IP associated with malicious activity. The IP was associated with 1000+ domain names. The activity was identified in AWS guard duty with a high severity level.

What malicious activity?

During the task pod runtime

This is where we saw the malicious activity

This image is used for the task pods, not the agent pod.