Fantastic! Public S3 Buckets and How to Find Them

Unwanted public S3 buckets are a continuous threat. They have been (and still are) causing havoc all over the web. There are several tools out there to help your company with finding public S3 buckets. They are almost all standalone scripts or lambda functions that query the AWS APIs via some sort of SDK (Python, Node.js, etc.).

But when centralized security is implemented, as we have done so at Auth0, this task can be performed using a data lake or any sort of system/service where logs are aggregated, analysed, and acted upon. In that regard, the first source for your AWS events is CloudTrail.

Digging around the Internet we didn't find enough resources that explained to us the different ways an S3 bucket can be made public and how to detect it in raw CloudTrail logs, so we started playing around, running tests and building queries to find that out. This blog post will guide you through our process, our findings, and our solutions.

Bitdefender compiled a list of the 10 worst Amazon S3 breaches.

Source

The Context

Before getting into the technical details, let’s have an overview of the context in which those tests were running, which technologies were involved, and how we linked them all together.

We’ve tested the creation of buckets in two ways: via the AWS command line interface (

aws

CLI) and the web console. We’ve also tested policy changes, both access control lists (ACLs) and individual permissions.

We wanted to cover all the possible ways that a user, malicious or not, could use to create a public S3 bucket: by mistake, for data exfiltration, or for command and control (yes, you can use it even for that, my dear pentester friends).

"Data exfiltration, also called data extrusion, is the unauthorized transfer of data from a computer." (TechTarget)

As a Security Information and Event Management (SIEM) solution we’re working with Sumo Logic. We send all logs to it and we’ve designed the CloudTrail logs coming from every AWS account to be collected in a centralized S3 bucket that is “drained” by the Sumo Logic collector and organized in the source category named

cloudtrail_aws_logs

.

“We wanted to cover all the possible ways that a user, malicious or not, could use to create a public S3 bucket. Learn more on how to do it.”

Tweet This

The S3 Bucket Permission Model

Let's have a quick overview of the type of permissions an S3 bucket can have and how they can be used to make one public.

For a complete and detailed explanation, we highly recommend reading the official AWS documentation.

Bucket ACLs

The easiest way to setup a bucket public is to use the canned policy

public-read

on the bucket. By default, both via the web console and the command-line interface (CLI), the buckets are created with an ACL

private.

Canned ACLs provide an easy and quick way to set up global permissions in one shot. However, one can apply specific policies to grant or deny access to specific entities.

There are five permissions that can be granted:

• READ

• READ_ACP

• WRITE

• WRITE_ACP

• FULL_PERMISSION

READ

,

WRITE

and

FULL_PERMISSION

are definitely self-explanatory and apply to every object on the bucket (and the bucket itself). The Adjacent Channel Protection (ACP) part of the other permissions are related to the ACLs: with those permissions granted, an user can read and/or write the ACL (but not the objects).

Those policies go together with the entity to which they are attached. More specifically, the Grantee. A Grantee is an object who holds three or four basic pieces of information (depending on the type): the type of the grantee (

CanonicalUser

or

Group

), the XML XSI schema and the ID (in case of a type CanonicalUser) or the URI (in case of a type Group). CanonicalUser types also carry a

DisplayName

property, not present in the Group ones.

Group can be one of the following:

• AuthenticatedUsers
• AllUsers
• LogDelivery

As a security best practice, you should watch out for

AuthenticatedUsers

and

AllUsers

groups.

Bucket Policies

Policies are a tricky way to allow/disallow actions on the bucket’s object. Even if the bucket is created with a canned ACL that sets it to private, by adding a very basic policy we can make the whole bucket public. The policy to do that is as simple as:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "ABC",
    "Effect": "Allow",
    "Principal": { "AWS": "*" },
    "Action": ["s3:GetObject"],
    "Resource": ["arn:aws:s3:::bucket_name/*" ]
  }]
}

AWS is able to recognize the effect of such policy and will show the “Public” tag on the bucket. While detecting the above policy is doable, detecting these kind of scenarios is a tricky task. Evaluating the impact that a policy could have on the bucket and its objects is something that cannot be done on a SIEM, but it requires external tools to simulate the policy, analyze the results, and evaluate the risks.

As Rich Mogull points out on his article on how S3 buckets become public, the way permissions are evaluated is bucket policies first and then bucket ACLs (both canned and extended). Of course, an explicit deny always takes precedence, regardless of where it is stated.

“For S3 buckets, the way permissions are evaluated is bucket policies first and then bucket ACLs. Learn more how this affects your security in the cloud.”

Tweet This

Object ACLs

S3 objects do inherit parent bucket’s permissions, but they can also have their own ACL that can bypass such permissions. You can make single objects public while the bucket ACL states it’s private, although to access that object one must know the full path to it. While this is a security concern that you should address, it’s out of the scope of this article.

The Tests

We were interested in knowing how many ways those policies can be applied and how they show up in Cloudtrail, so we ran several tests with different parameters to experiment and build up our Sumo Logic detection query.

Detecting Bucket ACLs: The Command Line Way

First shot was to alert on something we were familiar with: canned ACLs. The easiest way to create a public bucket with such policies is via the command line.

We used the following CLI command to create a bucket with a public-read policy:

$ aws s3api create-bucket --acl public-read --bucket davide-public-test --region us-east-1

And this is what we got in the trail:

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROADBDBDBDBDBDBDBDBD:1544653190000000000",
    "arn": "arn:aws:sts::107000000000:assumed-role/AssumedRole/1544653190000000000",
    "accountId": "107000000000",
    "accessKeyId": "ASIARRDBDBDBDBDBDBDB",
    "sessionContext": {
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2018-12-12T22:19:53Z"
      },
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROADBDBDBDBDBDBDBDBD",
        "arn": "arn:aws:iam::107000000000:role/AssumedRole",
        "accountId": "107000000000",
        "userName": "AssumedRole"
      }
    }
  },
  "eventTime": "2018-12-12T22:22:56Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "CreateBucket",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "73.xx.xx.xx",
  "userAgent": "[aws-cli/1.11.160 Python/2.7.10 Darwin/18.2.0 botocore/1.7.18]",
  "requestParameters": {
    "x-amz-acl": [
      "public-read"
    ],
    "bucketName": "davide-public-test"
  },
  "responseElements": null,
  "requestID": "0BCE63ACB970D013",
  "eventID": "76b81a5d-3e8f-4923-b065-dff822fe0af9",
  "eventType": "AwsApiCall",
  "recipientAccountId": "107000000000"
}

There are few interesting things to note here:

• Event name is

  CreateBucket

  (as expected)

• requestParamenters.x-amz-acl

  is set to

  public-read

  , which is the canned ACL we specified on the command line
• The username is the assumed role (

  AssumedRole

  ) and not the user who assumed that role

Building a query which catches that is pretty straight forward:

(_sourceCategory=cloudtrail_aws_logs AND ("CreateBucket"))
| json "eventName", "requestParameters.bucketName", "requestParameters.x-amz-acl", “userIdentity.accountId", "userIdentity.arn", "userIdentity.sessionContext.sessionIssuer.userName", "sourceIPAddress", "eventTime" as event, bucket, acl, account, arn, user, src_ip, datetime nodrop
| where acl matches "*public*"
| if (isNull(acl), "null", acl) as acl
| count by datetime, account, account_name, bucket, acl, user, src_ip, arn, event
| fields -_count
| sort by datetime

We specified in the first row the “CreateBucket” event action to speed up the filtering process. We then parsed some of the fields to be able to apply some logic. You can notice the

requestParameters.x-amz-acl

which is named

acl

and the following

where

clause:

where acl matches "*public*"

. In this way we are not only detecting the

public-read

ACL, but also the

public-write

and

public-read-write

.

Note that, since the username is the assumed role, the real user who assumed that role is not exposed, making attribution a little bit harder (depending on how your team structures IAM roles and users).

Detecting Bucket ACLs: The Web Console Way

Funny enough, you cannot create a bucket with a canned ACL straight from the web console wizard. To apply a canned ACL, first you have to create the bucket and after that you have to manually set the “Everyone” permission on it. By enabling the following properties, you can get:

• “List” applies the

  public-read

  ACL
• “Write” applies the

  public-write

  ACL
• “List” and “Write” apply the

  public-read-write

  ACL

This generates the following create bucket event:

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROADBDBDBDBDBDBDBDBD:davideruser",
    "arn": "arn:aws:sts::107000000000:assumed-role/AssumedRole/davideruser",
    "accountId": "107000000000",
    "accessKeyId": "ASIARRDBDBDBDBDBDBDB",
    "sessionContext": {
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2018-12-18T16:10:22Z"
      },
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROADBDBDBDBDBDBDBDBD",
        "arn": "arn:aws:iam::107000000000:role/AssumedRole",
        "accountId": "107000000000",
        "userName": "AssumedRole"
      }
    }
  },
  "eventTime": "2018-12-18T16:21:48Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "CreateBucket",
  "awsRegion": "us-west-2",
  "sourceIPAddress": "73.xx.xx.xx",
  "userAgent": "[S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.467 Linux/4.9.124-0.1.ac.198.71.329.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.192-b12 java/1.8.0_192]",
  "requestParameters": {
    "CreateBucketConfiguration": {
      "LocationConstraint": "us-west-2",
      "xmlns": "http://s3.amazonaws.com/doc/2006-03-01/"
    },
    "bucketName": "davide-public-test-02"
  },
  "responseElements": null,
  "additionalEventData": {
    "vpcEndpointId": "vpce-83a16cea"
  },
  "requestID": "E4BA1F14D9D2E18F",
  "eventID": "6d772d8c-c0b6-471c-b9a5-986d618788f8",
  "eventType": "AwsApiCall",
  "recipientAccountId": "107000000000",
  "vpcEndpointId": "vpce-83a16cea"
}

Followed by the change in the S3 bucket policy:

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROADBDBDBDBDBDBDBDBD:davideruser",
    "arn": "arn:aws:sts::107000000000:assumed-role/AssumedRole/davideruser",
    "accountId": "107000000000",
    "accessKeyId": "ASIARRDBDBDBDBDBDBDB",
    "sessionContext": {
      "attributes": {
        "mfaAuthenticated": "true",
        "creationDate": "2018-12-18T16:10:22Z"
      },
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROADBDBDBDBDBDBDBDBD",
        "arn": "arn:aws:iam::107000000000:role/AssumedRole",
        "accountId": "107000000000",
        "userName": "AssumedRole"
      }
    }
  },
  "eventTime": "2018-12-18T16:21:49Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "PutBucketAcl",
  "awsRegion": "us-west-2",
  "sourceIPAddress": "73.xx.xx.xx",
  "userAgent": "[S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.467 Linux/4.9.124-0.1.ac.198.71.329.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.192-b12 java/1.8.0_192]",
  "requestParameters": {
    "bucketName": "davide-public-test-02",
    "AccessControlPolicy": {
      "AccessControlList": {
        "Grant": [
          {
            "Grantee": {
              "xsi:type": "CanonicalUser",
              "DisplayName": "my-funny-aws-account",
              "xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
              "ID": "102myid"
            },
            "Permission": "FULL_CONTROL"
          },
          {
            "Grantee": {
              "xsi:type": "Group",
              "xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
              "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
            },
            "Permission": "READ"
          },
          {
            "Grantee": {
              "xsi:type": "CanonicalUser",
              "DisplayName": "my-funny-aws-account",
              "xmlns:xsi": "http://www.w3.org/2001/XMLSchema-instance",
              "ID": "102myid"
            },
            "Permission": "FULL_CONTROL"
          }
        ]
      },
      "xmlns": "http://s3.amazonaws.com/doc/2006-03-01/",
      "Owner": {
        "DisplayName": "my-funny-aws-account",
        "ID": "102myid"
      }
    },
    "acl": [
      ""
    ]
  },
  "responseElements": null,
  "additionalEventData": {
    "vpcEndpointId": "vpce-83a16cea"
  },
  "requestID": "1CEC6F7002EED8AA",
  "eventID": "6363c8a7-56dd-4e1f-92cd-699ba8a55613",
  "eventType": "AwsApiCall",
  "recipientAccountId": "107000000000",
  "vpcEndpointId": "vpce-83a16cea"
}

From this event data, there are a few items that call for our immediate attention:

• userIdentity

  now shows my real username, even though I’ve assumed one (

  AssumedRole

  )

• userAgent

  confirms we’re performing the operation through the web console

• requestParameters

  looks completely different

The

requestParameters

field is where all the magic happens: a new field appear as

AccessControlPolicy.AccessControlList.Grant

(reported as JSON syntax here). Grant keeps a list of all the

Grantee

, i.e. entities who have access to the bucket. There are several entities that we can find on the aforementioned links above. We will focus only on the following:

http://acs.amazonaws.com/groups/global/AllUsers

That grantee is clearly something we don’t want, regardless of the permission associated to it.

We can then add it to the new Sumo Logic query, paying attention to adding the new event type (PutBucketAcl) and the new headers we found:

(_sourceCategory=cloudtrail_aws_logs AND ("PutBucketAcl" OR "CreateBucket"))
| json "eventName", "requestParameters.bucketName", "requestParameters.x-amz-acl", "userIdentity.accountId", "userIdentity.arn", "userIdentity.sessionContext.sessionIssuer.userName", "sourceIPAddress", "requestParameters.AccessControlPolicy.AccessControlList.Grant[*].Grantee.URI", "requestParameters.AccessControlPolicy.AccessControlList.Grant[*].Permission", "eventTime" as event, bucket, acl, account, arn, user, src_ip, grant_uri, permission, datetime nodrop
| where (grant_uri matches "*AllUsers*" or grant_uri matches "*AuthenticatedUsers*") or (acl matches "*public*") 
| if (isNull(grant_uri), "null", grant_uri) as grant_uri
| if (isNull(acl), "null", acl) as acl
| count by datetime, account, account_name, bucket, acl, user, src_ip, arn, permission, grant_uri, event, grant_read, grant_write
| fields -_count
| sort by datetime

More Ways to Detect a Public Bucket Using the CLI

We have seen how to detect public S3 buckets created with a canned ACL (via the command line) and created with "Everyone" permissions via the web console.

There is another way we can make an S3 bucket public: by specifying the Grant ACP permissions via the command line.

The concept is the same as clicking on the Everyone group on the web console, something we already discussed on the previous section. Via the command line, however, AWS adds new headers that are not present in the web console activity, thus we risk to lose events if we don’t add them to our query.

They are:

requestParameters.x-amz-grant-read

,

requestParameters.x-amz-grant-read-acp

,

requestParameters.x-amz-grant-write

and

requestParameters.x-amz-grant-write-acp

.

You can test it out with:

aws s3api create-bucket --bucket db-test-bucket-public --region us-east-1 --grant-write-acp 'uri="http://acs.amazonaws.com/groups/global/AllUsers"'

Starting off our last Sumo Logic query, we can easily integrate this new information:

(_sourceCategory=cloudtrail_aws_logs AND ("PutBucketAcl" OR "CreateBucket"))
| json "eventName", "requestParameters.bucketName", "requestParameters.x-amz-acl", "requestParameters.x-amz-grant-read", "requestParameters.x-amz-grant-read-acp","requestParameters.x-amz-grant-write", "requestParameters.x-amz-grant-write-acp","userIdentity.accountId", "userIdentity.arn", "userIdentity.sessionContext.sessionIssuer.userName", "sourceIPAddress", "requestParameters.AccessControlPolicy.AccessControlList.Grant[*].Grantee.URI", "requestParameters.AccessControlPolicy.AccessControlList.Grant[*].Permission", "eventTime" as event, bucket, acl, grant_read, grant_read_acp, grant_write, grant_write_acp, account, arn, user, src_ip, grant_uri, permission, datetime nodrop
| where (grant_uri matches "*AllUsers*" or grant_uri matches "*AuthenticatedUsers*") or (acl matches "*public*") or (grant_read matches "*AllUsers*") or (grant_read matches "AuthenticatedUsers") or (grant_read_acp matches "*AllUsers*") or (grant_read_acp matches "AuthenticatedUsers") or (grant_write matches "*AllUsers*") or (grant_write matches "*AuthenticatedUsers*") or (grant_write_acp matches "*AllUsers*") or (grant_write_acp matches "*AuthenticatedUsers*")
| if (isNull(grant_uri), "null", grant_uri) as grant_uri
| if (isNull(permission), "null", permission) as permission
| if (isNull(acl), "null", acl) as acl
| if (isNull(grant_read), "null", grant_read) as grant_read
| if (isNull(grant_write), "null", grant_write) as grant_write
| count by datetime, account, account_name, bucket, acl, user, src_ip, arn, permission, grant_uri, event, grant_read, grant_write
| fields -_count
| sort by datetime

Conclusion

In this article we’ve learned how to detect public S3 buckets with AWS CloudTrail. We identified the different ways AWS uses to communicate the creation of an S3 bucket and the implications of relaxed permissions. Depending on the interface used (web console or CLI), AWS adds different headers, which makes detection a little bit tricky. I encourage you to use this information to identify your public S3 buckets and determine what truly needs to remain public and what should be immediately made private.

If you do so, I recommend that you dig into remediation via security orchestration: make private a public S3 bucket while notifying the user and your security team. Would you be interested in a follow-up blog post that teaches you how to easily perform this remediation step? Let me know in the comments below, please.