The S3 Balancing Act: Security vs Simplicity

On April 27, 2023, AWS rolled out a long-anticipated change to Amazon S3. This change blocked public access and custom ACL rules, by default, for all newly created buckets on S3.

Bad S3 bucket permissions are among the most widely exploited "security incidents" in the cloud. This occurs when someone creates a bucket for private data (eg. personal customer information) but accidentally makes that bucket publicly available. By blocking public access, AWS makes this sort of misconfiguration impossible (without explicitly disabling the block policy).

But no good deed goes unpunished - for every step forward AWS makes security-wise, it feels that they go two steps back on the developer experience (DX). While public buckets can no longer be created by accident, they are also now much more difficult to create on purpose.

This is a problem because creating public buckets is a common use case for hosting static sites and assets - this workflow is so common that it is one of the featured workflows for S3.

Let's illustrate what the change looks like by trying to create a public bucket. If you're using the AWS console, you now must explicitly disable the "Block all public access" toggle as well as check a consent dialogue to create a bucket.

Note that this is for ensuring that a bucket can be public - you still need to apply the proper object or bucket-level policies to make objects public.

If you're a DevOps wizard that never touches the console, you might create a new bucket using Terraform or the CDK.

In this case, you might write the following code:

const publicBucket = new s3.Bucket(this, 'MyPublicBucket', {
  publicReadAccess: true,
});

When you run a diff, things look normal

$ cdk diff
Stack AutoCdkStack
IAM Statement Changes
┌───┬─────────────────────────┬────────┬──────────────┬───────────┬───────────┐
│   │ Resource                │ Effect │ Action       │ Principal │ Condition │
├───┼─────────────────────────┼────────┼──────────────┼───────────┼───────────┤
│ + │ ${MyPublicBucket.Arn}/* │ Allow  │ s3:GetObject │ AWS:*     │           │
└───┴─────────────────────────┴────────┴──────────────┴───────────┴───────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Resources
[+] AWS::S3::Bucket MyPublicBucket MyPublicBucketBCB7A648 
[+] AWS::S3::BucketPolicy MyPublicBucket/Policy MyPublicBucketPolicyDE00CFED 


✨  Done in 3.94s.

Try deploying it, however, and you run into an unhelpful "Access Denied" error with no further context

$ cdk deploy

✨  Synthesis time: 2.88s

AutoCdkStack:  start: Building 46900dc420169a4d5dc632927ff6ca3ba8c1ca93cdd60512a3c0950e2d75
...
AutoCdkStack: deploying... [1/1]
AutoCdkStack: creating CloudFormation changeset...
7:37:38 AM | CREATE_FAILED        | AWS::S3::BucketPolicy | MyPublicBucketPolicyDE00CFED
API: s3:PutBucketPolicy Access Denied

 ❌  AutoCdkStack failed: Error: The stack named AutoCdkStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: API: s3:PutBucketPolicy Access Denied
...
The stack named AutoCdkStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: API: s3:PutBucketPolicy Access Denied

The error comes from the change in default bucket policy - when you create a new bucket and want to make it public, you also must disable the "Block Public Access" settings on the bucket.

In our cdk example, this would look like the following

    new s3.Bucket(this, 'MyPublicBucket', {
      publicReadAccess: true,
      blockPublicAccess: { 
	      blockPublicAcls: true, blockPublicPolicy: false, 
	      ignorePublicAcls: true, restrictPublicBuckets: false},
    });

Unfortunately, this isn't well documented in the CDK docs. You will also run into similar problems when trying to provision a public bucket using Terraform. As of 2023-06-10, Both Terraform and CDK have S3 bucket creation tickets as the number one pinned issues in their respective repositories, here and here.

AWS CDK

Terraform

While the issue with public S3 Buckets seems minor, it is just the tip of the iceberg in terms of complexity. Creating a public/private bucket is a trivial example. Things become more complicated once you involve encrypted objects, other AWS services, and cross-account access. I wrote an article many moons ago about debugging eight different permission layers to diagnose an S3 bucket permission issue. Since that article was written, S3 has added even more complexity features like object lambdas and access points that add more layers to an already complicated access control model.

Zooming out - S3 is a single service. AWS has over 200 services. Each service is complicated by itself but that complexity compounds when they need to talk to one another. It then further compounds once you involve multiple accounts.

So what do we do about this?

Some solutions address the complexity by building platforms that abstract away the cloud provider. Services like Heroku and Vercel do a great job to improve DX outside of the cloud. They provide this DX at the cost of control and extensibility (plus significant markups in pricing).

Other solutions address the complexity by building better tooling inside the cloud. Infrastructure-as-Code (IaC) primitives like Terraform, CDK, and Pulumi make it easier to programmatically create infrastructure and replicable environments. Unfortunately, they still require users to deal with all the messy implementation details of each service. The latest addition in the IaC space is Winglang, a new project made by the original creator of the AWS CDK. It aims to be a higher-level IaC primitive that further abstracts many of the messy details of cloud services and is to terraform what Python is to C.

The change in S3 permission defaults highlights the challenges of building a cloud that is simple, secure, and extensible. This can feel like a CAP theorem-like problem but there are no laws of physics that prevent us from having all three. I think there's a lot more that can be done in this space, particularly the approach of improving the DX inside the cloud vs offering a limited experience outside of it. This is a space I have a few ideas on which I aim to share in the coming weeks. Meanwhile, may your clouds be unfettered and your buckets secure (even if they are not simple).