Secure file exchange via Amazon S3

The software located in AWS cloud often needs to exchange files with other applications to export some files making them available for download or import & process files that are uploaded by other applications.

Secure File Exchange

At first glance it seems that it is much simpler to implement some API endpoint both at your software side and on the other side. Enabling some standard file transfer protocols like SFTP might also seem like a good option.

However, in terms of security, SFTP is not an effective solution. Implementing endpoints might often be an issue due to corporate security restrictions. This approach can make it impossible to generate files for exchange in real time. This happens, for example, when you want to prepare data for processing by machine learning algorithms - in this case the files are usually large and processing them in real time consumes lots of resources. You need a highly secure storage solution as the information inside these files is sensitive in its nature.

One of the ways to avoid keeping parts of your architecture outside the cloud is to secure files exchange using amazon S3 service. A few tweaks will allow you to make storage and exchange files in S3 way more secure than usual and dramatically reduce the amount of possible threats. Follow these steps to set it up:

1. Create users, keys and buckets

First step is to create 2 AWS users via amazon IAM.

Set the usernames. Let’s call them:

• you
• client

Create two s3 buckets:

• to-client - this bucket will be used to send files to the client
• from-client - this bucket will be used to receive files from the client

Use an AWS service called KMS (Key Management Service). Go to KMS → Customer managed keys and create 2 symmetric keys:

• to-client-key - the key to be used to encrypt / decrypt files in to-client bucket
• from-client-key - the key to be used to encrypt / decrypt files in from-client bucket

The keys must be symmetric to be available for use together with s3

2. After users, keys and buckets have been created, security and permissions should be set up.

Buckets

Files put into the buckets must be encrypted automatically using appropriate keys.

In order to set it up, go to the “properties” tab of the bucket, select “Default encryption”, click “AWS-KMS” radio button and select appropriate key there.

So to-client-key should be selected for to-client bucket and from-client-key should be selected for from-client bucket.

After buckets are set up this way - the files will be encrypted / decrypted automatically if the user who uploads / downloads the file has permission to use the key, so the next step is to set up permissions for the keys.

Keys

User “you” needs to be able to encrypt using “to-client-key” and decrypt using “from-client-key”. The opposite works for user named “client” - this user needs to be able to decrypt using “to-client-key” and encrypt using “from-client-key”.

So we need to edit from-client-key key policy:

{
  "Version": "2012-10-17",
  "Id": "key-consolepolicy-3",
  "Statement": [
     {
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
           "AWS": "arn:aws:iam::<YOUR_AWS_ID>:user/you"
        },
        "Action": [
           "kms:Encrypt",
           "kms:GenerateDataKey*"
        ],
        "Resource": "*"
     },
     {
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
           "AWS": "arn:aws:iam::<YOUR_AWS_ID>:user/client"
        },
        "Action": "kms:Decrypt",
        "Resource": "*"
     }
  ]
}
                

The opposite must be done for “from-client-key” key policy

{
  "Version": "2012-10-17",
  "Id": "key-consolepolicy-3",
  "Statement": [
     {
       "Sid": "Allow use of the key",
       "Effect": "Allow",
       "Principal": {
           "AWS": "arn:aws:iam::<YOUR_AWS_ID>:user/client"
       },
       "Action": [
           "kms:Encrypt",
           "kms:GenerateDataKey*"
       ],
       "Resource": "*"
     },
     {
       "Sid": "Allow use of the key",
       "Effect": "Allow",
       "Principal": {
           "AWS": "arn:aws:iam::<YOUR_AWS_ID>:user/you"
       },
       "Action": "kms:Decrypt",
       "Resource": "*"
     }
  ]
}

Users

The users need to be able to encrypt / decrypt using appropriate keys as well as write / read to appropriate buckets.

You can achieve this using inline policies. For “client” user the policy will look like this:

{
  "Version": "2012-10-17",
  "Statement": [
      {
        "Sid": "MySid1",
        "Effect": "Allow",
        "Action": [
            "kms:Encrypt"
        ],
        "Resource": [                
            "arn:aws:kms:<REGION>:<YOUR_AWS_ID>:key/<ID_OF_TO-CLIENT_KEY>"
        ]
      },
      {
        "Sid": "MySid2",
        "Effect": "Allow",
        "Action": [
            "kms:Decrypt"
        ],
        "Resource": [                
            "arn:aws:kms:<REGION>:<YOUR_AWS_ID>:key/<ID_OF_FROM-CLIENT_KEY>"
        ]
      },
      {
        "Sid": "MySid3",
        "Effect": "Allow",
        "Action": [
            "s3:PutObject",
            "s3:PutObjectAcl"
        ],
        "Resource": [
            "arn:aws:s3:::<TO-CLIENT_BUCKET_NAME>",
            "arn:aws:s3:::<TO-CLIENT_BUCKET_NAME>/*"
        ]
      },
      {
        "Sid": "MySid4",
        "Effect": "Allow",
        "Action": [
           "s3:ListBucket",
           "s3:GetObject"
        ],
        "Resource": [
           "arn:aws:s3:::<FROM-CLIENT_BUCKET_NAME>",
           "arn:aws:s3:::<FROM-CLIENT_BUCKET_NAME>/*"
        ]
      }
  ]
}

Add a similar policy for “you” user.

After the setup is done, the files can be encrypted and decrypted automatically without any additional efforts from your side. For example, if you put a file using AWS CLI using this command (using “you” user credentials)

aws s3 cp my_file.txt s3://<TO-CLIENT_BUCKET_NAME>/my_file.txt
            

The file my_file.txt in the bucket will be encrypted.

This approach might look a little bit heavy-weight but it adds a new layer of security protecting your corporate data and makes your enterprise less vulnerable.