Deploying AWS Transfer Family with Lambda, EFS and DynamoDB

I previously had SFTP configured on an AWS EC2 instance.

However, due to increasing operational overhead and maintenance challenges, I decided to migrate the workload to a serverless, fully managed architecture using AWS services.

In this new setup, I’ll be using:

• AWS Transfer Family – to provide a managed SFTP server

• AWS Lambda – for implementing custom authentication logic

• Amazon DynamoDB – to store user credentials

• Amazon EFS – as the home directory for SFTP users

• AWS KMS – for encrypting credentials

Setup

I set up an AWS Transfer Family server using the SFTP protocol, deployed across two subnets for high availability. For authentication, I used a Custom Identity Provider with AWS Lambda, enabling support for both password and public key authentication.

AWS Transfer Family natively supports only public key authentication, but since my use case required password-based access, Lambda allowed me to implement that custom logic.

For storage, I configured Amazon EFS as the user home directory. This is where clients land after login and where they can upload or download files.

When a client attempts to connect, Transfer Family triggers the Lambda function. The Lambda function then retrieves the user's credentials from DynamoDB and compares them with the provided information.

If the credentials match, the function returns a success response, and the client is connected to their EFS directory. If not, it returns an empty response, and the connection is denied.

Workflow

When a client attempts to log in to AWS Transfer Family, they can choose either password-based or SSH key-based authentication.

Once the Transfer Family server receives the request, it triggers the associated Lambda function, passing in connection metadata.

For password-based authentication, the Lambda function receives details like:

{
  "username": "Client_XXX",
  "sourceIp": "1XX.2XX.2XX.2X",
  "protocol": "SFTP",
  "serverId": "s-598XXXXXXXX",
  "password": "XXXXXX@4321"
}

This data is used to validate the user against stored credentials in DynamoDB.

However, when a client uses SSH key-based authentication, the public key is not included in the Lambda payload.

Lambda Authentication Logic

Once triggered, the Lambda function retrieves the client's details from DynamoDB, where both the password and SSH public key are stored in encrypted format using AWS KMS. The Lambda code then determines the authentication method based on the login request.

• If the client provides a password:

  • Lambda decrypts the password stored in DynamoDB.

  • It compares it with the password from the login request.

  • If they match, Lambda returns a success response to Transfer Family:

      homeDirectoryDetails = [
        {
          'Entry': '/',
          'Target': '/fs-fXXXXXXXX/clients/Client_XXXX'
        }
      ]
    
      response = {
        'Role': 'arn:aws:iam::123XXXXXX:role/transfer-access-role',
        'PosixProfile': {"Gid": 65534, "Uid": 65534, "SecondaryGids": [2000]},
        'HomeDirectoryDetails': json.dumps(homeDirectoryDetails),
        'HomeDirectoryType': "LOGICAL"
      }
    

  • If credentials don’t match, Lambda returns an empty response, and the connection is denied.

• If the client uses SSH key authentication:

  • The password is not included in the request.

  • Lambda adds the public key from DynamoDB to the response:

      response['PublicKeys'] = [
        "ssh-rsa abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789"
      ]
    

  • AWS Transfer Family handles the key validation; Lambda has no control over authenticating the SSH key.

In the Lambda response, the Role field specifies the IAM role that the client will assume upon successful authentication. This role must have the necessary permissions to access Amazon EFS, enabling the client to upload and download files.

The IAM policy attached to the role should include the following permissions:

"Action": [
    "elasticfilesystem:ClientRootAccess",
    "elasticfilesystem:ClientMount",
    "elasticfilesystem:ClientWrite"
  ]

The PosixProfile and HomeDirectoryDetails fields, which define the client’s user/group IDs and their logical directory path, are also stored in DynamoDB. These are fetched by Lambda and included in the response to Transfer Family, ensuring the client lands in the correct EFS directory with appropriate file system permissions.

Limitations

• Password Exposure: Avoid logging the full Lambda event, as it may contain user passwords in plaintext.

• Concurrent Sessions: Each Transfer Family server supports up to 10,000 concurrent sessions. To avoid abuse, consider limiting sessions per user. More details.

• Client Lockout: Transfer Family doesn’t have built-in lockout policies, so I implemented a separate mechanism using AWS Lambda to track failed login attempts and temporarily block users.

• Idle Timeout: Connections are automatically closed after 30 minutes of inactivity. This timeout is not configurable.

• File Size Limit: Maximum upload size is 5 TB per file.

• Unsupported Ciphers: Some older ciphers supported in my EC2-based SFTP were not supported by Transfer Family. More details.

Thanks for reading,

-Alon