Seamless File Transfer from S3 to External SFTP endpoint using Go

As cloud computing becomes increasingly central to modern infrastructure, many applications require efficient ways to move files between cloud storage and external servers. In this post, I'll show you how to automate file transfers from AWS S3 to an external SFTP server using Go and AWS Lambda. We'll break down each step, leveraging AWS SDKs, AWS Secrets Manager, and SFTP libraries to create a seamless transfer pipeline.

Prerequisites

To follow along, you’ll need:

• An AWS account with access to S3, Lambda, and Secrets Manager.

• A configured SFTP server where files will be sent.

• Basic knowledge of Go, AWS SDK, and Lambda functions.

Solution Architecture

The solution relies on AWS Lambda to handle the file transfer. Each time the Lambda function is triggered, it:

1. Connects to AWS S3 and lists the files under a specific folder.

2. Retrieves SFTP credentials from AWS Secrets Manager.

3. Connects to the external SFTP server and uploads each file.

Setting Up AWS Secrets Manager

For security, store your SFTP credentials (host, port, username, password) in AWS Secrets Manager. Here’s a sample JSON format:

{
  "sftpHost": "your-sftp-host",
  "sftpPort": "22",
  "sftpUsername": "your-username",
  "sftpPassword": "your-password"
}

Code Breakdown

• The code uses the AWS SDK, AWS Lambda Go SDK, SFTP library, and SSH libraries:

import (
 "context"
 "encoding/json"
 "fmt"
 "io"
 "log"
 "path/filepath"

 "github.com/aws/aws-lambda-go/lambda"
 "github.com/aws/aws-sdk-go/aws"
 "github.com/aws/aws-sdk-go/aws/session"
 "github.com/aws/aws-sdk-go/service/secretsmanager"
 "github.com/aws/aws-sdk-go/service/s3"
 "github.com/pkg/sftp"
 "golang.org/x/crypto/ssh"
)

const (
 s3Bucket       = "your-s3-bucket"
 s3FolderPrefix = "your-folder-prefix"
 region         = "your-region"
 secretName     = "your-secret-name"
)

• The lambdaHandler function is the main entry point. We start by creating an AWS session and fetching SFTP credentials from Secrets Manager.

func lambdaHandler(ctx context.Context) error {
 log.Println("Lambda handler started")

 sess, err := session.NewSession(&aws.Config{Region: aws.String(region)})
 if err != nil {
  return fmt.Errorf("failed to create AWS session: %w", err)
 }

 sftpConfig, err := getSFTPConfig(sess)
 if err != nil {
  return fmt.Errorf("failed to get SFTP config: %w", err)
 }

• Using the S3 client, we retrieve the list of objects in the specified S3 folder.

svc := s3.New(sess)
resp, err := svc.ListObjectsV2(&s3.ListObjectsV2Input{
 Bucket: aws.String(s3Bucket),
 Prefix: aws.String(s3FolderPrefix),
})
if err != nil {
 return fmt.Errorf("failed to list objects: %w", err)
}

• Next, we connect to the SFTP server using SSH:

sshConfig := &ssh.ClientConfig{
 User: sftpConfig.SFTPUsername,
 Auth: []ssh.AuthMethod{ssh.Password(sftpConfig.SFTPPassword)},
 HostKeyCallback: ssh.InsecureIgnoreHostKey(),
}

address := fmt.Sprintf("%s:%s", sftpConfig.SFTPHost, sftpConfig.SFTPPort)
conn, err := ssh.Dial("tcp", address, sshConfig)
if err != nil {
 return fmt.Errorf("failed to dial SFTP server: %w", err)
}
defer conn.Close()

• The actual file transfer occurs by first downloading the object from S3 and then copying it to the SFTP server.

getObjectOutput, err := svc.GetObject(&s3.GetObjectInput{
 Bucket: aws.String(s3Bucket),
 Key:    aws.String(key),
})
defer getObjectOutput.Body.Close()

remoteFilePath := fmt.Sprintf("/uploads/%s", filepath.Base(key))
dstFile, err := sftpClient.Create(remoteFilePath)
if err != nil {
 return fmt.Errorf("failed to create remote file: %w", err)
}
defer dstFile.Close()

_, err = io.Copy(dstFile, getObjectOutput.Body)
if err != nil {
 return fmt.Errorf("failed to copy file to remote: %w", err)
}

Error Handling and Troubleshooting

Here are some troubleshooting tips for common errors:

• AWS Permissions: Ensure that the Lambda function has the necessary IAM permissions for S3 and Secrets Manager.

• Network Issues: If the SFTP connection fails, verify that your Lambda has access to the SFTP server’s network.

• Logging: Enable logging to help debug issues by inspecting the Lambda execution logs in CloudWatch.