Streamlining Data Flow: Integrating SFTP User Access with S3, SQS, and

Introduction :

In the blog post, we can explore how to set up a seamless data flow by integrating various components such as SFTP user access, S3 storage, SQS queues, and VM directory mounting. We'll discuss the steps involved in configuring SFTP users with limited access to specific directories, mounting S3 buckets to a virtual machine (VM) directory, and utilizing SQS for event notification.

Pre-requisite

Create s3 bucket.
Create sqs queue.
create ec2 instance (ubuntu)
create IAM User for accessing s3 bucket and add s3 permission to it.

Creating user for sftp

Install s3fs

sudo apt update
sudo apt install s3fs

Setup creds for s3fs

Create file

sudo vim /etc/passwd-s3fs

# write below content with its actual value for iam user created
ACCESS_KEY_ID:SECRET_KEY_ID

create user for sftp

#!/bin/bash

# Usage: sudo ./create-sftp-user.sh username

USERNAME=$1
S3_BUCKET_NAME="s3-bucket-name"
S3_BUCKET_PATH="$S3_BUCKET_NAME:/incoming/$USERNAME"
MOUNT_DIR="/mnt/sftp-s3/$USERNAME"

sudo useradd -m -d "$MOUNT_DIR" -s /bin/bash -G sftponly "$USERNAME"
# Set password for the user
sudo passwd "$USERNAME"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
sudo mkdir -p "$MOUNT_DIR/data"
sudo s3fs "$S3_BUCKET_PATH" "$MOUNT_DIR/data" -o passwd_file=/etc/passwd-s3fs -o allow_other -o nonempty
echo "User $USERNAME added and configured."

sudo vim /etc/ssh/sshd_config

at last write

Match User userName
    ChrootDirectory /mnt/sftp-s3/$USERNAME
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
    PasswordAuthentication yes

now try to access over sftp .

Update sqs queue access policy :

{
  "Version": "2012-10-17",
  "Id": "Policy1679925546977",
  "Statement": [
    {
      "Sid": "Stmt1679925532180",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:*",
      "Resource": "arn:aws:sqs:region:1111111111:sqs_name",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::s3-buket-name"
        }
      }
    }
  ]
}

Create s3 event notification.

Thanks

For more such content follow me on :

Twitter : https://x.com/akash202k_

Streamlining Data Flow: Integrating SFTP User Access with S3, SQS, and VM Directory Mounting

Table of contents

Pre-requisite

Creating user for sftp

Update sqs queue access policy :

Subscribe to my newsletter

Akash Pawar

Akash Pawar