Guide to Access S3 Storage as Local Filesystem

S3 object storage offers scalable and cost-effective storage solutions but working with it directly can be challenging when your applications expect traditional filesystem access. This guide explores two powerful tools - rclone and s3fs - that bridge this gap by mounting S3 buckets as local filesystems.

Prerequisites

Before getting started, ensure you have installed the required third-party software:

• rclone: A versatile command-line tool for managing files on cloud storage

  • Installation: https://rclone.org/install/

  • Supports numerous storage providers beyond S3

• s3fs: A FUSE-based filesystem specifically designed for S3

  • Installation: https://github.com/s3fs-fuse/s3fs-fuse

  • Available in most package managers: apt install s3fs or yum install s3fs-fuse

Configuring Your S3 Mount Tools

Step 1: Setting Up Configuration Files

Each tool requires specific configuration to connect to your S3 bucket:

rclone Configuration

Create a configuration file at /etc/rclone.conf:

[s3-mount]
type = s3
provider = AWS
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = YOUR_ENDPOINT_URL
acl = private

See rclone.config.example for a complete template.

s3fs Configuration

Create a credentials file at /etc/passwd-s3fs with the following format:

ACCESS_KEY_ID:SECRET_ACCESS_KEY

Set appropriate permissions:

chmod 600 /etc/passwd-s3fs

See s3fs-passwd.example for reference.

Step 2: Creating Mount Scripts

Create shell scripts to manage the mounting process with proper parameters:

rclone Mount Script

Create /usr/local/bin/rclone-mount.sh:

#!/bin/bash

# Configuration variables
bucket="your-bucket-name"
url="https://your-endpoint.com"
mount_point="/mnt/s3-bucket"
config_file="/etc/rclone.conf"
log_file="/var/log/rclone-mount.log"
log_level="DEBUG"
provider="s3"  # Options: vstorage, s3, etc.

# Create mount point if it doesn't exist
mkdir -p "${mount_point}"

# Mount the bucket
rclone mount \
  --config "${config_file}" \
  --log-file "${log_file}" \
  --log-level "${log_level}" \
  --allow-other \
  --file-perms 0644 \
  --dir-perms 0755 \
  --vfs-cache-mode full \
  --vfs-cache-max-size 1G \
  --vfs-read-chunk-size 10M \
  --daemon \
  "${provider}:${bucket}" "${mount_point}"

exit 0

Make the script executable:

chmod +x /usr/local/bin/rclone-mount.sh

s3fs Mount Script

Create /usr/local/bin/s3fs-mount.sh:

#!/bin/bash

# Configuration variables
bucket="your-bucket-name"
url="https://your-endpoint.com"
mount_point="/mnt/s3-bucket"
passwd_file="/etc/passwd-s3fs"
log_file="/var/log/s3fs-mount.log"
log_level="debug"
region="HCM03"  # Your specific region

# Create mount point if it doesn't exist
mkdir -p "${mount_point}"

# Mount the bucket
s3fs "${bucket}" "${mount_point}" \
  -o passwd_file="${passwd_file}" \
  -o url="${url}" \
  -o use_path_request_style \
  -o allow_other \
  -o umask=0022 \
  -o dbglevel="${log_level}" \
  -o curldbg \
  -o endpoint="${region}" \
  > "${log_file}" 2>&1

exit 0

Make the script executable:

chmod +x /usr/local/bin/s3fs-mount.sh

Step 3: Creating Systemd Service Units

To ensure your S3 bucket mounts automatically at boot and is properly managed by systemd:

rclone Systemd Service

Create /lib/systemd/system/rclone-mount.service:

[Unit]
Description=Mount S3 Bucket using rclone
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
ExecStart=/usr/local/bin/rclone-mount.sh
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

s3fs Systemd Service

Create /lib/systemd/system/s3fs-mount.service:

[Unit]
Description=Mount S3 Bucket using s3fs
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/s3fs-mount.sh
RemainAfterExit=yes
ExecStop=/bin/fusermount -u /mnt/s3-bucket

[Install]
WantedBy=multi-user.target

Step 4: Enable and Start the Service

Choose which tool you prefer (rclone or s3fs) and enable its service:

# For rclone
sudo systemctl daemon-reload
sudo systemctl enable rclone-mount.service --now
sudo systemctl status rclone-mount.service

# For s3fs
sudo systemctl daemon-reload
sudo systemctl enable s3fs-mount.service --now
sudo systemctl status s3fs-mount.service

Performance Considerations

• rclone:

  • Offers better performance for large files

  • More feature-rich with built-in caching

  • Uses more memory but provides better throughput

  • Excellent for backup/sync operations

• s3fs:

  • Simpler, lighter resource footprint

  • Better for direct file access patterns

  • More POSIX-compliant but slower for metadata operations

  • Good for applications that need basic file access

Troubleshooting Common Issues

Mount Failure

If your mount fails to initialize:

1. Check credentials: Verify your access keys are correct in the configuration files

    cat /var/log/rclone-mount.log | grep "auth"
    # or
    cat /var/log/s3fs-mount.log | grep "auth"
   

2. Test connectivity: Confirm network access to your S3 endpoint

    curl -I https://your-endpoint.com
   

3. Permissions: Ensure your mount scripts are executable

    ls -la /usr/local/bin/rclone-mount.sh
    ls -la /usr/local/bin/s3fs-mount.sh
   

4. Bucket existence: Verify the bucket name is spelled correctly and exists

    # For AWS S3
    aws s3 ls s3://your-bucket-name
   
    # For other S3 providers, use their CLI tools
   

Performance Issues

If you experience slow access:

1. Increase cache size: For rclone, modify the --vfs-cache-max-size parameter

2. Adjust chunk size: Modify --vfs-read-chunk-size for your workload

3. Check network latency: High latency to your S3 endpoint will impact performance

4. Consider local caching: For frequently accessed files

References

• rclone Documentation

• rclone S3 Configuration

• s3fs Documentation

• FUSE Filesystem Overview

• AWS S3 Documentation