This article is part of a series about using AWS Simple Storage Services (S3) in OutSystems Developer Cloud applications. In this first part, we will discuss the challenges you might encounter when using the default method of OutSystems Developer Cloud to store and retrieve files. We will also introduce AWS S3 and its potential benefits as an alternative.

Default File Handling in ODC

The built-in way to store and retrieve files in OutSystems applications is by using the Binary Data Type in the data model. Files are stored as BLOBs (Binary Large Objects) in the database and can be queried using aggregates.

The typical file storage process involves having a screen with a File Upload widget. When you submit the form, you call a server or service action that takes the filename and binary data as input. It then executes an entity create action to store a record that includes the binary data.

This method works well for smaller files and small quantities. However, as the volume or size of files increases, you may face several challenges.

Challenges

The main drawbacks of using Binary Data Attributes for files are:

• When you upload a file from your frontend to the backend using a server or service action, the Server Request Timeout setting of your application is applied. In OutSystems Developer Cloud, the maximum request timeout is 60 seconds, with a default of 10 seconds. If your upload takes longer than this, you will face a timeout error. The same rule applies when retrieving a file through a backend server or service action.

• Request restrictions: When uploading files from your frontend to the backend, there is a maximum request payload limit of 28MB.

• No support for File Streams and Byte Range Requests — When querying with an aggregate, the entire content of the Binary Data Attribute must be read from the database and processed at the application container. Then, the transfer to the client application is a full download again. This is not suitable for streaming media because, for example, you can't jump directly to a specific point in a video. Additionally, this causes latency, negatively affecting the user experience.

Byte Range support is important not only for streaming but also for allowing multiple connections through download managers (multipart download) or resuming interrupted downloads.

Streaming media isn't just about audio and video; it also includes web-optimized PDF documents, known as linearized PDFs. Linearized PDFs greatly reduce the time it takes to display the first page by loading additional pages on demand as you scroll or jump through the document.

• Memory Usage — Data from the database is retrieved by the application container and stored in memory. When many large files are fetched at the same time, this can quickly cause memory swaps. This drastically reduces performance and can even cause timeouts.

• In OutSystems Developer Cloud, an entity can have only one attribute of the Binary Data type. This limitation might lead to increased Application Object Count usage, depending on your implementation.

Overall, providing files through the OutSystems database has many drawbacks. For high-volume file delivery or if streaming content is needed, it is recommended to switch to a third-party service like AWS Simple Storage Service (S3).

AWS Simple Storage Service (S3)

AWS Simple Storage Service, or S3, is a type of object storage where you can store any data, like documents, videos, images, and even entire websites. Instead of storing data as files in a file system, S3 turns them into digital objects. These objects can have extra information (metadata), which is useful for automated processing later on.

S3 organizes these objects into buckets, and a single AWS account can have up to 100 buckets. If needed, you can request an increase to 1,000 buckets from AWS. While you can create folders in the AWS S3 console, S3 doesn't actually use hierarchical storage. Instead, it uses prefixes in object keys—strings separated by a forward slash—to create a virtual hierarchy.

S3, or the S3 API, is widely recognized as a standard in the market. Besides AWS S3, there are various free and commercial products that fully or partially implement this standard, including storage solutions from several vendors.

Choosing S3 to store and retrieve objects or files for your ODC applications offers several potential benefits, including:

• Forge Components – Existing Forge Components allow for quick integration.

• Client-Side Upload – Allows files to be uploaded directly from the frontend.

• Multipart Upload – Enables uploading very large files in parallel chunks.

• Streaming – Supports streaming and byte range requests for media and document streaming.

• Data Lifecycle Management – S3 Lifecycle Rules enable automatic archiving or deletion of objects, including moving data to cheaper storage options.

• Versioning – Store and retrieve unlimited versions of the same object.

• Object Events – Keep OutSystems entity records and stored objects in sync using S3 events, or use events for additional processing with Amazon Web Services.

Regarding Forge components, I want to highlight two existing S3 integrations:

• AWS_S3_Connector - This is the officially supported S3 connector maintained by OutSystems.

• AWSSimpleStorage - This is my own S3 connector library, which includes extra features like the ability to connect with third-party S3-compatible stores.

Both the official S3 connector and my version are built on the official AWS SDK for .NET and are implemented as ODC external logic functions.

External Logic functions in ODC are based on AWS Lambda. Lambda functions have a maximum request and response limit of 6MB, and the same limit applies to ODC External Logic functions, with a bit of overhead. For the external logic connectors for S3, this means you cannot handle files larger than 5.5MB.

You might wonder why not directly use REST to access the S3 API, where this restriction doesn't apply. The short answer, at least for my implementation, is that manually creating AWS Signature V4 requests for different S3 operations is quite complex, and the SDK simplifies this process (there is a AWS Signature V4 component on Forge, but it is not mature enough for some request types).

In the end, it doesn't matter because there are a few patterns you can use to overcome the 5.5MB limit of external logic functions. Plus, you'll gain some extra benefits that address the challenges mentioned above.

Platform Limits

When working with files, it's important to understand the overall platform limits of OutSystems Developer Cloud. This is especially crucial when dealing with larger files, as you might hit a limit during processing. The image below highlights the key limits to keep in mind when handling larger files.

Summary

In this introduction, I aimed to provide an overview of the challenges you might encounter when storing and retrieving files in ODC applications. These challenges include:

• Application timeout settings that can cause uploads and downloads to be interrupted.

• Request limits from the browser to the application backend might prevent the uploading of large files to the backend.

• Lack of streaming support, requiring files to be fully transferred between the client application and the backend. Partial transfers, like byte range requests, are not supported by OutSystems Developer Cloud.

• Querying a set of files from the database can lead to high memory usage in the application container, which may result in performance degradation.

• An entity in OutSystems Developer Cloud supports only one attribute of type Binary Data, which can increase the Application Object Count.

Adding Amazon Simple Storage Service (S3) for file storage and retrieval can be a valuable addition to your ODC environment to address these challenges, if they apply to you. OutSystems provides a connector library for S3 based on the official AWS SDK for .NET. However, being an external logic library, the connector cannot be used to upload or download files larger than 5.5MB.

In the following articles, we describe specific implementation patterns to integrate with S3 and overcome the 5.5MB limit of external logic connectors.