AWS S3 is a main Object and File storage service and can be integrated with many other AWS Services. Also, S3 has many features you need to be familiar with in order to utilize the full functionality of S3.
Optimizing S3 Performance
- Amazon S3 automatically scales to high request rates.
- An application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket.
- Previously, the randomizing prefix naming with hashed characters is recommended for performance. But you do not need to randomize the prefix names anymore. You can use sequential date-based naming for your prefixes.
S3 Object Lock
- S3 Object lock enables you to store objects using a Write-once-Read-many (WORM) model.
- Object locks can be applied on individual objects or across the bucket as a whole.
- Once an object is locked, the object cannot be modified or deleted for a fixed amount of time or indefinitely.
- It is used for regulatory requirements (WORM storage) or the layer of protection.
- Retention Period: S3 Object lock protects an object for a specified amount of time.
- S3 stores a timestamp in the object’s metadata. After the retention period expires, an object can be overwritten or deleted.
- Governance Mode: A user needs special permission to change the retention period or delete objects.
- Compliance Mode: An object cannot be overwritten or deleted by any user (even a root user). In this mode, the retention period cannot be modified.
Glacier Vault Lock
- S3 Glacier Value lock is used to enforce compliance controls to S3 Glacier vaults with a vault lock policy. It provides the locking mechanism in the Glacier.
- Once locked, the policy can no longer be changed.
S3 Select and Glacier Select
- S3 Select is used to retrieve a subset of data from an object using simple SQL expressions by rows and columns.
- It can boost performance increases of your underlying applications (up to 400%).
- You can save money on data transfer.
- Similarly, Glacier Select allows to run simple SQL expressions to Glacier storage.
S3 Presigned URLs
A presigned URL provides access to an object using the creator’s access permissions and is used to download/upload objects (GET and PUT operations).
- It can be used for anonymous users to access an object.
- URL is temporary and will be expired (7-day max).
- You may get an error when you use presigned URLS when:
- expired (7-day max)
- the permission of a creator has been changed
- the temporary credential of a role is expired (if the URL is created using a role. – 36 hours)
S3 Cross-region Replication (S3 CRR)
S3 CRR allows one-way replication to another region.
- Replicate objects keep their storage class, object name (key), owner, object permissions by default. But it is possible to override the storage class, owner, and permissions.
- The replication only happens to the object that is added after the configuration. (Not retroactive)
- All subsequently updated files will be replicated automatically.
- The replication is one-way only.
- The versioning needs to be enabled both in a source bucket and a destination bucket.
- An IAM role is required to replicate objects.
- SSE-S3 and SSE-KMS encrypted objects or (None) can be replicated. SSE-C objects are excluded.
- Delete markers are not replicated.
- The following will be excluded from replication: System actions (Lifecycle events), any objects before replication is enabled, SSE-C encrypted objects.
S3 Transfer Acceleration (S3TA)
S3 Transfer Acceleration provides fast and secure data transfers to the S3 buckets.
- Transfer data faster over long distance
- It takes advantage of CloudFront’s globally distributed edge locations and AWS backbone networks for faster transfers.
- Rather than uploading files directly to S3 buckets, you are uploading files to the closest edge location and then transfer the files to S3 using AWS networks.
- The feature is enabled per bucket.
- A distinct URL is used to upload files.
- ex) <bucketname>.s3-accelerate.amazonaws.com
- Additional cost per usage (GB)
- S3TA vs. CloudFront PUT/POST
- Use S3TA when a higher throughput is required and you want to use all bucket-level features such as multipart uploads
- Consider CloudFront PUT/POST of the data set is less than 1GB in size.
S3 Event Notification
- S3 notification feature enables you to receive notifications when certain events (Create, Delete, Restore) happen in your bucket.
- S3 Supports the following destinations: SNS Topic, SQS Queue, Lambda Function