AWS Cost Efficiency
15
mins read
How to Reduce Amazon S3 Costs by Deleting Incomplete Multipart Uploads
Best Practices for Managing Multipart Uploads and Reducing Unfinished Upload Costs
Amazon S3’s multipart upload feature is essential for handling large files, enabling improved throughput and resilience against network issues. This feature allows you to upload a single large object as a set of parts, which can be uploaded in parallel, enhancing overall upload speed and reliability. However, incomplete multipart uploads can accumulate over time, leading to unnecessary storage costs and potentially impacting your bucket's performance.
This detailed guide provides step-by-step instructions and insights to help you manage your S3 buckets more efficiently and cost-effectively.
Amazon S3 charges for all stored data, including incomplete multipart uploads. These hidden charges can add up, especially if you frequently upload large files in parts. Over time, the accumulated cost of these incomplete uploads can become a financial burden.
1. Number of failed uploads per month: 10,000 × 0.05 = 500
2. Number of incomplete parts per failed upload: 10
3. Total incomplete parts per month: 500×10 = 5000
4. Size of incomplete parts per month: 5000×100 MB=50,000 MB=500 GB
The company incurs an additional cost of $138.0 annually due to storage consumed by failed multipart uploads.
Amazon S3 Multipart Uploads allow you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. This feature is especially useful for uploading large files, as it can improve upload efficiency and reliability.
Here is a detailed explanation for Amazon S3 Multipart Upload features:
Amazon S3 Storage Lens provides a comprehensive set of metrics to help you monitor and optimize your S3 storage. The metrics are divided into Free and Advanced categories, each offering different levels of detail and insight:
For detailed information consider checking Amazon S3 Storage Lens metrics glossary
By focusing on these metrics, you can gain better insights into your S3 usage and optimize the performance and cost-effectiveness of your multipart uploads.
Monitoring for failed multipart uploads in Amazon S3 is crucial to ensure that storage costs are minimized, and storage efficiency is maintained. Here's how you can do it:
To effectively monitor failed multipart uploads, you should focus on the following metrics:
To avoid multipart uploads in Amazon S3 from becoming incomplete and incurring unnecessary costs, follow these strategies:
S3 Storage Lens helps reduce Amazon S3 costs by providing clear visibility into incomplete multipart uploads. The dashboard shows how much storage these incomplete uploads occupy and how long they've been stored. By identifying and managing these uploads either you can delete unnecessary data, free up storage space, and lower your costs.
space, and lower your costs.
Manual Steps
To create or select a Storage Lens dashboard, use a default option or click "Create dashboard" to set up a new one. Configure the scope by choosing buckets and regions, select between free or advanced metrics, and optionally set up metrics export to S3 for detailed analysis.
To configure the dashboard with desired buckets and metrics, select the relevant buckets and focus on metrics related to cost optimization. Primary metrics include total storage, object count, and average object size for an overview of storage usage. Secondary metrics offer detailed insights for cost optimization, such as noncurrent version bytes, noncurrent version object count, delete marker object count, delete marker storage bytes, % incomplete multipart upload bytes, % incomplete multipart upload object count, and incomplete multipart upload metrics older than 7 days.
The graph above illustrates the trends and distributions of the primary metric, 'total storage,' and the secondary metric, '% incomplete multipart upload bytes,' over a specified time range.
Specifically, select the metric for " % Incomplete multipart upload bytes" to view the distribution and impact of incomplete multipart uploads.
This will export the metrics to an S3 bucket, allowing you to analyze the data over extended periods.
Identify any incomplete multipart uploads that are consuming unnecessary storage space and take corrective actions, such as deleting incomplete uploads, to optimize storage usage and reduce costs.
When you use Amazon S3’s Lifecycle policies to abort incomplete multipart uploads, the process significantly impacts your storage costs. During a multipart upload, files are divided into smaller parts, which are stored temporarily until the upload is completed. If an upload is interrupted or fails, the incomplete parts can remain in your S3 bucket indefinitely, consuming storage space and potentially increasing your costs.
Manual Steps
When creating a new lifecycle rule, you can filter objects by prefix, object tags, object size, or a combination that suits your use case. Name the rule appropriately, such as "AbortIncompleteMultipartUploadRule." You can apply this rule to the entire bucket or specific prefixes, depending on your requirements.
When configuring lifecycle rule actions, choose the specific actions you want the rule to perform. Options include moving current or noncurrent versions of objects between storage classes, expiring current versions, permanently deleting noncurrent versions, and deleting expired object delete markers or incomplete multipart uploads. Note that these actions are not supported when filtering by object tags or object size. Per-request fees apply; learn more or see Amazon S3 pricing for details.
Implementing multipart upload notifications and management in Amazon S3 helps reduce costs by quickly identifying and handling incomplete uploads. By setting up S3 event notifications to trigger alerts and using AWS Lambda to automatically abort these uploads, you prevent unused storage from accumulating. This ensures that only completed data is stored, thus avoiding unnecessary storage charges and keeping your S3 costs under control.
Manual Steps
To access Amazon SQS, first log in to the AWS Management Console, then select "Services" and choose "Simple Queue Service (SQS)" under the Application Integration category. To create a new queue, click on "Create queue," select "Standard" for the queue type, enter a name for your queue, and configure any additional settings as needed. Finally, click "Create queue" to complete the setup.
To configure S3 event notifications, first access Amazon S3 by selecting "Services" from the AWS Management Console and then choosing "S3" under the Storage category. Next, select the bucket where you want to monitor multipart uploads. Click on the "Properties" tab, scroll down to the "Event notifications" section, and click "Create event notification." Enter a name for the event notification, then select "All object create events" and "All object remove events" in the "Event types" section. In the "Destination" section, choose "SQS queue" and select the SQS queue you previously created. Finally, click "Save changes" to complete the configuration.
To add an SQS trigger to your Lambda function, first navigate to the Lambda function console and click on the "Add trigger" button. Select "SQS" as the trigger type and choose the SQS queue you created earlier. Click "Add" to configure the trigger. Next, ensure that your Lambda function has the necessary permissions to read messages from the SQS queue and perform operations on S3. Update the Lambda function's execution role to include permissions for both SQS and S3 to enable proper functioning.
Optimizing the size of the parts during a multipart upload can significantly enhance upload efficiency and reduce the likelihood of incomplete uploads. By selecting the appropriate part size based on file size and network conditions, you can achieve a more reliable and cost-effective upload process.
Incomplete multipart uploads in Amazon S3 can significantly inflate storage costs, accounting for up to 20% of your expenses. To manage and reduce these costs, enable S3 Storage Lens to monitor usage, implement lifecycle policies to automatically delete incomplete uploads after a set period, and set up notifications with AWS Lambda to abort unfinished uploads promptly. These strategies help ensure that only fully uploaded data consumes storage, optimizing your S3 costs efficiently.
Looking to streamline this process with automation and reduce efforts?
Don't miss Part 2 of this guide where we dive into automated solutions for reducing Amazon S3 Costs by Deleting Incomplete Multipart Uploads.
Part 2 - https://www.astuto.ai/blogs/automating-deletion-of-incomplete-multipart-uploads
.svg)
.svg)
Strategical use of SCPs saves more cloud cost than one can imagine. Astuto does that for you!
.png)
.svg)
