Thursday, April 12, 2012

Delete Multiple Objects from s3

Deleting s3 objects used to be slow if you have lots of files in a s3 buckets: you have to issue a delete request for each file. On Dec 7, 2011 , Amazon announced Multi-Object Delete, a new API that allows user to delete multiple objects with a single web request.

As this api is relatively new, those popular ruby s3 libraries including rightaws has not supported it. So I wrote a ruby script to do it, the source code is here:

And it can be used as a command line tool:

$ruby s3_multi_object_delete.rb bucket s3id s3key key1 key2

Or you want to use it in your Rails project, just drop it into 'config/initializers' folder.

It is based on happening, although the code can be ported to rightaws easily. I picked happening as it may archive better performance due to higher concurrency.



Anonymous Ronak said...

Amazon S3 and its services are most demanding.Multi-object-delete will delete in the batch of 1000 objects in single request.
You can easily use Bucket Explorer my very own tool to manage content at S3..

Bulk delete: How to delete large number of files from an Amazon S3 Bucket?

Download Bucket Explorer

11:13 PM  

Post a Comment

<< Home