Today I needed to migrate a bunch of files from one S3 account (charged to me) or another (paid for by company). There would be a few changes to the paths but on the face of it this is a simple task. The tricky part was I wanted to copy the files on S3 not have to download and re-upload them all again as that would take way too long and cost more money.
Ruby Script (using Right AWS)
http://rightaws.rubyforge.org/
As per usual I started with a quick ruby script. I have learned that right_aws is better than the standard s3 ruby gem, so I used this. It supports copy and move so I figured I would be all would be OK. Not so.
bucket_old = s3_old.bucket('old_bucket')
bucket_new = s3_new.bucket('new_bucket')
keys_old.each do |key_old|
name_new = key_old.name.gsub(/^\/media\/([^\/]+)\/(.*)/) { "#{$1}/archive/#{$2}"}
key_new = RightAws::S3::Key.create(bucket_new, name_new)
key_old.copy(key_new) # Big Fail
end
The above code throws a big fat error because its trying to use the old credentials for the new key (I think). FAIL. I took a look around the source code, could not see an easy fix so moved on.
CloudBerry S3
http://www.cloudberrylab.com/default.aspx?id=1
Although windows only, this software looked like it might do the trick. It certainly said it would support copies between S3 accounts. For all I know it probably does, but it didn’t work for me. Why? Because of a leading / in my key paths. This was left over from S3Sync and I have since learned that it seems to upset quite a few programs. If only they would let me enter the path I needed manually I could get past the problem, but alas I was stuck with a non working GUI. SEMI FAIL.
Bucket Explorer
http://www.bucketexplorer.com/
This software is available for Mac, Linux, and Windows.. which is good. It does however have an ugly looking UI. Oh well its a tool not a beauty contest so I gave it a go. Connecting to both accounts was easy enough, just open two windows. Copying files is a case of browsing to the files you want, selecting them, then choosing copy from the right click menu. Then go over to the target S3 account and right click paste in the desired folder. This opens up a queue processing dialog which chugs through. So far it seems to be doing the trick and I’m a happy bunny. SUCCESS
If bucket explorer had failed I would have probably tried boto or resorted to downloading the re-uploading using s3 sync.
