Amazon S3 is a good place to back up and store your files cheaply on the cloud. While some people use it to store their personal data, there are others that use it to store images and scripts for their websites, and even use it as a CDN. On your desktop, you can easily access and manage your Amazon S3 account with apps like CloudBerry, DragonDisk, etc., but what if you need to access it from the command line? This is where S3cmd comes in useful.
S3cmd is a free command line tool for uploading, retrieving and managing data in Amazon S3. Other than S3, it also supports other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects. The best use for this tool is on remote server where everything is done on the command-line. Alternatively, you can also use it to create batch scripts and automated backup to S3.
S3cmd is written in Python, so it is just about supported in all operating systems, as long as Python is installed.
Installation
S3cmd is found in most Linux repositories. In Debian/Ubuntu, or any other apt-based distro, you can install S3cmd with the command:
sudo apt-get install s3cmd
For Fedora, Centos, or any other yum-based distro:
sudo yum install s3cmd
Alternatively, you can download the source code, unzip the package and run the installer:
sudo python setup.py install
Getting Started
Before you get started, you will need to have your Amazon S3’s Access and Secret keys ready. You can locate the Access and Secret keys in the Amazon Management Console.
To get started, open the terminal and type:
s3cmd --configure
It will first prompt you to enter your Access and Secret keys.
Next, you have to enter an encryption key and the path to the GPG program.
After the encryption key is the prompt for using HTTPS protocol. The default option is “No”, but you can set to “Yes” for a more secure connection to Amazon S3. Do note that using the HTTPS protocol will slow down the transfer speed and can’t be used if you are using a proxy.
Once you have configured and tested the settings, you can proceed to use S3cmd. If you need to change the settings, you can either edit the .s3cfg file in your Home directory, or run the s3cmd --configure command again.
Usage
First, to view the buckets in your S3 account, you can use the ls command.
s3cmd ls
To create a bucket, use the mb command:
s3cmd mb s3://test-s3cmd-bucket
Note that you will need to prefix the bucket name with “s3://”
To list the content in a bucket, use the ls command together with the bucket name. For example:
s3cmd ls s3://test-s3cmd-bucket
Uploading file to bucket
The easiest way to upload a file via S3cmd is with the put command. For example:
s3cmd put text.txt s3://test-s3cmd-bucket
In addition, you can use the --encrypt parameter to encrypt the file before uploading to S3.
s3cmd put --encrypt text.txt s3://test-s3cmd-bucket
The default permission for the put command is private, which means the file can only be viewed by you. If you need the file to be publicly accessible, you can add the parameter --acl-public
s3cmd put --acl-public text.txt s3://test-s3cmd-bucket
For more granular control, use the --acl-grant=PERMISSION:EMAIL or USER_CANONICAL_ID parameter. For example:
s3cmd put --acl-grant=read:damien@gmail.com text.txt s3://test-s3cmd-bucket
The “PERMISSION” can be “read,” “write,” “read_acp,” “write_acp,” “full_control,” “all.”
Retrieving files from bucket
To retrieve a file, use the get command.
s3cmd get s3://test-s3cmd-bucket/text.txt text.txt
To download all files in the bucket, simply append the --recursive parameter.
s3cmd get --recursive s3://test-s3cmd-bucket
Deleting files from bucket
The delete command for s3cmd is simply del. For example:
s3cmd del s3://test-s3cmd-bucket/text.txt
You can also use the --recursive parameter to delete all files in the bucket.
s3cmd del --recursive s3://test-s3cmd-bucket
For more S3cmd commands, check out its usage guide here.
Advanced Usage: Synchronize a folder to S3
Let’s say you store all your important files in a folder and you want it to be synced to S3. S3cmd comes with a sync command that can synchronize the local folder to the remote destination.
s3cmd sync --delete-removed ~/SecretFolder s3://my-secret-bucket/
All you have to do is to create a cronjob to run the sync command regularly.
1. Open the crontab.
crontab -e
2. Add the following line to the end of the crontab. Save and exit the crontab.
5 * * * * s3cmd sync --delete-removed ~/SecretFolder s3://my-secret-bucket/
That’s it. Your system will now sync the secret folder to S3 every 5 minutes. You can change the value to run the sync command at your preferred interval. Every file you removed from the secret folder will be removed from S3 too.
Conclusion
For those who need to work in the command line environment, S3cmd is really a great tool for you to access and manage Amazon S3 from the command line. Not only is it easy to use, it also comes with plenty of options for advanced usage and scripting needs.
Subscribe to our newsletter!
Our latest tutorials delivered straight to your inbox
Sign up for all newsletters.
By signing up, you agree to our Privacy Policy and European users agree to the data transfer policy. We will not share your data and you can unsubscribe at any time. Subscribe
Damien Oh
Damien Oh started writing tech articles since 2007 and has over 10 years of experience in the tech industry. He is proficient in Windows, Linux, Mac, Android and iOS, and worked as a part time WordPress Developer. He is currently the owner and Editor-in-Chief of Make Tech Easier.
Comments (1)