aws - simple access to Amazon EC2 and S3 and SQS and SDB and ELB

  timkay.com » tools » aws

aws is a command-line tool that gives you easy access to Amazon EC2 and Amazon S3. aws is designed to be simple to install and simple to use.

Thanks to your feedback ([email protected]), aws is the top-rated "community code" for all of Amazon EC2 and S3! See the ratings and reviews at EC2 and S3. They make me blush! Thank you!

This document describes the basic features. For additional examples, see the HowTo document.

SDB support - zikes! it was easy to add "aws" now supports Amazon SDB. See the documentation. ELB support "aws" now supports Amazon ELB. See the documentation. SQS support "aws" now supports Amazon SQS. See the documentation. Some new options (as of v1.29): --simple simplified output for some commands --wait=SEC wait for EC2 instances to start --cut truncate output to screen width --md5 verify MD5 checksums --max-time=N time out after N seconds (will retry up to 3 times) --fail suppress error message display (but sets $?) --curl-options="blah"   specify additional curl options --set-acl   --public     --private set or change ACL Sets $? correctly in more cases. Retries all requests up to 3 times. s3put will read from stdin (creating a temp file, which is then uploaded). Regions support (as of v1.23), as well as support for Signing Version 2, --progress indicator, better error messages, better formatting, and some small bug fixes. See the change log. Many, many changes (as of v1.21). ~/.awsrc file support, --exec option to iterate over results, --secrets-file, --expire-time, --limit-rate, Cache-Control:, Range: and If-*: support. And a few bug fixes/tweaks.

See the change log.

Download

To use aws, follow these steps:

1. Install curl if necessary (either apt-get install curl or yum install curl)
2. Download aws to your computer

   curl https://raw.githubusercontent.com/timkay/aws/master/aws -o aws

3. Put your AWS credentials in ~/.awssecret: the Access Key ID on the first line and the Secret Access Key on the second line. Example:

   1B5JYHPQCXW13GWKHAG2
   2GAHKWG3+1wxcqyhpj5b1Ggqc0TIxj21DKkidjfz

4. Alternately, set the EC2_ACCESS_KEY and EC2_SECRET_KEY environment variables.
5. OPTIONAL. Perform this step if you want to use commands like s3ls and ec2run. If you prefer commands like aws ls or aws run, then you may skip this step. Perform the optional install with:

   perl aws --install

   (This step sets aws to be executable, copies it to /usr/bin if you are root, and then symlinks the aliases, the same as aws --link does.)
6. Alternately, set "aws" to be executable:

   chmod +x aws

If you are logged in as root, aws will be installed in /usr/bin. Otherwise, it will be installed in the current directory.

You are ready to use Amazon EC2 and Amazon S3. Try the following example. You'll need to provide your own BUCKET_NAME, unique from all other bucket names.

s3mkdir BUCKET_NAME
s3put BUCKET_NAME/aws-backup /usr/bin/aws
s3ls BUCKET_NAME
s3get BUCKET_NAME/aws-backup
s3delete BUCKET_NAME/aws-backup
s3delete BUCKET_NAME

ec2run
ec2din
ec2tin INSTANCEID

As a security measure, make sure that ~/.awssecret is not readable by group or other (chmod go-rwx ~/.awssecret).

Windows Download and Configuration

Kenzo writes:

Curl checks the directory that curl.exe runs from, followed by all the directories in the PATH, searching for the file "curl-ca-bundle.crt".

So, folks need to rename ca-bundle.crt to curl-ca-bundle.crt, and then place it in the directory where they installed curl.

There is probably a way to configure curl to look for the certificate file elsewhere; I found that curl attempted to open %appdata%\_curlrc, so a configuration file could likely be placed here that specifies another location. However, I didn't experiment with this, as I was satisfied with keeping it with the executable (although this is mildly less secure).

To download and configure, follow these steps:

1. Install curl. Search for "curl download". The first result should be http://curl.haxx.se/download.html. In the "Win32 Generic" section, choose the "Win32" binary with SSL. The download is a .zip file. Extract curl.exe and place in the C:\Windows directory, or some other place that is in the path. Test by opening a cmd window, then do "cd \" and "curl". It should find curl.
2. Download Strawberry Perl from strawberryperl.com.
3. Download the latest aws to C:\Windows or your current directory, and rename to aws.pl.
4. Put your AWS credentials in C:\Documents and Settings\<YOUR NAME>\.awssecret (see the example file in the above Download section) or set the EC2_ACCESS_KEY and EC2_SECRET_KEY environment variables.
5. Try aws.pl --insecure-aws ls -- you should get a list of your S3 buckets.

The following command will list your S3 buckets:

aws.pl --insecure-aws ls

set AWS_ACCESS_KEY_ID=XXX set AWS_SECRET_ACCESS_KEY=XXX aws.pl --insecure-aws %1 %2 %3 %4 %5 %6 %7 %8 %9

Overview

aws is a very simple, light weight interface to Amazon EC2 and Amazon S3, that implements basic commands like ec2run and s3ls.

The command-line interface works like this:

aws ACTION [PARAMETERS]

For example, you can start an EC2 server instance:

aws run

Or, you can store a file named file7.txt to S3 in bucket mybucket, object sample.txt:

aws put mybucket/sample.txt file7.txt

If you like, you can have aws create command-line aliases, such as ec2run and s3put, so that you don't have to prepend "aws" to each command. aws can create its own aliases as follows. This step was done for you if you installed with --install.

aws --link

Then you can perform the operations without the "aws" prefix:

ec2run
s3put mybucket/sample.txt file7.txt

~/.awssecret File Format

Note: support for s3cmd's ~/.s3cfg format has been added. If that file exists (and ~/.awssecret does not), the keys will be read from there.

1B5JYHPQCXW13GWKHAG2
2GAHKWG3+1wxcqyhpj5b1Ggqc0TIxj21DKkidjfz

Keep in mind that your Secret Access Key must be kept secret; anybody with access to that key gets full access to your EC2 and S3 instances and files. If your file permissions are too generous, a warning is generated.

You may use environment variables to specify the secrets rather than the .awssecret file. For example,

AWS_ACCESS_KEY_ID=XXX AWS_SECRET_ACCESS_KEY=XXX aws ls

EC2_ACCESS_KEY=XXX EC2_SECRET_KEY=XXX aws ls

It is possible to configure aws to use a remote signing service (see below), enabling your infrastructure to make use of the Secret Access Key without having access to it.

~/.awsrc File Format

--insecure-aws
--simple

Sanity Checking

~/.awssecret is missing ~/.awssecret not readable ~/.awssecret has wrong permissions	You must create a .awssecret file that contains the access keys for your AWS account.
local host certificates aren't working	The curl command is used to access Amazon Web Services. The connection is secured via TLS (also called SSL or https). If your host certificates aren't set properly, the TLS connection fails. Fix the problem or add --insecure-aws to ~/.awsrc.
AWS is not reachable	For some reason, curl could not access https://s3.amazonaws.com/
local host time is wrong	If your host's clock is set wrong, then requests might be signed with an invalid timestamp. The sanity check code will automatically correct for a wrong host clock.

For optimal performance, sanity checks should be disabled by creating a ~/.awsrc file. If you have reason to create .awsrc and still want sanity checks enabled, add --sanity-check to ~/.awsrc. If you want the sanity checks enabled (perhaps to correct for a wrong host clock), but you want to disable warnings, add --sanity-check and --silent to ~/.awsrc.

Sanity check warnings will not be displayed if stdin is set to something other than a tty. Thus, you might not see sanity check warnings when invoking "aws" from within a script.

S3 Reference Guide

Use --progress to display progress for large S3 transfers.

New! aws put using STDIN

When using aws put with no source file, or if the source file is -, then a temporary file is created, stdin is copied into the temporary file, and that file is uploaded. For example,

echo hello, world |aws put test681/hello.txt

The MIME type is set as appropriate for the target file name.

Using "get" with a Directory Target

If the target of a "get" operation ends with "/", or if the target is a directory, then the file name will be taken from the object name. Examples:

aws get timkay/aws-1.12 .

aws get timkay/aws-1.12 x/y/z/

aws get timkay/aws-1.12 x/y/z

Note that it is possible to use slashes in object names, such as "x/y/foo". It would be useful to say

aws get timkay/x/y/foo .

Specifying x-amz-*: and Content-Type: Headers

It is possible to send x-amz-*: and Content-Type: headers by including them on the aws command line after the verb. The following example sets the object access policies to be world-readable and also sets the content type as appropriate for a .jpg file:

aws put "x-amz-acl: public-read" "Content-Type: image/jpg" timkay681 tim.jpg

As of v1.12, if it isn't set otherwise, the Content-Type: header is automatically set using the mime.types file, which is read from both the current directory and /etc/, if they exist.

Changing the ACL of an Existing Object New!

To change the ACL of an object to a canned access policy, store an empty ACL file and include a header to indicate the desired canned access policy:

aws put "x-amz-acl: public-read" test681/tim2.jpg?acl /dev/null

aws put test681/tim2.jpg?acl --set-acl=public-read

aws put test681/tim2.jpg?acl --public

The choices for --set-acl are private, public-read, public-read-write, and authenticated-read. As of aws-1.25, --public is equivalent to --set-acl=public-read, and --private is equivalent to --set-acl=private.

For more information about canned access policies, see the Amazon S3 API. To control the ACL more precisely, see the following section, Setting Access Policies (ACLs).

Setting Access Policies (ACLs)

You can set canned access policies as described in the previous section. Otherwise, setting access policies requires that you create an XML file containing the access policies. You might start by retrieving the existing policies, modify those policies, then upload the changes, as shown in the example below. You can see that the results are publicly readable.

bash-3.00$ aws --xml get public681/tim.jpg?acl >acl.xml bash-3.00$ xmlpp acl.xml <?xml version="1.0" encoding="UTF-8"?> <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Owner> <ID>c1438ce900acb0db547b3708dc29ca60370d8174ee55305050d2990dcf27109c</ID> <DisplayName>timkay681</DisplayName> </Owner> <AccessControlList> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"> <ID>c1438ce900acb0db547b3708dc29ca60370d8174ee55305050d2990dcf27109c</ID> <DisplayName>timkay681</DisplayName> </Grantee> <Permission>FULL_CONTROL</Permission> </Grant> </AccessControlList> </AccessControlPolicy> bash-3.00$ vi acl.xml bash-3.00$ aws put public681/tim.jpg?acl acl.xml bash-3.00$ aws --xml get public681/tim.jpg?acl |xmlpp <?xml version="1.0" encoding="UTF-8"?> <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Owner> <ID>c1438ce900acb0db547b3708dc29ca60370d8174ee55305050d2990dcf27109c</ID> <DisplayName>timkay681</DisplayName> </Owner> <AccessControlList> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"> <ID>c1438ce900acb0db547b3708dc29ca60370d8174ee55305050d2990dcf27109c</ID> <DisplayName>timkay681</DisplayName> </Grantee> <Permission>FULL_CONTROL</Permission> </Grant> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"> <URI>http://acs.amazonaws.com/groups/global/AllUsers</URI> </Grantee> <Permission>READ</Permission> </Grant> </AccessControlList> </AccessControlPolicy>

For more information about about setting access policies, see the Amazon S3 documentation: "Setting Access Policy with REST".

Note that some browsers cache the content types of URLs. If you browse a URL then change it's content type, then hit refresh, the broswer might not notice the new content type. To fix this problem, you should flush the content cache. (With FireFox, use Tools/Clear Private Data with the "Cache" option checked.)

Sending <CreateBucketConfiguration> Data with "mkdir"

"put" and "mkdir" used to be the same code (you could create buckets with either). To support configuration options, they are now different. If you don't care about the Location of a bucket, then you can continue to use either. However, if you want to specify Location, then you have to use "mkdir". ("aws mkdir foo eu.xml" and "aws put foo eu.xml" do two different things: The first sets the configuration of the new bucket foo as per the file eu.xml, while the latter stores the file eu.xml in bucket foo.)

1. Create a file with the <CreateBucketConfiguration> XML tag in it. See eu.xml as an example.
2. Create the bucket with

   aws mkdir BUCKETNAME CONFIGURATIONFILE

   For example, you could say

   aws mkdir my-new-bucket eu.xml

3. To see the Location constraint of a bucket, use the ?location attribute:

   aws get BUCKETNAME?location

   For example, you could say

   aws --xml get my-new-bucket?location |xmlpp

   returns

   <?xml version="1.0" encoding="UTF-8"?> <LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">EU</LocationConstraint>

   Notes: I am using the --xml output format, until I get xml2tab working in this case. xmlpp is at http://timkay.com/xmlpp/

EC2 Reference Guide

Use "aws COMMAND -h" to see the options available for each command. For example,

$ ./aws run -h
$ ./aws run -h Usage: aws run-instance run [ImageId (ami-23b6534a)] [-i InstanceType (ml.small)] [-a AddressingType (public)] [-n MinCount (1)] [-n MaxCount (1)] [-g SecurityGroup... (default)] [-k KeyName (default)] [-d UserData] [-f UserData]

Note. The following command groups are now supported, but the following table has not been updated. The syntax is the same as Amazon's command-line tools.

• registering images
• availability zones
• kernel types
• persistent IP addresses

Note: Elastic Block Storage is now supported (as of v1.14), but the following table has not yet been updated. It works the same as the Amazon documentation: Amazon Elastic Compute Cloud.

Type "aws" at the command-line to see a list of supported commands.

Alternate Syntax		Description
aws add-group GROUP aws addgrp GROUP	ec2-add-group GROUP ec2addgrp GROUP	Creates a security group named GROUP.
aws add-keypair KEYNAME aws addkey KEYNAME	ec2-add-keypair KEYNAME ec2addkey KEYNAME	Creates a new key pair named KEYNAME and displays the private key.
aws allocate-address aws allad	ec2-allocate-address ec2allad	Allocates a new public IP address.
aws associate-address ... aws aad ...	ec2-associate-address ec2aad address -i instance [-d device]	Associates an IP address with an instance.
aws authorize ... aws auth ...	ec2-authorize ... ec2auth [-P PROTOCOL] [-p PORT_RANGE] [-u USER]       [-o SECURITY_GROUP] [-s SOURCE_SUBNET]	Creates new firewall entries.
aws delete-keypair KEYNAME aws delkey KEYNAME	ec2-delete-keypair KEYNAME ec2delkey KEYNAME	Deletes the key pair named KEYNAME.
aws delete-group GROUP aws delgrp GROUP	ec2-delete-group GROUP ec2delgrp GROUP	Deletes the security group named GROUP.
aws describe-groups [GROUP...] aws dgrp [GROUP...]	ec2-describe-group [GROUP...] ec2dgrp [GROUP...]	Describe security groups.
aws describe-keypairs [KEY...] aws dkey [KEY...]	ec2-describe-keypairs [KEY...] ec2key [KEY...]	Describe EC2 images.
aws describe-images ... aws dim [IMAGE...] [-o OWNER] [-u USER]	ec2-describe-images ... ec2dim [IMAGE...] [-o OWNER] [-u USER]	Describe EC2 images.
aws describe-instances aws din [INSTANCE...]	ec2-describe-instances ec2din [INSTANCE...]	Describe all EC2 server instances.
aws describe-regions aws dreg	ec2-describe-regions ec2dreg	Describe available EC2 regions. When issuing EC2 commands specify --region=REGION to indicate a region.
aws reboot-instances INSTANCE [INSTANCE...] aws reboot INSTANCE [INSTANCE...]	ec2-reboot-instances INSTANCE [INSTANCE...] ec2reboot INSTANCE [INSTANCE...]	Reboot EC2 server instances.
aws run-instances ... aws run	ec2-run-instances ... ec2run [--simple] [--wait=SEC]	Start N EC2 server instances. --wait=SEC to poll for instances to leave "pending" --simple to display the instanceId's in tab-separated format
aws terminate-instances INSTANCE [INSTANCE...] aws tin INSTANCE [INSTANCE...]	ec2-terminate-instances INSTANCE [INSTANCE...] ec2tin INSTANCE [INSTANCE...]	Terminate EC2 server instances.

New! --wait=SEC

As of aws-1.24, the --wait=SEC switch tells ec2run to poll for the status of the new instances (using ec2-describe-instances), waiting for them to leave "pending" status (presumably entering "running" status). Waits SEC seconds between polling attempts.

$ ./aws-1.24 run -k k2 --wait=10 i-97f276fe pending i-97f276fe pending i-97f276fe pending i-97f276fe pending i-97f276fe pending i-97f276fe running ec2-75-101-242-20.compute-1.amazonaws.com

Regions

Use --region=eu-west-1 to select the European EC2 region. eu and us are synonums for eu-west-1 and us-east-1.

Add --region=eu to ~/.awsrc to switch the default region.

Robustness and Scripting

To improve the likelihood that a transaction succeeds succeeds, "aws" sets the appropriate curl option, so that all failed transactions are retried up to 3 times.

The "aws" command --max-time=N indicates the maximum number of seconds to allow a given try to succeed. If the timeout is exceeded, it is terminated and retried up to 3 times. (By default, there is no timeout, so a transaction can potentially hang forever.)

As of v1.28, the --md5 option computes the MD5 checksum for all objects and generates an error when a mismatch occurs.

Also as of v1.28, $? is set correctly in most cases, so that scripts can tell if an "aws" transaction completes successfully.

Remote Signature Mode

It's not always a good idea to store your key on any server that needs access to Amazon Web Services.

So how does an EC2 instance access S3 data without using the Secret Access Key? By using aws in remote signature mode. Remote signature mode is a web service that signs EC2 and S3 requests remotely. The Secret Access Key is stored only on the server(s) that provide the signing service.

aws can then run on an EC2 instance, performing all operations as it normally would, except that it signs its requests using a remote call to the web service.

The web service can be protected with a password, by restricting access to known IP addresses, and/or by limiting what requests are allowed.

To use remote signature mode, two things have to happen. First, the web service needs to be configured by following these steps on the server that is to host it.

1. Make sure that aws runs properly at the command line when logged in as the user that apache runs as. (Log in as www-data, and do aws ls, for example, to see that aws and ~/.awssecret are properly configured.
2. Create a CGI that contains just this one line:

   #!/usr/bin/env aws

   and configure the web server, so that the script is password (or otherwise) protected and executable. It is also a good idea to use SSL (https) to access this service, so that the password is encrypted. To use a self-signed certificate, see the --insecure-signing setting.

   The following PHP code implements a remote signing service:

   <? list($head, $body) = explode("\n\n", shell_exec("aws --sign=" . file_get_contents("php://input"))); header("Content-type: text/plain"); echo $body; ?>

   Make sure that aws is in the web server's path, or else add the explicit path (probably /usr/bin/aws) to the script.
3. Test this remote signing service by pointing your browser to the URL. It should display something like

   VKL5xvaVRCTAsBTF63dCr5zc3I4=.

   (This particular string is the empty string, signed by my Secret Access Key.)

Next, you need to configure the client to use the remote signing service:

1. Create ~/.awssecret like this:

   _
   _
   https://user:[email protected]/aws/sign/

   The first line two lines contains underscore ("_") characters where your Access Key ID and Secret Access Key would normally go. The third line contains the URL of the signing service. If the service is password protected, the URL should contain the username and password, as indicated.
2. Test the remote signing by running "aws -v ls". You should see a message like this, followed by the output from the "ls" command.

   signing [GET\n\n\n1172520135\n/] via https://******:******@timkay.com/aws/sign/

Changes

Implementation

The rest of the program contains the command-line interface, the pretty-printer, and HMAC and SHA1 code, none of which is needed when calling ec2() or s3() from your program. In that case, use Digest::SHA1 (or Digest::SHA1::PurePerl) and parse the XML however you like. (I have no idea if ec2() and s3() can be run in isolation any more.)

export S3_DIR=bucket