A C++ library for communicating with Amazon S3
Amazon web services (AWS) has many SDKs available for interacting with its services.
But the one that they don't support and have no SDK for is C++.
Because of the lack of an official SDK from Amazon many people have written their own C++ code for communicating with the REST api. I decided to have a quick look to see what was out there and compile a version that could be used commercially.
There is a lack (I couldn't find any) of free to use source code examples, including project files, for use with Windows (my main development environment). I believe this is due to the complexity of having to compile and install so many libraries in order to get started (I believe OSX has some of these libraries installed by default).
So the platform that I am targeting initially is Win32 using Visual Studio 2005 and Visual Studio 2008 Express Edition as the compilers. The results of all this will be a working Visual Studio solution that anyone can download and quickly get up and running using Amazon S3 under a windows development environment.
Note: If you can't be bothered reading all this thrilling information on compilers and C++ and just want to get straight to work using a working Visual Studio solution that you can run away and compile, then you can skip to the end of this article for a link. But just read the next bit first before you do.
Before you start
Before you rush off in glee and start downloading and compiling code the very first thing you must do is create an Amazon S3 Account. If you don't have an account then the resulting tool that you are about to compile will not be of any use to you other than to let you marvel at your masterful skills of being able to compile oodles of glorious C++ code.
Once your in then go to the Account section (a link can be found at the top of the AWS Management Console).
Next go the Security Credentials section.
Then under Access Credentials select the link Create a new Access Key.
Make a copy of the "Access Key ID" and store it somewhere.
Lastly press the "Show" link under the Secret Access Key settting, copy this information as well along with your Access Key.
NEVER give anyone your Secret Access Key! Keep it hidden, keep it safe.
OK now onto the good stuff... code.
There are a number of C++ libraries available on internet.
The one I have decided to have a go at compiling is the one from http://www.arklyffe.com/main/s3/. This is under a BSD licence which means if everything proves to be successfull then you can use the library for commercial purposes as long as you display the licence with the product.
What you need to get started
Most of the above libraries that people have developed require both libcurl and OpenSSL. The one I am using also requires curlpp which is a C++ wrapper for libcurl. So first you will have to download these and get each of them compiling on your system. Some may also use the boost libraries.
If you want to see a small description of how these libraries are related then look here.
Before you can compile libcURL you will first need to compile OpenSSL (well actually this is not strictly true, you can compile without it but for communicating with Amazon S3 we need an implementation of hmac for creating the signatures for the http headers when working with the REST api. This could be done using some other libraries but the s3tools are using OpenSSL).
First thing to do is download openssl.
OpenSSL requires a few things as well to get going. First you will need perl which you can get from here.
In the OpenSSL directory you will see many INSTALL.XXX text files. The one I am following is INSTALL.W32. I am also using Visual Studio for compiling. Here is a copy of their instructions for compiling OpenSSL under Win32.
OpenSLL Visual C++ Instructions
If you want to compile in the assembly language routines with Visual
C++, then you will need the Netwide Assembler binary,
nasmw.exe or nasm.exe, to be available on your %PATH%.
Firstly you should run Configure with platform VC-WIN32:
> perl Configure VC-WIN32 --prefix=c:\some\openssl\dir
Where the prefix argument specifies where OpenSSL will be installed to.
Next you need to build the Makefiles and optionally the assembly
- If you are using NASM then run:
- If you don't want to use the assembly language files at all then run:
> perl Configure VC-WIN32 no-asm --prefix=c:/some/openssl/dir
If you get errors about things not having numbers assigned then check the
troubleshooting section: you probably won't be able to compile it as it
Then from the VC++ environment at a prompt do:
> nmake -f ms\ntdll.mak
If all is well it should compile and you will have some DLLs and
executables in out32dll. If you want to try the tests then do:
> nmake -f ms\ntdll.mak test
To install OpenSSL to the specified location do:
> nmake -f ms\ntdll.mak install
There are various changes you can make to the Win32 compile
environment. By default the library is not compiled with debugging
symbols. If you use the platform debug-VC-WIN32 instead of VC-WIN32
then debugging symbols will be compiled in.
By default in 1.0.0 OpenSSL will compile builtin ENGINES into the
separate shared librariesy. If you specify the "enable-static-engine"
option on the command line to Configure the shared library build
(ms\ntdll.mak) will compile the engines into libeay32.dll instead.
The default Win32 environment is to leave out any Windows NT specific
If you want to enable the NT specific features of OpenSSL (currently
only the logging BIO) follow the instructions above but call the batch
file do_nt.bat instead of do_ms.bat.
You can also build a static version of the library using the Makefile
After finished compiling OpenSLL
Now that it is compiled you will find the files you require in the path you set above.
You will have a directory with files in a structure like the following
- lots of files here...
- lots of files here...
Now that you have OpenSSL compiling the next step is to compile libcURL with support for OpenSSL included.
Download the Win32 - MSVC (with SSL) source from here. Open up the visual studio solution file.
Add the following pre processor definitions to the configuration you wish to build
For static libraries also add
Add the include directory of your OpenSSL directory to the visual studio project settings.
Add the lib directory of your OpenSSL directory to the visual studio project settings.
Add the two OpenSSL libraries to the project. Those being... libeay32.lib ssleay32.lib
You should now be able to compile the libcURL.
The resulting files that you will need can be found in
Download the source code from HERE. Set the include and lib directories to libcurl.
In my version a couple of files were in the wrong place in the project file. Delete the files
LifetimeLibrary.cpp and PrivateMembers.cpp and re-add them from the utilspp directory.
Add some pre-processor definitions
Build the static libraries for debug and release.
Compiling the S3Tool code
Unzip the code from here.
Create an empty visual studio console project and add all the following files
Add the libcurl, openssl and curlpp include and lib directories to the project.
Set the preprocessor definitions
You will then have to modifiy the following method
time_t t = time(NULL);
tm gmt; gmtime_s(&gmt,&t);
size_t n = strftime(bfr, 256, "%a, %d %b %Y %H:%M:%S GMT", &gmt);
bfr[n] = '\0';
And also remove some code around line 213 that trys to load in user credential files, since it doesn't compile under windows and isn't really needed anyway.
Add the libs ws2_32.lib wldap32.lib ssleay32.lib libeay32.lib curlppd.lib libcurld.lib
Now if you can't be bothered doing all that then you can just download the resulting project that I have put together (see link below).
This has been compiled and tested under Visual Studio 2005 and Visual Studio 2008 Express Edition.
It includes all the source code for libcurl, curlpp and s3tools as well as prebuilt Win32 libraries and dlls for openssl. I have removed alot of the other non-windows files from the libraries in order to strip them down, but you can still find a copy of the licences in the libcurl and curlpp directories (the files are named COPYING).
I have only setup the Win32 build for now but may add a Win64 build at a later date.
The project contains the following configurations and is all compiled with Multithreaded (/MT) flag.
The static builds only depend on the two dlls from openssl, those being libeay32.dll and ssleay32.dll.
The dynamic builds also require libcurl.dll and curlpp.dll.
When you build the solution you will get the following directory structure.
Each of these directories will contain all the required dlls and the s3tool executable.
To use the tool you can either use the command line arguments to feed in a file that contains your key and secret access key (which you need to sign up to AWS to get), or you just edit the main method of s3tool.cpp
Find the following line and enter your details.
keyID = "YOUR_KEY";
secret = "YOUR_SECRET";
Special thanks goes to the original developer of the s3tools code that enabled me to gain a better understanding of openssl, libcurl, curlpp and most importantly understanding how to actually create the signature, which was the the whole reason why I started this code hunt in the first place. And of course an extra special thanks to the developers of openssl, libcurl and curlpp that the s3tools code is built upon.