A C++ library for communicating with Amazon S3

Introduction

Amazon web services (AWS) has many SDKs available for interacting with its services.
But the one that they don't support and have no SDK for is C++.

 Because of the lack of an official SDK from Amazon many people have written their own C++ code for communicating with the REST api. I decided to have a quick look to see what was out there and compile a version that could be used commercially.
There is a lack (I couldn't find any) of free to use source code examples, including project files, for use with Windows (my main development environment). I believe this is due to the complexity of having to compile and install so many libraries in order to get started (I believe OSX has some of these libraries installed by default).

 So the platform that I am targeting initially is Win32 using Visual Studio 2005 and Visual Studio 2008 Express Edition as the compilers. The results of all this will be a working Visual Studio solution that anyone can download and quickly get up and running using Amazon S3 under a windows development environment.

 Note: If you can't be bothered reading all this thrilling information on compilers and C++ and just want to get straight to work using a working Visual Studio solution that you can run away and compile, then you can skip to the end of this article for a link. But just read the next bit first before you do.

Before you start

Before you rush off in glee and start downloading and compiling code the very first thing you must do is create an Amazon S3 Account. If you don't have an account then the resulting tool that you are about to compile will not be of any use to you other than to let you marvel at your masterful skills of being able to compile oodles of glorious C++ code.

Once your in then go to the Account section (a link can be found at the top of the AWS Management Console).

Next go the Security Credentials section.

 Then under Access Credentials select the link Create a new Access Key.

 Make a copy of the "Access Key ID" and store it somewhere.

 Lastly press the "Show" link under the Secret Access Key settting, copy this information as well along with your Access Key.

 NEVER give anyone your Secret Access Key! Keep it hidden, keep it safe.

 OK now onto the good stuff... code.

Current Implementations

There are a number of C++ libraries available on internet.

 The one I have decided to have a go at compiling is the one from http://www.arklyffe.com/main/s3/.  This is under a BSD licence which means if everything proves to be successfull then you can use the library for commercial purposes as long as you display the licence with the product.

What you need to get started

Most of the above libraries that people have developed require both libcurl and OpenSSL. The one I am using also requires curlpp which is a C++ wrapper for libcurl. So first you will have to download these and get each of them compiling on your system. Some may also use the boost libraries.

 If you want to see a small description of how these libraries are related  then look here.

Compiling libcURL

Before you can compile libcURL you will first need to compile OpenSSL (well actually this is not strictly true, you can compile without it but for communicating with Amazon S3 we need an implementation of hmac for creating the signatures for the http headers when working with the REST api. This could be done using some other libraries but the s3tools are using OpenSSL).

OpenSSL

First thing to do is download openssl.

 OpenSSL requires a few things as well to get going. First you will need perl which you can get from here.

 In the OpenSSL directory you will see many INSTALL.XXX text files.  The one I am following is INSTALL.W32. I am also using Visual Studio for compiling. Here is a copy of their instructions for compiling OpenSSL under Win32.

OpenSLL Visual C++ Instructions

If you want to compile in the assembly language routines with Visual
C++, then you will need the Netwide Assembler binary,
nasmw.exe or nasm.exe, to be available on your %PATH%.
Firstly you should run Configure with platform VC-WIN32:
> perl Configure VC-WIN32 --prefix=c:\some\openssl\dir
Where the prefix argument specifies where OpenSSL will be installed to.
Next you need to build the Makefiles and optionally the assembly
language files:
- If you are using NASM then run:
  > ms\do_nasm
- If you don't want to use the assembly language files at all then run:
  > perl Configure VC-WIN32 no-asm --prefix=c:/some/openssl/dir
  > ms\do_ms
If you get errors about things not having numbers assigned then check the
troubleshooting section: you probably won't be able to compile it as it
stands.
Then from the VC++ environment at a prompt do:
> nmake -f ms\ntdll.mak
If all is well it should compile and you will have some DLLs and
executables in out32dll. If you want to try the tests then do:
> nmake -f ms\ntdll.mak test
To install OpenSSL to the specified location do:
> nmake -f ms\ntdll.mak install
Tweaks:
There are various changes you can make to the Win32 compile
environment. By default the library is not compiled with debugging
symbols. If you use the platform debug-VC-WIN32 instead of VC-WIN32
then debugging symbols will be compiled in.
By default in 1.0.0 OpenSSL will compile builtin ENGINES into the
separate shared librariesy. If you specify the "enable-static-engine"
option on the command line to Configure the shared library build
(ms\ntdll.mak) will compile the engines into libeay32.dll instead.
The default Win32 environment is to leave out any Windows NT specific
features.
If you want to enable the NT specific features of OpenSSL (currently
only the logging BIO) follow the instructions above but call the batch
file do_nt.bat instead of do_ms.bat.
You can also build a static version of the library using the Makefile
ms\nt.mak.

After finished compiling OpenSLL

Now that it is compiled you will find the files you require in the path you set above.

 You will have a directory with files in a structure like the following
  • bin
    • libeay32.dll
    • openssl.exe
    • ssleay32.dll
  • include
    • openssl
      • lots of files here...
  • lib
    • engines
      • lots of files here...
    • libeay32.lib
    • ssleay32.lib
  • ssl
    • openssl.cnf

libcURL

Now that you have OpenSSL compiling the next step is to compile libcURL with support for OpenSSL included.

 Download the Win32 - MSVC (with SSL) source from here. Open up the visual studio solution file. 

 Add the following pre processor definitions to the configuration you wish to build
  • USE_SSLEAY
  • USE_OPENSSL
  • BUILDING_LIBCURL

 For static libraries also add
  • CURL_STATICLIB

 Add the include directory of your OpenSSL directory to the visual studio project settings.

 Add the lib directory of your OpenSSL directory to the visual studio project settings.

 Add the two OpenSSL libraries to the project.  Those being...  libeay32.lib ssleay32.lib

 You should now be able to compile the libcURL.

 The resulting files that you will need can be found in
libcurl\include
libcurl\lib\DLL-Release\libcurl.dll
libcurl\lib\DLL-Release\libcurl_imp.lib

Compling cURLpp

Download the source code from HERE.  Set the include and lib directories to libcurl.

 In my version a couple of files were in the wrong place in the project file. Delete the files 
LifetimeLibrary.cpp and PrivateMembers.cpp and re-add them from the utilspp directory.

 Add some pre-processor definitions
  • CURLPP_STATICLIB
  • CURL_STATICLIB
  • BUILDING_CURLPP

 Build the static libraries for debug and release.

Compiling the S3Tool code

Unzip the code from here.

 Create an empty visual studio console project and add all the following files
  • aws_s3.cpp
  • aws_s3.h
  • aws_s3_misc.cpp
  • aws_s3_misc.h
  • multidict.h
  • s3tools.cpp

 Add the libcurl, openssl and curlpp include and lib directories to the project.

 Set the preprocessor definitions
  • CURLPP_STATICLIB
  • CURL_STATICLIB

 You will then have to modifiy the following method
string HTTP_Date()
{
time_t t = time(NULL);
tm gmt;  gmtime_s(&gmt,&t);
char bfr[256];
size_t n = strftime(bfr, 256, "%a, %d %b %Y %H:%M:%S GMT", &gmt);
bfr[n] = '\0';
return bfr;
}

 And also remove some code around line 213 that trys to load in user credential files, since it doesn't compile under windows and isn't really needed anyway.

 Add the libs ws2_32.lib wldap32.lib ssleay32.lib libeay32.lib curlppd.lib libcurld.lib

Conclusion

Now if you can't be bothered doing all that then you can just download the resulting project that I have put together (see link below).

 This has been compiled and tested under Visual Studio 2005 and Visual Studio 2008 Express Edition.

 It includes all the source code for libcurl, curlpp and s3tools as well as prebuilt Win32 libraries and dlls for openssl. I have removed alot of the other non-windows files from the libraries in order to strip them down, but you can still find a copy of the licences in the libcurl and curlpp directories (the files are named COPYING).

 I have only setup the Win32 build for now but may add a Win64 build at a later date.

 The project contains the following configurations and is all compiled with Multithreaded (/MT) flag.
  • DebugStatic
  • DebugDynamic
  • ReleaseStatic
  • ReleaseDynamic

 The static builds only depend on the two dlls from openssl, those being libeay32.dll  and ssleay32.dll.

 The dynamic builds also require libcurl.dll and curlpp.dll.

 When you build the solution you will get the following directory structure.
  • bin
    • debugdynamic
    • debugstatic
    • releasedynamic
    • releasestatic

 Each of these directories will contain all the required dlls and the s3tool executable.

 To use the tool you can either use the command line arguments to feed in a file that contains your key and secret access key (which you need to sign up to AWS to get), or you just edit the main method of s3tool.cpp

 Find the following line and enter your details.
keyID = "YOUR_KEY";
secret = "YOUR_SECRET";

 Special thanks goes to the original developer of the s3tools code that enabled me to gain a better understanding of openssl, libcurl, curlpp and most importantly understanding how to actually create the signature, which was the the whole reason why I started this code hunt in the first place. And of course an extra special thanks to the developers of openssl, libcurl and curlpp that the s3tools code is built upon.

Comments

Post a Comment

Popular posts from this blog

Creating a renderer using a video post plugin

Introduction In this tutorial we will be setting up the basic framework for rendering images to the viewport in Cinema4D. And to test this we will be creating a VideoPost plugin that will add a new renderer, our renderer, to the list of available renderers in Cinema4D. We will then set the renderer to simply color the entire viewport red. Before we begin make sure you have Visual Studio C++ Express Edition and Cinema4D 11.5 installed. If you do not have these tools setup then I suggest you follow through this tutorial to get your development environment setup .
Read more

Selecting a Game Engine

Before you can select your game engine you need to make a few choices. Who is my target audience? Is it PC, PS3, iPhones or anyone with a webbrowser? Then you will need to start looking for technology that you can use to develop for that platform. This may give you a number of different solutions to choose from, after which you then pick one that best suits your needs based on your skill set and requirements. The game platform(s) Before looking for the game engine you need to decide what platform (or platforms) you are going to be targeting and how you would like to deliver the MMO to your user. Firstly lets look at a few delivery mechanisms.
Read more