Purpose
The purpose of this project is to show a way to get your AWS Lambda (Node.js) function code to communicate with an RDS/AuroraDB instance in its own VPC (this means it is not exposed to the whole Internet - which was something that was done prior to policy rollouts by AWS in the late 2015, early 2016 timeframe).
There really is not much you have to do to get a Lambda function to communicate with a mySQL table in an RDS/AuroraDB instance (in its own VPC), thanks to new policy capability that you attach to your Lambda function. The more complex requirement (if you are not writing your own Node.js mySQL connectors) is the use of a 3rd party Node.js mySQL connector. The one that I'll be using in this example (an NPM) has been out for a while, but little documentation exists on how to use it in an AWS Lambda scenario (aside from needing a bunch of other 3rd party NPM files that it is dependent upon). Luckily, there is a single zip file (this project) with all the files needed for your Lambda function to use. While that will save some hassle on determining what you need to get and where to put it all (and how to zip it up and upload to AWS Lambda), the zip is only current to 9/2016 - so it won't have future updates that one of the 3rd party libraries may roll out.
(Enlarge)
|
- As you know, a Lambda function must be granted "permissons" to perform activities such as execution. In this example, we add the policy "AWSLambdaVPCAccessExecutionRole" to the Lambda role that you attach to a Lambda function.
|
(Enlarge)
|
- As an aside, if you have not created a "package" for AWS Lambda with Node.js, the library of functions (in this case, the mysql library that allows you to connect with the mysql server in RDS/Aurora) is placed in a folder called "node_modules". YOUR lambda function (in this example) is "index.js" external to that folder...so this is where you have YOUR code.
- The screenshot here shows how to create a zip file (the infamous "package") to upload as a Lambda function.
|
(Enlarge)
|
- In this screenshot I show the creation of a new Lambda function and the steps you take to be able to connect it to an RDS/Aurora instance in its own VPC. Pretty straight forward, isn't it?
- As mentioned earlier in this tutorial, in order to connect to RDS/Aurora DB from a Lambda Node.js function you will need the Node.js mysql library (which has many dependencies). You can find the mysql library for Node.js here.
- You can download the project used for this tutorial (with all dependencies so you don't need to go hunting for dependencies) HERE.
Remember, the project used for this tutorial uses the root "index.js" as the equivalent of your custom AWS Lambda Node.js function (that you would change for your needs) - the example code in that file is meant to just show the basics of making a mysql request to the RDS/Aurora DB.
- If you have an RDS snapshot already taken of your database, it is really simple to copy into Aurora (via "Migrate Snapshot"). For more, please go here.
- And lastly, if you are wondering if you are able to connect and run a query or not, you can run a query such as that given below which will provide a list of table names in the database:
SELECT table_name FROM information_schema.tables WHERE table_schema='your-database-name';
If you are testing to see if you are connected to the database you want to connect to then you could use:
SHOW TABLES;
|
Have more than one mySQL query in a single function?
If you have more than one mySQL query in a Lambda function (with Node.js), you cannot use a regular connection.end() at the end of the code (as shown in the prior example). You have a few solutions to work around that limitation (due to Node.js), but a straight-forward way to do it would be:
function mySQLexitHandler(options, err) {
/* mySQL connection closes after the 1st invocation, so this handler ensures it does not close until the Lamba function exits */
connection.end();
//if (options.cleanup) { console.log('clean'); }
if (err) { console.log(err.stack); }
if (options.exit) { process.exit(); }
}
exports.handler = function(event, context) {
/* Create the mySQL connection to use for this function */
var connection = mysql.createConnection(mysqlconnect);
/* Execute Query 1 */
connection.query(mysqlstatement, function(err, rows, fields) {
/* Execute Query 2 */
connection.query(mysqlstatement, function(err, rows, fields) {
...
});
});
/* Free the connection - due to multiple mySQL calls the regular "connection.end()" cannot be used here */
process.on('exit', mySQLexitHandler.bind(null, {cleanup: true}));
};
AWS Lambda Dynamo and Aurora Interoperability Limitation
Interestingly enough, with AWS Lambda you can create a function that can connect to Dynamo OR you can create a function that can connect to Aurora. But you cannot create a function that can connect to Dynamo AND Aurora. Let’s say, for example, you have a Dynamo table containing authentication information for a user, and in Aurora you have multiple tables with data for the user that they may access by using different search queries. You cannot create a single AWS Lambda function that would connect to Dynamo to check their authentication and then (if they passed) send a query over to Aurora to return data the user may be searching for. At this time, if you do not want to create a separate function for Dynamo and another for Aurora you'll need to decide which database you want to use and drop the other. The only other solution in this context (if you want to retain usage of both) is to duplicate the Dynamo table(s) for authentication over to Aurora and then write the AWS Lambda function so it does the authentication and the search query.
AWS API Gateway Automatically Caps Your Data from Lambda/Aurora
When you are developing a Lambda function that connects to Aurora DB to get data, you may find out that the Lambda function will get matching ALL rows of data, up to 6MB, for your query in Aurora DB (e.g., more than 1000 rows). However, after you plug your Lambda function into the AWS API Gateway, instead of the outgoing data being capped at 10MB for the AWS API Gateway payload, the outgoing data is instead capped at 1000 rows regardless of data size - for example: if your Lambda function returns more than 1000 rows that contain nothing but a single column of ID's from Aurora DB (the data size being well below the 6MB/10MB data limit), the row restriction will be triggered and limit data to 1000 rows instead. You can find out more about these AWS API Gateway limitations here.
