Andrew Morgan on Databases

Introduction

This is the third in a series of blog posts examining the technologies that are driving the development of modern web and mobile applications.

Part 1: Introducing The MEAN Stack (and the young MERN upstart) introduced the technologies making up the MEAN (MongoDB, Express, Angular, Node.js) and MERN (MongoDB, Express, React, Node.js) Stacks, why you might want to use them, and how to combine them to build your web application (or your native mobile or desktop app).

The remainder of the series is focused on working through the end to end steps of building a real (albeit simple) application. – MongoPop. Part 2: Using MongoDB With Node.js created an environment where we could work with a MongoDB database from Node.js; it also created a simplified interface to the MongoDB Node.js Driver.

This post builds on from those first posts by using Express to build a REST API so that a remote client can work with MongoDB. You will be missing a lot of context if you have skipped those posts – it’s recommended to follow through those first.

The REST API

A Representational State Transfer (REST) interface provides a set of operations that can be invoked by a remote client (which could be another service) over a network, using the HTTP protocol. The client will typically provide parameters such as a string to search for or the name of a resource to be deleted.

Many services provide a REST API so that clients (their own and those of 3rd parties) and other services can use the service in a well defined, loosely coupled manner. As an example, the Google Places API can be used to search for information about a specific location:

Breaking down the URI used in that curl request:

• No method is specified and so the curl command defaults to a HTTP GET.
• maps.googleapis.com is the address of the Google APIs service.
• /maps/api/place/details/json is the route path to the specific operation that’s being requested.
• placeid=ChIJKxSwWSZgAUgR0tWM0zAkZBc is a parameter (passed to the function bound to this route path), identifying which place we want to read the data for.
• key=AIzaSyC53qhhXAmPOsxc34WManoorp7SVN_Qezo is a parameter indicating the Google API key, verifying that it’s a registered application making the request (Google will also cap, or bill for, the number of requests made using this key).

There’s a convention as to which HTTP method should be used for which types of operation:

• GET: Fetches data
• POST: Adds new data
• PUT: Updates data
• DELETE: Removes data

Mongopop’s REST API breaks this convention and uses POST for some read requests (as it’s simpler passing arguments than with GET).

These are the REST operations that will be implemented in Express for Mongopop:

/pop/

{
"AppName": "MongoPop",
"Version": 1.0
}
        

/pop/ip

{"ip": string}

/pop/config

{
mongodb: {
    defaultDatabase: string,
    defaultCollection: string,
    defaultUri: string
},
mockarooUrl: string
}
        

/pop/addDocs

{
MongoDBURI: string;
collectionName: string;
dataSource: string;
numberDocs: number;
unique: boolean;
}
        

{
success: boolean;
count: number;
error: string;
}
        

/pop/sampleDocs

{
MongoDBURI: string;
collectionName: string;
numberDocs: number;
}
        

{
success: boolean;   
documents: string;
error: string;
}
        

/pop/countDocs

{
MongoDBURI: string; 
collectionName: string;
}
        

{
success: boolean;   
count: number;
error: string;
}
        

/pop/updateDocs

{
MongoDBURI: string;
collectionName: string;
matchPattern: Object;
dataChange: Object;
threads: number;
}
        

{
success: boolean;
count: number;
error: string;
}
        

Express

Express is the web application framework that runs your back-end application (JavaScript) code. Express runs as a module within the Node.js environment.

Express can handle the routing of requests to the right functions within your application (or to different apps running in the same environment).

You can run the app’s full business logic within Express and even use an optional view engine to generate the final HTML to be rendered by the user’s browser. At the other extreme, Express can be used to simply provide a REST API – giving the front-end app access to the resources it needs e.g., the database.

The Mongopop application uses Express to perform two functions:

• Send the front-end application code to the remote client when the user browses to our app
• Provide a REST API that the front-end can access using HTTP network calls, in order to access the database

Downloading, running, and using the application

The application’s Express code is included as part of the Mongopop package installed in Part 2: Using MongoDB With Node.js.

What are all of these files?

A reminder of the files described in Part 2:

• package.json: Instructs the Node.js package manager (npm) on what it needs to do; including which dependency packages should be installed
• node_modues: Directory where npm will install packages
• node_modues/mongodb: The MongoDB driver for Node.js
• node_modues/mongodb-core: Low-level MongoDB driver library; available for framework developers (application developers should avoid using it directly)
• javascripts/db.js: A JavaScript module we’ve created for use by our Node.js apps (in this series, it will be Express) to access MongoDB; this module in turn uses the MongoDB Node.js driver.

Other files and directories that are relevant to our Express application:

• config.js: Contains the application–specific configuration options
• bin/www: The script that starts an Express application; this is invoked by the npm start script within the package.json file. Starts the HTTP server, pointing it to the app module in app.js
• app.js: Defines the main application module (app). Configures:
  • That the application will be run by Express
  • Which routes there will be & where they are located in the file system (routes directory)
  • What view engine to use (Jade in this case)
  • Where to find the /views/ to be used by the view engine (views directory)
  • What middleware to use (e.g. to parse the JSON received in requests)
  • Where the static files (which can be read by the remote client) are located (public directory)
  • Error handler for queries sent to an undefined route
• views: Directory containing the templates that will be used by the Jade view engine to create the HTML for any pages generated by the Express application (for this application, this is just the error page that’s used in cases such as mistyped routes (“404 Page not found”))
• routes: Directory containing one JavaScript file for each Express route
  • routes/pop.js: Contains the Express application for the /pop route; this is the implementation of the Mongopop REST API. This defines methods for all of the supported route paths.
• public: Contains all of the static files that must be accessible by a remote client (e.g., our Angular to React apps). This is not used for the REST API and so can be ignored until Parts 4 and 5.

The rest of the files and directories can be ignored for now – they will be covered in later posts in this series.

Architecture

The new REST API (implemented in routes/pop.js) uses the javascripts/db.js database layer implemented in Part 2 to access the MongoDB database via the MongoDB Node.js Driver. As we don’t yet have either the Angular or React clients, we will user the curl command-line tool to manually test the REST API.

Code highlights

config.js

The config module can be imported by other parts of the application so that your preferences can be taken into account.

expressPort is used by bin/www to decide what port the web server should listen on; change this if that port is already in use.

client contains defaults to be used by the client (Angular or React). It’s important to create your own schema at Mockaroo.com and replace client.mockarooUrl with your custom URL (the one included here will fail if used too often).

bin/www

This is mostly boiler-plate code to start Express with your application. This code ensures that it is our application, app.js, that is run by the Express server:

This code uses the expressPort from config.js as the port for the server to listen on; it will be overruled if the user sets the PORT environment variable:

app.js

This file defines the app module ; much of the contents are boilerplate (and covered by comments in the code) but we look here at a few of the lines that are particular to this application.

Make this an Express application:

Define where the views (templates used by the Jade view engine to generate the HTML code) and static files (files that must be accessible by a remote client) are located:

Create the /pop route and associate it with the file containing its code (routes/pop.js):

routes/pop.js

This file implements each of the operations provided by the Mongopop REST API. Because of the the /pop route defined in app.js Express will direct any URL of the form http://<mongopop-server>:3000/pop/X here. Within this file a route handler is created in order direct incoming requests to http://<mongopop-server>:3000/pop/X to the appropriate function:

As the /pop route is only intended for the REST API, end users shouldn’t be browsing here but we create a top level handler for the GET method in case they do:

This is the first time that we see how to send a response to a request; res.json(testObject); converts testObject into a JSON document and sends it back to the requesting client as part of the response message.

The simplest useful route path is for the GET method on /pop/ip which sends a response containing the IP address of the back-end server. This is useful to the Mongopop client as it means the user can see it and add it to the MongoDB Atlas whitelist. The code to determine and store publicIP is left out here but can be found in the full source file for pop.js.

We’ve seen that it’s possible to test GET methods from a browser’s address bar; that isn’t possible for POST methods and so it’s useful to be able to test using the curl command-line command:

The GET method for /pop/config is just as simple – responding with the client-specific configuration data:

The results of the request are still very simple but the output from curl is already starting to get messy; piping it through python -mjson.tool makes it easier to read:

The simplest operation that actually accesses the database is the POST method for the /pop/countDocs route path:

database is an instance of the object prototype defined in javascripts/db (see The Modern Application Stack – Part 2: Using MongoDB With Node.js) and so all this method needs to do is use that object to:

• Connect to the database (using the address of the MongoDB server provided in the request body). The results from the promise returned by database.connect is passed to the function(s) in the first .then clause. Refer back to Part 2: Using MongoDB With Node.js if you need a recap on using promises.
• The function in the .then clause handles the case where the database.connect promise is resolved (success). This function requests a count of the documents – the database connection information is now stored within the database object and so only the collection name needs to be passed. The promise returned by database.countDocuments is passed to the next .then clause. Note that there is no second (error) function provided, and so if the promise from database.connect is rejected, then that failure passes through to the next .then clause in the chain.
• The second .then clause has two functions:
  • The first is invoked if and when the promise is resolved (success) and it returns a success response (which is automatically converted into a resolved promise that it passed to the final .then clause in the chain). count is the value returned when the promise from the call to database.countDocuments was resolved.
  • The second function handles the failure case (could be from either database.connect or database.countDocuments) by returning an error response object (which is converted to a resolved promise).
• The final .then clause closes the database connection and then sends the HTTP response back to the client; the response is built by converting the resultObject (which could represent success or failure) to a JSON string.

Once more, curl can be used from the command-line to test this operation; as this is a POST request, the --data option is used to pass the JSON document to be included in the request:

curl can also be used to test the error paths. Cause the database connection to fail by using the wrong port number in the MongoDB URI:

Cause the count to fail by using the name of a non-existent collection:

The POST method for the pop/sampleDocs route path works in a very similar way:

Testing this new operation:

The POST method for pop/updateDocs is a little more complex as the caller can request multiple update operations be performed. The simplest way to process multiple asynchronous, promise-returning function calls in parallel is to build an array of the tasks and pass it to the Promise.all method which returns a promise that either resolves after all of the tasks have succeeded or is rejected if any of the tasks fail:

Testing with curl:

The final method uses example data from a service such as Mockaroo to populate a MongoDB collection. A helper function is created that makes the call to that external service:

That function is then used in the POST method for /pop/addDocs:

This method is longer than the previous ones – mostly because there are two paths:

• In the first path, the client has requested that a fresh set of 1,000 example documents be used for each pass at adding a batch of documents. This path is much slower and will eat through your Mockaroo quota much faster.
• In the second path, just one batch of 1,000 example documents is fetched from Mockaroo and then those same documents are repeatedly added. This path is faster but it results in duplicate documents (apart from a MongoDB-created _id field). This path cannot be used if the _id is part of the example documents generated by Mockaroo.

So far, we’ve used the Chrome browser and the curl command-line tool to test the REST API. A third approach is to use the Postman Chrome app:

Debugging Tips

One way to debug a Node.js application is to liberally sprinkle console.log messages throughout your code but that takes extra effort and bloats your code base. Every time you want to understand something new, you must add extra logging to your code and then restart your application.

Developers working with browser-side JavaScript benefit from the excellent tools built into modern browsers – for example, Google’s Chrome Developer Tools which let you:

• Browse code (e.g. HTML and JavaScript)
• Add breakpoints
• View & alter contents of variables
• View and modify css styles
• View network messages
• Access the console (view output and issue commands)
• Check security details
• Audit memory use, CPU, etc.

You open the Chrome DevTools within the Chrome browser using “View/Developer/Developer Tools”.

Fortunately, you can use the node-debug command of node-inspector to get a very similar experience for Node.js back-end applications. To install node-inspector:

node-inspector can be used to debug the Mongopop Express application by starting it with node-debug via the express-debug script in package.json:

To run the Mongopop REST API with node-debug, kill the Express app if it’s already running and then execute:

Note that this automatically adds a breakpoint at the start of the app and so you will need to skip over that to run the application.

Depending on your version of Node.js, you may see this error:

If you do, apply this patch to /usr/local/lib/node_modules/node-inspector/lib/InjectorClient.js.

Summary & what’s next in the series

Part 1: Introducing The MEAN Stack provided an overview of the technologies that are used by modern application developers – in particular, the MERN and MEAN stacks. Part 2: Using MongoDB With Node.js set up Node.js and the MongoDB Driver and then used them to build a new Node.js module to provide a simplified interface to the database.

This post built upon the first two of the series by stepping through how to implement a REST API using Express. We also looked at three different ways to test this API and how to debug Node.js applications. This REST API is required by both the Angular (Part 4) and React (Part 5) web app clients, as well as by the alternative UIs explored in Part 6.

The next part of this series implements the Angular client that makes use of the REST API – at the end of that post, you will understand the end-to-end steps required to implement an application using the MEAN stack.

Continue to follow this blog series to step through building the remaining stages of the MongoPop application:

• Part 1: Introducing The MEAN Stack (and the young MERN upstart)
• Part 2: Using MongoDB With Node.js
• Part 3: Building a REST API with Express.js
• Part 4: Building a Client UI Using Angular 2 (formerly AngularJS) & TypeScript
• Part 5: Using ReactJS, ES6 & JSX to Build a UI (the rise of MERN)
• Part 6: Browsers Aren’t the Only UI – Mobile Apps, Amazon Alexa, Cloud Services…

A simpler way to build your app – MongoDB Stitch, Backend as a Service

MongoDB Stitch is a backend as a service (BaaS), giving developers a REST-like API to MongoDB, and composability with other services, backed by a robust system for configuring fine-grained data access controls. Stitch provides native SDKs for JavaScript, iOS, and Android.

Built-in integrations give your application frontend access to your favorite third party services: Twilio, AWS S3, Slack, Mailgun, PubNub, Google, and more. For ultimate flexibility, you can add custom integrations using MongoDB Stitch’s HTTP service.

MongoDB Stitch allows you to compose multi-stage pipelines that orchestrate data across multiple services; where each stage acts on the data before passing its results on to the next.

Unlike other BaaS offerings, MongoDB Stitch works with your existing as well as new MongoDB clusters, giving you access to the full power and scalability of the database. By defining appropriate data access rules, you can selectively expose your existing MongoDB data to other applications through MongoDB Stitch’s API.

If you’d like to try it out, step through building an application with MongoDB Stitch.