Andrew Morgan on Databases

Introduction

This is the fourth in a series of blog posts examining technologies such as Angular that are driving the development of modern web and mobile applications.

“Modern Application Stack – Part 1: Introducing The MEAN Stack” introduced the technologies making up the MEAN (MongoDB, Express, Angular, Node.js) and MERN (MongoDB, Express, React, Node.js) Stacks, why you might want to use them, and how to combine them to build your web application (or your native mobile or desktop app).

The remainder of the series is focussed on working through the end to end steps of building a real (albeit simple) application. – MongoPop. Part 2: Using MongoDB With Node.js created an environment where we could work with a MongoDB database from Node.js; it also created a simplified interface to the MongoDB Node.js Driver. Part 3: Building a REST API with Express.js built on Part 2 by using Express.js to add a REST API which will be used by the clients that we implement in the final posts.

This post demonstrates how to use Angular 2 (the evolution of Angular.js) to implement a remote web-app client for the Mongopop application.

Angular 2 (recap)

Angular, originally created and maintained by Google, runs your JavaScript code within the user’s web browsers to implement a reactive user interface (UI). A reactive UI gives the user immediate feedback as they give their input (in contrast to static web forms where you enter all of your data, hit “Submit” and wait.

Version 1 of Angular was called AngularJS but it was shortened to Angular in Angular 2 after it was completely rewritten in Typescript (a superset of JavaScript) – Typescript is now also the recommended language for Angular apps to use.

You implement your application front-end as a set of components – each of which consists of your JavaScript (Typescript) code and an HTML template that includes hooks to execute and use the results from your Typescript functions. Complex application front-ends can be crafted from many simple (optionally nested) components.

Angular application code can also be executed on the back-end server rather than in a browser, or as a native desktop or mobile application.

Downloading, running, and using the Mongopop application

The Angular client code is included as part if the Mongopop package installed in Part 2: Using MongoDB With Node.js.

The back-end application should be run in the same way as in parts 2 & 3. The client software needs to be transpiled from Typescript to JavaScript – the client software running in a remote browser can then download the JavaScript files and execute them.

The existing package.json file includes a script for transpiling the Angular 2 code:

  "scripts": {
        ...
    "tsc:w": "cd public && npm run tsc:w",
        ...  
},

That tsc:w delegates the work to a script of the same name defined in public/package.json;

  "scripts": {
        ...
    "tsc:w": "tsc -w",
        ...  
},

tsc -w continually monitors the client app’s Typescript files and reruns the transpilation every time they are edited.

To start the continual transpilation of the Angular 2 code:

npm run rsc:w

Component architecture of the Mongopop Angular UI

Angular applications (both AngularJS and Angular2) are built from one or more, nested components – Mongopop is no exception:

The main component (AppComponent)contains the HTML and logic for connecting to the database and orchestrating its sub-components. Part of the definition of AppComponent is meta data/decoration to indicate that it should be loaded at the point that a my-app element (<my-app></my-app>) appears in the index.html file (once the component is running, its output replaces whatever holding content sits between <my-app> and </my-app>). AppComponent is implemented by:

• A Typescript file containing the AppComponent class (including the data members, initialization code, and member functions
• A HTML file containing
  • HTML layout
  • Rendering of data members
  • Elements to be populated by sub-components
  • Data members to be passed down for use by sub-components
  • Logic (e.g. what to do when the user changes the value in a form)
• (Optionally) a CSS file to customise the appearance of the rendered content

Mongopop is a reasonably flat application with only one layer of sub-components below AppComponent, but more complex applications may nest deeper.

Changes to a data value by a parent component will automatically be propagated to a child – it’s best practice to have data flow in this direction as much as possible. If a data value is changed by a child and the parent (either directly or as a proxy for one of its other child components) needs to know of the change, then the child triggers an event. That event is processed by a handler registered by the parent – the parent may then explicitly act on the change, but even if it does nothing explicit, the change flows to the other child components.

This table details what data is passed from AppComponent down to each of its children and what data change events are sent back up to AppComponent (and from there, back down to the other children):

AddComponent

CountComponent

UpdateComponent

SampleComponent

What are all of these files?

To recap, the files and folders covered earlier in this series:

• package.json: Instructs the Node.js package manager (npm) what it needs to do; including which dependency packages should be installed
• node_modues: Directory where npm will install packages
• node_modues/mongodb: The MongoDB driver for Node.js
• node_modues/mongodb-core: Low-level MongoDB driver library; available for framework developers (application developers should avoid using it directly)
• javascripts/db.js: A JavaScript module we’ve created for use by our Node.js apps (in this series, it will be Express) to access MongoDB; this module in turn uses the MongoDB Node.js driver.
• config.js: Contains the application–specific configuration options
• bin/www: The script that starts an Express application; this is invoked by the npm start script within the package.json file. Starts the HTTP server, pointing it to the app module in app.js
• app.js: Defines the main back-end application module (app). Configures:
  • That the application will be run by Express
  • Which routes there will be & where they are located in the file system (routes directory)
  • What view engine to use (Jade in this case)
  • Where to find the views to be used by the view engine (views directory)
  • What middleware to use (e.g. to parse the JSON received in requests)
  • Where the static files (which can be read by the remote client) are located (public directory)
  • Error handler for queries sent to an undefined route
• views: Directory containing the templates that will be used by the Jade view engine to create the HTML for any pages generated by the Express application (for this application, this is just the error page that’s used in cases such as mistyped routes (“404 Page not found”))
• routes: Directory containing one JavaScript file for each Express route
  • routes/pop.js: Contains the Express application for the /pop route; this is the implementation of the Mongopop REST API. This defines methods for all of the supported route paths.
• public: Contains all of the static files that must be accessible by a remote client (e.g., our Angular to React apps).

Now for the new files that implement the Angular client (note that because it must be downloaded by a remote browser, it is stored under the public folder):

• public/package.json: Instructs the Node.js package manager (npm) what it needs to do; including which dependency packages should be installed (i.e. the same as /package.json but this is for the Angular client app)
• public/index.html: Entry point for the application; served up when browsing to http://<backend-server>/. Imports public/system.config.js
• public/system.config.js: Configuration information for the Angular client app; in particular defining the remainder of the directories and files:
  • public/app: Source files for the client application – including the Typescript files (and the transpiled JavaScript files) together the HTML and any custom CSS files. Combined, these define the Angular components.
    • public/app/main.ts: Entry point for the Angular app. Bootstraps public/app/app.module.ts
    • public/app/app.module.ts: Imports required modules, declares the application components and any services. Declares which component to bootstrap (AppComponent which is implemented in public/app/app.component.*)
    • public/app/app.component.html: HTML template for the top-level component. Includes elements that are replaced by sub-components
    • public/app/app.component.ts: Implements the AppComponent class for the top-level component
    • public/app/X.component.html: HTML template for sub-component X
    • public/app/X.component.ts: Implements the class for sub-component X
    • AddDocsRequest.ts, ClientConfig.ts, CountDocsRequest.ts, MongoResult.ts, MongoReadResult.ts, SampleDocsRequest.ts, & UpdateDocsRequest.ts: Classes that match the request parameters and response formats of the REST API that’s used to access the back-end
    • data.service.ts: Service used to access the back-end REST API (mostly used to access the database)
    • X.js* & *X.js.map: Files which are generated by the transpilation of the Typescript files.
  • public/node-modules: Node.js modules used by the Angular app (as opposed to the Express, server-side Node.js modules)
  • public/styles.css: CSS style sheet (imported by public/index.html) – applies to all content in the home page, not just content added by the components
  • public/stylesheets/styles.css: CSS style sheet (imported by public/app/app.component.ts and the other components) – note that each component could have their own, specialized style sheet instead

“Boilerplate” files and how they get invoked

This is an imposing number of new files and this is one of the reasons that Angular is often viewed as the more complex layer in the application stack. One of the frustrations for many developers, is the number of files that need to be created and edited on the client side before your first line of component/application code is executed. The good news is that there is a consistent pattern and so it’s reasonable to fork you app from an existing project – the Mongopop app can be cloned from GitHub or, the Angular QuickStart can be used as your starting point.

As a reminder, here is the relationship between these common files (and our application-specific components):

Contents of the “boilerplate” files

This section includes the contents for each of the non-component files and then remarks on some of the key points.

public/package.json

The scripts section defines what npm should do when you type npm run <command-name> from the command line. Of most interest is the tsc:w script – this is how the transpiler is launched. After transpiling all of the .ts Typescript files, it watches them for changes – retranspiling as needed.

Note that the dependencies are for this Angular client. They will be installed in public/node_modules when npm install is run (for Mongopop, this is done automatically when building the full project ).

public/index.html

Focussing on the key lines, the application is started using the app defined in systemjs.config.js:

And the output from the application replaces the placeholder text in the my-app element:

<my-app>Loading MongoPop client app...</my-app>

public/systemjs.config.js

packages.app.main is mapped to public/app/main.js – note that main.js is referenced rather than main.ts as it is always the transpiled code that is executed. This is what causes main.ts to be run.

public/app/main.ts

This simply imports and bootstraps the AppModule class from public/app/app.module.ts (actually app.module.js)

public/app/app.module.ts

This is the first file to actually reference the components which make up the Mongopop application!

Note that NgModule is the core module for Angular and must always be imported; for this application BrowserModule, HttpModule, and FormsModule are also needed.

The import commands also bring in the (.js) files for each of the components as well as the data service.

Following the imports, the @NgModule decorator function takes a JSON object that tells Angular how to run the code for this module (AppModule) – including the list of imported modules, components, and services as well as the module/component needed to bootstrap the actual application (AppComponent).

Typescript & Observables (before getting into component code)

As a reminder from Part 1: Introducing The MEAN Stack (and the young MERN upstart); the most recent, widely supported version is ECMAScript 6 – normally referred to as /ES6/. ES6 is supported by recent versions of Chrome, Opera, Safari, and Node.js). Some platforms (e.g. Firefox and Microsoft Edge) do not yet support all features of ES6. These are some of the key features added in ES6:

• Classes & modules
• Promises – a more convenient way to handle completion or failure of synchronous function calls (compared to callbacks)
• Arrow functions – a concise syntax for writing function expressions
• Generators – functions that can yield to allow others to execute
• Iterators
• Typed arrays

Typescript is a superset of ES6 (JavaScript); adding static type checking. Angular 2 is written in Typescript and Typescript is the primary language to be used when writing code to run in Angular 2.

Because ES6 and Typescript are not supported in all environments, it is common to transpile the code into an earlier version of JavaScript to make it more portable. tsc is used to transpile Typescript into JavaScript.

And of course, JavaScript is augmented by numerous libraries. The Mongopop Angular 2 client uses Observables from the RxJS reactive libraries which greatly simplify making asynchronous calls to the back-end (a pattern historically referred to as AJAX).

RxJS Observables fulfil a similar role to ES6 promises in that they simplify the code involved with asynchronous function calls (removing the need to explicitly pass callback functions). Promises are more contained than Observables, they make a call and later receive a single signal that the asynchronous activity triggered by the call succeeded or failed. Observables can have a more complex lifecycle, including the caller receiving multiple sets of results and the caller being able to cancel the Observable.

The Mongopop application uses two simple patterns when calling functions that return an Observable; the first is used within the components to digest the results from our own data service:

In Mongopop’s use of Observables, we don’t have anything to do in the final arrow function and so don’t use it (and so it could have used the second pattern instead – but it’s interesting to see both).

The second pattern is used within the data service when making calls to the Angular 2 http module (this example also shows how we return an Observable back to the components):

Calling the REST API

The DataService class hides the communication with the back-end REST API; serving two purposes:

• Simplifying all of the components’ code
• Shielding the components’ code from any changes in the REST API signature or behavior – that can all be handled within the DataService

By adding the @Injectable decorator to the class definition, any member variables defined in the arguments to the class constructor function will be automatically instantiated (i.e. there is no need to explicitly request a new Http object):

After the constructor has been called, methods within the class can safely make use of the http data member.

As a reminder from Part 3: Building a REST API with Express.js, this is the REST API we have to interact with:

/pop/

{
"AppName": "MongoPop",
"Version": 1.0
}
        

/pop/ip

{"ip": string}

/pop/config

{
mongodb: {
    defaultDatabase: string,
    defaultCollection: string,
    defaultUri: string
},
mockarooUrl: string
}
        

/pop/addDocs

{
MongoDBURI: string;
collectionName: string;
dataSource: string;
numberDocs: number;
unique: boolean;
}
        

{
success: boolean;
count: number;
error: string;
}
        

/pop/sampleDocs

{
MongoDBURI: string;
collectionName: string;
numberDocs: number;
}
        

{
success: boolean;   
documents: string;
error: string;
}
        

/pop/countDocs

{
MongoDBURI: string; 
collectionName: string;
}
        

{
success: boolean;   
count: number;
error: string;
}
        

/pop/updateDocs

{
MongoDBURI: string;
collectionName: string;
matchPattern: Object;
dataChange: Object;
threads: number;
}
        

{
success: boolean;
count: number;
error: string;
}
        

Most of the methods follow a very similar pattern and so only a few are explained here; refer to the DataService class to review the remainder.

The simplest method retrieves a count of the documents for a given collection:

This method returns an Observable, which in turn delivers an object of type MongoResult. MongoResult is defined in MongoResult.ts:

The pop/count PUT method expects the request parameters to be in a specific format (see earlier table); to avoid coding errors, another Typescript class is used to ensure that the correct parameters are always included – CountDocsRequest:

http.post returns an Observable. If the Observable achieves a positive outcome then the map method is invoked to convert the resulting data (in this case, simply parsing the result from a JSON string into a Typescript/JavaScript object) before automatically passing that updated result through this method’s own returned Observable.

The timeout method causes an error if the HTTP request doesn’t succeed or fail within 6 minutes.

The catch method passes on any error from the HTTP request (or a generic error if error.toString() is null) if none exists.

The updateDBDocs method is a little more complex – before sending the request, it must first parse the user-provided strings representing:

• The pattern identifying which documents should be updated
• The change that should be applied to each of the matching documents

This helper function is used to parse the (hopefully) JSON string:

If the string is a valid JSON document then tryParseJSON returns an object representation of it; if not then it returns an error.

A new class (UpdateDocsRequest) is used for the update request:

updateDBDocs is the method that is invoked from the component code:

After converting the received string into objects, it delegates the actual sending of the HTTP request to sendUpdateDocs:

A simple component that accepts data from its parent

Recall that the application consists of five components: the top-level application which contains each of the add, count, update, and sample components.

When building a new application, you would typically start by designing the the top-level container and then work downwards. As the top-level container is the most complex one to understand, we’ll start at the bottom and then work up.

A simple sub-component to start with is the count component:

public/app/count.component.html defines the elements that define what’s rendered for this component:

You’ll recognise most of this as standard HTML code.

The first Angular extension is for the single input element, where the initial value (what’s displayed in the input box) is set to {{MongoDBCollectionName}}. Any name contained within a double pair of braces refers to a data member of the component’s class (public/app/count.component.ts).

When the button is clicked, countDocs (a method of the component’s class) is invoked with CountCollName.value (the current contents of the input field) passed as a parameter.

Below the button, the class data members of DocumentCount and CountDocError are displayed – nothing is actually rendered unless one of these has been given a non-empty value. Note that these are placed below the button in the code, but they would still display the resulting values if they were moved higher up – position within the HTML file doesn’t impact logic flow. Each of those messages is given a class so that they can be styled differently within the component’s CSS file:

The data and processing behind the component is defined in public/app/count.component.ts:

Starting with the @component decoration for the class:

This provides meta data for the component:

• selector: The position of the component within the parent’s HTML should be defined by a <my-count></my-count> element.
• templateUrl: The HMTL source file for the template (public/app/count.component.ts in this case – public is dropped as the path is relative)
• styleUrls: The CSS file for this component – all components in this application reference the same file: public/stylesheets/style.css

The class definition declares that it implements the OnInit interface; this means that its ngOnInit() method will be called after the browser has loaded the component; it’s a good place to perform any initialization steps. In this component, it’s empty and could be removed.

The two data members used for displaying success/failure messages are initialized to empty strings:

this.DocumentCount = "";
this.CountDocError = "";

Recall that data is passed back and forth between the count component and its parent:

CountComponent

To that end, two class members are inherited from the parent component – indicated by the @Input() decoration:

// Parameters sent down from the parent component (AppComponent)
@Input() dataService: DataService;
@Input() MongoDBCollectionName: string;

The first is an instance of the data service (which will be used to request the document count); the second is the collection name that we used in the component’s HTML code. Note that if either of these are changed in the parent component then the instance within this component will automatically be updated.

When the name of the collection is changed within this component, the change needs to be pushed back up to the parent component. This is achieved by declaring an event emitter (onCollection):

Recall that the HTML for this component invokes a member function: countDocs(CountCollName.value) when the button is clicked; that function is implemented in the component class:

After using the data service to request the document count, either the success or error messages are sent – depending on the success/failure of the requested operation. Note that there are two layers to the error checking:

1. Was the network request successful? Errors such as a bad URL, out of service back-end, or loss of a network connection would cause this check to fail.
2. Was the back-end application able to execute the request successfully? Errors such as a non-existent collection would cause this check to fail.

Note that when this.CountDocError or this.DocumentCount are written to, Angular will automatically render the new values in the browser.

Passing data down to a sub-component (and receiving changes back)

We’ve seen how CountComponent can accept data from its parent and so the next step is to look at that parent – AppComponent.

The HTML template app.component.html includes some of its own content, such as collecting database connection information, but most of it is delegation to other components. For example, this is the section that adds in CountComponent:

Angular will replace the <my-count></my-count> element with CountComponent; the extra code within that element passes data down to that sub-component. For passing data members down, the syntax is:

[name-of-data-member-in-child-component]="name-of-data-member-in-this-component"

As well as the two data members, a reference to the onCollection event handler is passed down (to allow CountComponent to propagate changes to the collection name back up to this component). The syntax for this is:

(name-of-event-emitter-in-child-component)="name-of-event-handler-in-this-component($event)"

As with the count component, the main app component has a Typescript class – defined in app.component.ts – in addition to the HTML file. The two items that must be passed down are the data service (so that the count component can make requests of the back-end) and the collection name – these are both members of the AppComponent class.

The dataService object is implicitly created and initialized because it is a parameter of the class’s constructor, and because the class is decorated with @Injectable:

MongoDBCollectionName is set during component initialization within the ngOnInit() method by using the data service to fetch the default client configuration information from the back-end:

Finally, when the collection name is changed in the count component, the event that it emits gets handled by the event handler called, onCollection, which uses the new value to update its own data member:

Conditionally including a component

It’s common that a certain component should only be included if a particular condition is met. Mongopop includes a feature to allow the user to apply a bulk change to a set of documents – selected using a pattern specified by the user. If they don’t know the typical document structure for the collection then it’s unlikely that they’ll make a sensible change. Mongopop forces them to first retrieve a sample of the documents before they’re given the option to make any changes.

The ngIf directive can be placed within the opening part of an element (in this case a <div>) to make that element conditional. This approach is used within app.component.html to only include the update component if the DataToPlayWith data member is TRUE:

Note that, as with the count component, if the update component is included then it’s passed the data service and collection name and that it also passes back changes to the collection name.

Angular includes other directives that can be used to control content; ngFor being a common one as it allows you to iterate through items such as arrays:

Returning to app.component.html, an extra handler (onSample) is passed down to the sample component:

sample.component.html is similar to the HTML code for the count component but there is an extra input for how many documents should be sampled from the collection:

On clicking the button, the collection name and sample size are passed to the sampleDocs method in sample.component.ts which (among other things) emits an event back to the AppComponent‘s event handler using the onSample event emitter:

Other code highlights

Returning to app.component.html; there is some content there in addition to the sub-components:

Most of this code is there to allow a full MongoDB URI/connection string to be built based on some user-provided attributes. Within the input elements, two event types (keyup & change) make immediate changes to other values (without the need for a page refresh or pressing a button):

The actions attached to each of these events call methods from the AppComponent class to set the data members – for example the setDBName method (from app.component.ts):

In addition to setting the dBInputs.MongoDBDatabaseName value, it also invokes the data service method calculateMongoDBURI (taken from data.service.ts ):

This method is run by the handler associated with any data member that affects the MongoDB URI (base URI, database name, socket timeout, connection pool size, or password). Its purpose is to build a full URI which will then be used for accessing MongoDB; if the URI contains a password then a second form of the URI, MongoDBURIRedacted has the password replaced with **********.

It starts with a test as to whether the URI has been left to the default localhost:27017 – in which case it’s assumed that there’s no need for a username or password (obviously, this shouldn’t be used in production). If not, it assumes that the URI has been provided by the MongoDB Atlas GUI and applies these changes:

• Change the database name from <DATATBASE> to the one chosen by the user.
• Replace <PASSWORD> with the real password (and with ********** for the redacted URI).
• Add the socket timeout parameter.
• Add the connection pool size parameter.

Testing & debugging the Angular application

Now that the full MEAN stack application has been implemented, you can test it from within your browser:

Debugging the Angular 2 client is straightforward using the Google Chrome Developer Tools which are built into the Chrome browser. Despite the browser executing the transpiled JavaScript the Dev Tools allows you to browse and set breakpoints in your Typescript code:

Summary & what’s next in the series

Previous posts stepped through building the Mongopop application back-end. This post describes how to build a front-end client using Angular 2. At this point, we have a complete, working, MEAN stack application.

The coupling between the front and back-end is loose; the client simply makes remote, HTTP requests to the back-end service – using the interface created in Part 3: Building a REST API with Express.js.

This series will finish out by demonstrating alternate methods to implement front-ends; using ReactJS for another browser-based UI (completing the MERN stack) and then more alternative methods.

Continue following this blog series to step through building the remaining stages of the Mongopop application:

• Part 1: Introducing The MEAN Stack (and the young MERN upstart)
• Part 2: Using MongoDB With Node.js
• Part 3: Building a REST API with Express.js
• Part 4: Building a Client UI Using Angular 2 (formerly AngularJS) & TypeScript
• Part 5: Using ReactJS, ES6 & JSX to Build a UI (the rise of MERN)
• Part 6: Browsers Aren’t the Only UI – Mobile Apps, Amazon Alexa, Cloud Services…

A simpler way to build your app – MongoDB Stitch, Backend as a Service

MongoDB Stitch is a backend as a service (BaaS), giving developers a REST-like API to MongoDB, and composability with other services, backed by a robust system for configuring fine-grained data access controls. Stitch provides native SDKs for JavaScript, iOS, and Android.

Built-in integrations give your application frontend access to your favorite third party services: Twilio, AWS S3, Slack, Mailgun, PubNub, Google, and more. For ultimate flexibility, you can add custom integrations using MongoDB Stitch’s HTTP service.

MongoDB Stitch allows you to compose multi-stage pipelines that orchestrate data across multiple services; where each stage acts on the data before passing its results on to the next.

Unlike other BaaS offerings, MongoDB Stitch works with your existing as well as new MongoDB clusters, giving you access to the full power and scalability of the database. By defining appropriate data access rules, you can selectively expose your existing MongoDB data to other applications through MongoDB Stitch’s API.

If you’d like to try it out, step through building an application with MongoDB Stitch.