Earlier this week, I presented on microservices at MongoDB’s Big Data event in Frankfurt. You can view the slides here.
Organisations are building their applications around microservice architectures because of the flexibility, speed of delivery, and maintainability they deliver. In this session, the concepts behind containers and orchestration will be explained and how to use them with MongoDB.
I recently co-presented a webinar with David Tucker from Confluent.
The replay is now available: Data Streaming with Apache Kafka & MongoDB.
A new generation of technologies is needed to consume and exploit today’s real time, fast moving data sources. Apache Kafka, originally developed at LinkedIn, has emerged as one of these key new technologies.
This webinar explores the use-cases and architecture for Kafka, and how it integrates with MongoDB to build sophisticated data-driven applications that exploit new sources of data.
Watch the webinar to learn:
Developers love working with MongoDB. One reason is the flexible data model, another is that there’s an idiomatic driver for just about every programming language and someone’s probably already built a framework on top of MongoDB that takes care of a lot of the grunt work. With high availability and scaling built in, they can also be confident that MongoDB will continue to meet their needs as their business grows.
MongoDB Atlas provides all of the features of MongoDB, without the operational heavy lifting required for any new application. MongoDB Atlas is available on demand through a pay-as-you-go model and billed on an hourly basis, letting you focus on what you do best.
It’s easy to get started – use a simple GUI to select the instance size, region, and features you need (Figure 1).
Figure 1: Create MongoDB Atlas Cluster
MongoDB Atlas provides:
This post provides instructions on how to use MongoDB Atlas directly from your application or how to configure your favorite framework to use it. It goes on to provide links to some worked examples for specific frameworks.
Detailed walkthroughs are available for specific programming languages and frameworks:
This list will be extended as new blog posts are produced. If your preferred language or framework isn’t listed above then read on as the following, generic instructions cover most other cases.
Launch your MongoDB cluster using MongoDB Atlas and then (optionally) create a user with read and write privileges for just the database that will be used for your application, as shown in Figure 2.
Figure 2: Creating an Application user in MongoDB Atlas
You must also add the IP address of your application server to the IP Whitelist in the MongoDB Atlas security tab (Figure 3). Note that if multiple application servers will be accessing MongoDB Atlas then an IP address range can be specified in CIDR format (IP Address/number of significant bits).
Figure 3: Add App Server IP Address(es) to MongoDB Atlas
The exact way that you specify how to connect to MongoDB Atlas will vary depending on your programming language and (optionally) the framework you’re using. However it’s pretty universal that you’ll need to provide a connection string/URI. The core of this URI can be retrieved by clicking on the CONNECT button for your cluster in the MongoDB Atlas GUI, selecting the MongoDB Drivers tab and then copying the string (Figure 4).
Figure 4: Copy MongoDB Atlas Connection String/URI
Note that this URI contains the administrator username for your MongoDB Atlas group and will connect to the admin database – you’ll probably want to change that.
Your final URI should look something like this:
mongodb://appuser:my_password@cluster0-shard-00-00-qfovx.mongodb.net:27017,cluster0-shard-00-01-qfovx.mongodb.net:27017,cluster0-shard-00-02-qfovx.mongodb.net:27017/appdatabase?ssl=true&authSource=admin'
The URI contains these components:
At this point, you should add some test data through your application and then confirm that it’s being correctly stored in MongoDB Atlas.
MongoDB Compass is the GUI for MongoDB, allowing you to visually explore your data and interact with your data with full CRUD functionality. The same credentials can be used to connect Compass to your MongoDB database (Figure 5).
Figure 5: Connect MongoDB Compass to MongoDB Atlas
Once connected, explore the data added to your collections (Figure 6).
Figure 6: Explore MongoDB Atlas Data Using MongoDB Compass
It is also possible to add, delete, and modify documents (Figure 7).
Figure 7: Modify a Document in MongoDB Compass
You can verify that the document has really been updated from the MongoDB shell:
Cluster0-shard-0:PRIMARY> use appdatabase
Cluster0-shard-0:PRIMARY> db.simples.find({
first_name: "Stephanie",
last_name: "Green"}).pretty()
{
"_id" : ObjectId("57a206be0e8ecb0d5b5549f9"),
"first_name" : "Stephanie",
"last_name" : "Green",
"email" : "sgreen1b@tiny.cc",
"gender" : "Female",
"ip_address" : "129.173.45.61",
"children" : [
{
"first_name" : "Eugene",
"birthday" : "8/25/1985"
},
{
"first_name" : "Nicole",
"birthday" : "12/29/1963",
"favoriteColor" : "Yellow"
}
]
}
This post has assumed that you’re building a new application but what if you already have one, with data stored in a MongoDB cluster that you’re managing yourself? Fortunately, the process to migrate your data to MongoDB Atlas (and back out again if desired) is straightforward and is described in Migrating Data to MongoDB Atlas.
We offer a MongoDB Atlas Migration service to help you properly configure MongoDB Atlas and develop a migration plan. This is especially helpful if you need to minimize downtime for your application, if you have a complex sharded deployment, or if you want to revise your deployment architecture as part of the migration. Contact us to learn more about the MongoDB Atlas Migration service.
While MongoDB Atlas radically simplifies the operation of MongoDB there are still some decisions to take to ensure the best performance and reliability for your application. The MongoDB Atlas Best Practices white paper provides guidance on best practices for deploying, managing, and optimizing the performance of your database with MongoDB Atlas.
The guide outlines considerations for achieving performance at scale with MongoDB Atlas across a number of key dimensions, including instance size selection, application patterns, schema design and indexing, and disk I/O. While this guide is broad in scope, it is not exhaustive. Following the recommendations in the guide will provide a solid foundation for ensuring optimal application performance.
KeystoneJS is an open source framework for building web applications and Content Management Systems. It’s built on top of MongoDB, Express, and Node.js – key components of the ubiquitous MEAN stack.
This post explains why MongoDB Atlas is an ideal choice for KeystoneJS and then goes on to show how to configure KeystoneJS to use it.
The MEAN stack is extremely popular and well supported and it’s the go to platform when developing modern applications. For its part, MongoDB brings flexible schemas, rich queries, an idiomatic Node.js driver, and simple to use high availability and scaling.
MongoDB Atlas provides all of the features of MongoDB, without the operational heavy lifting required for any new application. MongoDB Atlas is available on demand through a pay-as-you-go model and billed on an hourly basis, letting you focus on what you do best.
It’s easy to get started – use a simple GUI to select the instance size, region, and features you need. MongoDB Atlas provides:
Like KeystoneJS, MongoDB Atlas is a natural fit for users looking to simplify their development and operations work, letting them focus on what makes their application unique rather than commodity (albeit essential) plumbing.
Before starting with KeystoneJS, you should launch your MongoDB cluster using MongoDB Atlas and then (optionally) create a user with read and write privileges for just the database that will be used for this project, as shown in Figure 1. You must also add the IP address of your application server to the IP Whitelist in the MongoDB Atlas security tab.
Figure 1: Creating KeystoneJS user in MongoDB Atlas
If it isn’t already installed on your system, download and install Node.js:
You should then add the bin sub-folder to your .bash_profile file and then install KeystoneJS:
Before starting KeystoneJS you need to configure it with details on how to connect to your specific MongoDB Atlas cluster. This is done by updating the MONGO_URI value within the .env file:
The URI contains these components:
keystonejs_user is the name of the user you created in the MongoDB Atlas UImy_password is the password you chose when creating the user in MongoDB Atlascluster0-shard-00-00-qfovx.mongodb.net, cluster0-shard-00-01-qfovx.mongodb.net, & cluster0-shard-00-02-qfovx.mongodb.net are the hostnames of the instances in your MongoDB Atlas replica set (click on the “CONNECT” button in the MongoDB Atlas UI if you don’t have these)27017 is the standard MongoDB port numberclusterdb is the name of the database (schema) that KeystoneJS will use (note that this must match the project name used when installing KeystoneJS as well as the database you granted the user access to)ssl option is usedadmin is the database that’s being used to store the credentials for keystonejs_user
Clients connect to KeystoneJS through port 3000 and so you must open that port in your firewall.
You can then start KeystoneJS:
$ node keystone
Browse to the application at http://address-of-app-server:3000 as shown in Figure 2.
Figure 2: KeystoneJS Running on MongoDB Atlas
Sign in using the credentials shown and then confirm that you can upload some images to a gallery and create a new page as shown in Figure 3.
Figure 3: Create a Page in KeystoneJS with Data Stored in MongoDB Atlas
After saving the page, confirm that you can browse to the newly created post (Figure 4).
Figure 4: View KeystoneJS Post with Data Read from MongoDB Atlas
Optionally, confirm that, MongoDB Atlas really is being used by KeystoneJS, you can connect using the MongoDB shell:
To visually navigate through the schema and data created by KeystoneJS, download and install MongoDB Compass. The same credentials can be used to connect Compass to your MongoDB database – Figure 5.
Figure 5: Connect MongoDB Compass to MongoDB Atlas Database
Navigate through the structure of the data in the clusterdb database (Figure 6) and view the JSON documents (Figure 7).
Figure 6: Explore KeystoneJS Schema Using MongoDB Compass
Figure 7: View Documents Stored by KeystoneJS Using MongoDB Atlas
While MongoDB Atlas radically simplifies the operation of MongoDB there are still some decisions to take to ensure the best performance and reliability for your application. The MongoDB Atlas Best Practices white paper provides guidance on best practices for deploying, managing, and optimizing the performance of your database with MongoDB Atlas.
The guide outlines considerations for achieving performance at scale with MongoDB Atlas across a number of key dimensions, including instance size selection, application patterns, schema design and indexing, and disk I/O. While this guide is broad in scope, it is not exhaustive. Following the recommendations in the guide will provide a solid foundation for ensuring optimal application performance.
MongoDB Atlas was announced at this year’s MongoDB World. It’s great not just for new applications, but also your existing MongoDB databases running on other platforms. This post will focus on how you migrate your data and applications over to MongoDB Atlas.
MongoDB Atlas provides all of the features of MongoDB, without the operational heavy lifting required for any new application. MongoDB Atlas is available on demand through a pay-as-you-go model and billed on an hourly basis, letting you focus on what you do best.
It’s easy to get started – use a simple GUI to select the instance size, region, and features you need. MongoDB Atlas provides:
But what if you already have application data held in your own on-prem or cloud-based MongoDB database – is it possible to safely migrate that data to MongoDB Atlas? What if your data is held in a 3rd party hosted MongoDB service such as Compose or mLab? Conversely, is it possible to build your application against MongoDB Atlas and then move the data to a MongoDB database running on another platform in the future?
The answer to all of those questions is “yes”. In the future you should expect this to be a highly automated process but right now it involves some manual steps – the purpose of this blog post is to describe the process.
The procedure is very straightforward, but if you can’t tolerate losing any of your updates then it does involve stopping application writes for a period. That means it’s vital that you prepare in advance in order to minimize the impact.
mongodump & mongorestore steps but without stopping application writes to answer this.mongodump?mongorestore will be run. These will need to be added to your MongoDB Atlas group’s whitelist.laptop> mongo --host=ec2-52-208-185-213.eu-west-1.compute.amazonaws.com \
--eval "db.fsyncLock()"
dump):laptop> mongodump --host=ec2-52-208-185-213.eu-west-1.compute.amazonaws.com \
--port=27017
mongorestore --ssl --host cluster0-shard-00-00-qfovx.mongodb.net \
--port 27017 -u billy -p XXX dump
Want more help? We offer a MongoDB Atlas Migration service to help you properly configure MongoDB Atlas and develop a migration plan. This is especially helpful if you need to minimize downtime for your application, if you have a complex sharded deployment, or if you want to revise your deployment architecture as part of the migration. Contact us to learn more about the MongoDB Atlas Migration service.
To migrate data out, you can download a MongoDB Atlas backup and then copy the contents to the receiving MongoDB cluster; the documentation describes how to load the data into the receiving replica set. The backup can be either a periodic snapshot or a point-in-time view of the MongoDB Atlas database. If you can’t tolerate lost writes, they must be stopped by the application (fsyncLock is not available in MongoDB Atlas).
While MongoDB Atlas radically simplifies the operation of MongoDB there are still some decisions to take to ensure the best performance and reliability for your application. The MongoDB Atlas Best Practices white paper provides guidance on best practices for deploying, managing, and optimizing the performance of your database with MongoDB Atlas.
The guide outlines considerations for achieving performance at scale with MongoDB Atlas across a number of key dimensions, including instance size selection, application patterns, schema design and indexing, and disk I/O. While this guide is broad in scope, it is not exhaustive. Following the recommendations in the guide will provide a solid foundation for ensuring optimal application performance.
In today’s data landscape, no single system can provide all of the required perspectives to deliver real insight. Deriving the full meaning from data requires mixing huge volumes of information from many sources.
At the same time, we’re impatient to get answers instantly; if the time to insight exceeds 10s of milliseconds then the value is lost – applications such as high frequency trading, fraud detection, and recommendation engines can’t afford to wait. This often means analyzing the inflow of data before it even makes it to the database of record. Add in zero tolerance for data loss and the challenge gets even more daunting.
Kafka and data streams are focused on ingesting the massive flow of data from multiple fire-hoses and then routing it to the systems that need it – filtering, aggregating, and analyzing en-route.
This blog introduces Apache Kafka and then illustrates how to use MongoDB as a source (producer) and destination (consumer) for the streamed data. A more complete study of this topic can be found in the Data Streaming with Kafka & MongoDB white paper.
Kafka provides a flexible, scalable, and reliable method to communicate streams of event data from one or more producers to one or more consumers. Examples of events include:
Streams of Kafka events are organized into topics. A producer chooses a topic to send a given event to, and consumers select which topics they pull events from. For example, a financial application could pull NYSE stock trades from one topic, and company financial announcements from another in order to look for trading opportunities.
In Kafka, topics are further divided into partitions to support scale out. Each Kafka node (broker) is responsible for receiving, storing, and passing on all of the events from one or more partitions for a given topic. In this way, the processing and storage for a topic can be linearly scaled across many brokers. Similarly, an application may scale out by using many consumers for a given topic, with each pulling events from a discrete set of partitions.
Figure 1: Kafka Producers, Consumers, Topics, and Partitions
In order to use MongoDB as a Kafka consumer, the received events must be converted into BSON documents before they are stored in the database. In this example, the events are strings representing JSON documents. The strings are converted to Java objects so that they are easy for Java developers to work with; those objects are then transformed into BSON documents.
Complete source code, Maven configuration, and test data can be found further down, but here are some of the highlights; starting with the main loop for receiving and processing event messages from the Kafka topic:
The Fish class includes helper methods to hide how the objects are converted into BSON documents:
In a real application more would be done with the received messages – they could be combined with reference data read from MongoDB, acted on and then passed along the pipeline by publishing to additional topics. In this example, the final step is to confirm from the mongo shell that the data has been added to the database:
Fish.java
MongoDBSimpleConsumer.java
Note that this example consumer is written using the Kafka Simple Consumer API – there is also a Kafka High Level Consumer API which hides much of the complexity – including managing the offsets. The Simple API provides more control to the application but at the cost of writing extra code.
pom.xml
Fish.json
A sample of the test data injected into Kafka is shown below:
For simple testing, this data can be injected into the clusterdb-topic1 topic using the kafka-console-producer.sh command.
To learn much more about data streaming and how MongoDB fits in (including Apache Kafka and competing and complementary technologies) read the Data Streaming with Kafka & MongoDB white paper and watch the webinar replay.
PencilBlue is a Node.js based, open source blogging and Content Management System, targeted at enterprise grade websites.
This post explains why MongoDB Atlas is an ideal choice for PencilBlue and then goes on to show how to configure PencilBlue to use it.
MongoDB delivers flexible schemas, rich queries, an idiomatic Node.js driver, and simple to use high availability and scaling. This makes it the go-to database for anyone looking to build applications on Node.js.
MongoDB Atlas provides all of the features of MongoDB, without the operational heavy lifting required for any new application. MongoDB Atlas is available on demand through a pay-as-you-go model and billed on an hourly basis, letting you focus on what you do best.
It’s easy to get started – use a simple GUI to select the instance size, region, and features you need. MongoDB Atlas provides:
Like PencilBlue, MongoDB Atlas is a natural fit for users looking to simplify their development and operations work, letting them focus on what makes their application unique rather than commodity (albeit essential) plumbing.
Before starting with PencilBlue, you should launch your MongoDB cluster using MongoDB Atlas and then (optionally) create a user with read and write privileges for just the database that will be used for this project, as shown in Figure 1.
Figure 1: Adding a PencilBlue User to MongoDB Atlas
You must also add your IP address to the IP Whitelist in the MongoDB Atlas security tab (Figure 2).
Figure 2: Add IP Address to MongoDB Atlas Whitelist
If it isn’t already installed on your system, download and install Node.js:
$ curl https://nodejs.org/dist/v4.4.7/node-v4.4.7-linux-x64.tar.xz -o node.tar.xz
$ tar xf node.tar.xz
You should then add the bin sub-folder to your .bash_profile before installing the PencilBlue command line interface (CLI):
$ sudo npm install -g pencilblue-cli
Password:
npm WARN engine pencilblue-cli@0.3.1: wanted: {"node":">= 4.4.7"} (current: {"node":"0.12.5","npm":"2.11.2"})
/usr/local/bin/pencilblue -> /usr/local/lib/node_modules/pencilblue-cli/lib/pencilblue-cli.js
/usr/local/bin/pbctrl -> /usr/local/lib/node_modules/pencilblue-cli/lib/pencilblue-cli.js
pencilblue-cli@0.3.1 /usr/local/lib/node_modules/pencilblue-cli
├── process@0.11.8
├── colors@1.1.2
├── q@1.4.1
├── shelljs@0.7.3 (interpret@1.0.1, rechoir@0.6.2, glob@7.0.5)
└── prompt@1.0.0 (revalidator@0.1.8, pkginfo@0.4.0, read@1.0.7, winston@2.1.1, utile@0.3.0)
The CLI can then be used to install and configure PencilBlue itself:
$ pbctrl install PencilBlue
Site Name: (My PencilBlue Site) PokeSite
Site Root: (http://localhost:8080)
Address to bind to: (0.0.0.0)
Site Port: (8080)
MongoDB URL: (mongodb://127.0.0.1:27017/) mongodb://pencilblue_user:my_password@cluster0-shard-00-00-qfovx.mongodb.net:27017,cluster0-shard-00-01-qfovx.mongodb.net:27017,cluster0-shard-00-02-qfovx.mongodb.net:27017/?ssl=true&authSource=admin
Database Name: (pencilblue) clusterdb
Do you want to install Bower components?: (y/N)
Cloning PencilBlue from github...
Cloning into 'PencilBlue'...
Installing npm modules...
...
Creating config.js...
Installation completed.
Note that if you need to change the configuration (e.g., to specify a new URL to connect to MongoDB) then edit the config.js file that was created during this step.
The MongoDB URL you provided contains these components:
pencilblue_user is the name of the user you created in the MongoDB Atlas UImy_password is the password you chose when creating the user in MongoDB Atlascluster0-shard-00-00-qfovx.mongodb.net, cluster0-shard-00-01-qfovx.mongodb.net, & cluster0-shard-00-02-qfovx.mongodb.net are the hostnames of the instances in your MongoDB Atlas replica set (click on the “CONNECT” button in the MongoDB Atlas UI if you don’t have these – Figure 3)27017 is the standard MongoDB port numberssl option is usedadmin is the database that’s being used to store the credentials for pencilblue_user
Figure 3: Find the Hostnames From the MongoDB Atlas UI
clusterdb is the name of the database (schema) that PencilBlue will use (note that unlike some frameworks, the database name is specified separately rather than being embedded in the MongoDB URL).
The PencilBlue process can now be started:
$ cd PencilBlue/
$ pbctrl start
At this point, it is possible to connect to MongoDB Atlas using the MongoDB shell (we’ll look at an easier way to navigate the data later) to confirm that the schema has been created:
$ mongo mongodb://cluster0-shard-00-00-qfovx.mongodb.net:27017,cluster0-shard-00-01-qfovx.mongodb.net:27017,cluster0-shard-00-02-qfovx.mongodb.net:27017/admin?replicaSet=Cluster0-shard-0 --ssl --username billy --password my_password
Cluster0-shard-0:PRIMARY> show dbs
admin 0.000GB
clusterdb 0.008GB
local 0.007GB
Cluster0-shard-0:PRIMARY> use clusterdb
switched to db clusterdb
Cluster0-shard-0:PRIMARY> show collections
article
auth_token
comment
custom_object
custom_object_type
fs.chunks
fs.files
job_log
lock
media
page
password_reset
plugin
plugin_settings
section
server_registry
session
setting
theme_settings
topic
unverified_user
user
Browse to the application at http://localhost:8080 as shown in Figure 4 and create a user account.
Figure 4: Register User in PencilBlue
You’re then able to login and create your first page (Figure 5).
Figure 5: Create a New Page Using PencilBlue
After saving, the new page can be viewed (Figure 6).
Figure 6: View Pokémon Page in PencilBlue
To visually navigate through the PencilBlue schema and data, download and install MongoDB Compass. Use your MongoDB Atlas credentials to connect Compass to your MongoDB database – Figure 7.
Figure 7: Connect MongoDB Compass to MongoDB Atlas
Navigate through the structure of the data in the clusterdb database (Figure 8); view the JSON documents (Figure 9) and check the indexes (Figure 10).
Figure 8: Explore PencilBlue Schema Using MongoDB Compass
Figure 9: View PencilBlue Documents in MongoDB Compass
Figure 10: View PencilBlue Indexes Using MongoDB Compass
While MongoDB Atlas radically simplifies the operation of MongoDB there are still some decisions to take to ensure the best performance and reliability for your application. The MongoDB Atlas Best Practices white paper provides guidance on best practices for deploying, managing, and optimizing the performance of your database with MongoDB Atlas.
The guide outlines considerations for achieving performance at scale with MongoDB Atlas across a number of key dimensions, including instance size selection, application patterns, schema design and indexing, and disk I/O. While this guide is broad in scope, it is not exhaustive. Following the recommendations in the guide will provide a solid foundation for ensuring optimal application performance.
This blog post demonstrates how to build and deploy an application on AWS Elastic Beanstalk, and have that application connect to MongoDB Atlas as its back-end database service:
AWS Elastic Beanstalk is a service offered by Amazon to make it simple for developers to deploy and manage their cloud-based applications. After you’ve uploaded your application, Elastic Beanstalk automatically takes care of:
MongoDB Atlas provides all of the features of the MongoDB database, without the operational heavy lifting. MongoDB Atlas is available on demand through a pay-as-you-go model and billed on an hourly basis, letting you focus on your application code.
It’s easy to get started – use a simple GUI to select the instance size, region, and features you need. MongoDB Atlas provides:
There is clearly a lot of synergy between these technologies – both of them handling the enabling infrastructure, letting the developer spend their precious time on writing great applications. To continue in the spirit of developer productivity, the application used in this post is developed using Node.js, the Express web application framework, and the Pug (formerly Jade) template engine.
Let’s start by taking a look at what the new Mongopop application provides.
Getting your MongoDB Atlas cluster up and running is a breeze but what do you do with it next? Wouldn’t it be great to populate it with some realistic data so that you can start experimenting? This is what MongoPop does – even letting you tailor the format and contents of the data using the Mockaroo service.
Mockaroo is a flexible service, allowing you to define a rich schema and then generate realistic sample data sets. Supported types include:
Data files can be downloaded from Mockaroo in multiple formats, including: JSON, CSV, and SQL.
MongoPop pulls data from Mockaroo and then automatically writes the data to your database. It defaults to our example Mockaroo schema but you can replace that with a URL for any schema that you’ve defined in Mockaroo (or any other service providing arrays of JSON documents). Mockaroo takes care of connecting to MongoDB Atlas and runs multithreaded, speeding up the process of loading large datasets into MongoDB.
Figure 1: Identify IP Address of MongoPop Server for MongoDB Atlas IP Whitelisting
When you first access MongoPop (Figure 1), you’re presented with a form to provide details on how to connect to your MongoDB Atlas instance, and what you’d like the data to look like. Before completing the form, take a note of the IP address that’s displayed. This IP address needs to be added to the whitelist for your group, which is done through the security tab of the MongoDB Atlas UI (Figure 2).
Figure 2: Add MongoPop IP Address to MongoDB Atlas Group Whitelist
In a production Elastic Beanstalk environment, the IP whitelisting can be a little more involved – that will be covered later in this post.
Figure 3: Find the Node.js Driver Connect String in MongoDB Atlas
While in the MongoDB Atlas UI, click the “CONNECT” button, select the “MongoDB Drivers” tab and then the “COPY” button (Figure 3). Paste the copied URI directly into MongoPop. You should also enter the password and the database you want to use.
Note that the URI needs editing before it’s actually used but MongoPop handles that using the password and database name you provide; the final URI will take this form: mongodb://mongodb_user:my_password@cluster0-shard-00-00-qfovx.mongodb.net:27017,cluster0-shard-00-01-qfovx.mongodb.net:27017,cluster0-shard-00-02-qfovx.mongodb.net:27017/mongopop?ssl=true&authSource=admin.
This URI contains these components:
mongodb_user is the name of the user you gave when creating the group in the MongoDB Atlas UI. Alternatively, create a new user in the MongoDB Atlas UI with more restricted privileges.my_password is the password you chose when creating the user in MongoDB Atlas.cluster0-shard-00-00-qfovx.mongodb.net, cluster0-shard-00-01-qfovx.mongodb.net, & cluster0-shard-00-02-qfovx.mongodb.net are the hostnames of the instances in your MongoDB Atlas replica set.27017 is the default MongoDB port number.mongopop is the name of the database (schema) that MongoPop will use.ssl option is used.admin is the database that’s being used to store the credentials for mongodb_user.The remaining fields define the collection to store the documents, the source of the document schema, and the number of documents (in thousands) to be added. The source URL defaults to a document format already defined but you can create your own by registering at the Mockaroo site, defining the document structure and then using the URL provided.
After clicking the “populate” button, MongoPop fetches the data set(s) from Mockaroo and then adds the documents to your MongoDB Atlas collection. Once the data has been added, the page refreshes and you’re shown a sample of the documents now stored in your collection (Figure 4).
Figure 4: Sample of Data Added to MongoDB Atlas Collection
Congratulations, you now have some data in your database! An optional step is to start exploring that data using MongoDB Compass. The same credentials can be used to connect Compass to your MongoDB database (Figure 5).
Figure 5: Connect MongoDB Compass to MongoDB Atlas
Once connected, explore the data added to the collection (Figure 6).
Figure 6: Explore MongoDB Atlas Data Using MongoDB Compass
In this version (1.3) of MongoDB Compass (currently in beta), it is also possible to add, delete, and modify documents (Figure 7).
Figure 7: Modify a Document in MongoDB Compass
You can verify that the document has really been updated from the MongoDB shell:
The tools for deploying your application to AWS Elastic Beanstalk integrate with git, which makes it the best way to get the code. Assuming that git is already installed, downloading the code is simple:
If you then want to refresh your local repository with the latest version:
Alternatively, simply download the zip file.
Deploying to Elastic Beanstalk is straightforward but there is a delay each time you update and redeploy your application. For that reason, it’s still useful to be able to test and debug locally.
After downloading the application, installing its dependencies and then running it is trivial (this assumes that you already have Node.js installed):
npm_install installs all of the required dependencies (which are described in package.json). npm start starts the application – once it it running browse to http://localhost:3000/pop to try it out.
You can create your Elastic Beanstalk environment and deploy and monitor your application from the AWS console. If you don’t already have an account then that’s where you would create it. If you already have an account, and a copy of your Access Key ID and Secret Access Key, then using the EB CLI provides a more efficient workflow.
The method for installing the EB CLI varies by platform but if you already have Homebrew installed on OS X then it’s as simple as:
eb init sets default values for Elastic Beanstalk applications created with the EB CLI by prompting you with a series of questions:
eb create creates a new environment and deploys the current application to that environment:
Finally, eb open connects to the MongoPop app from your browser.
If you want to make changes to the application then the EB CLI makes it simple to redeploy the new version. As an example, edit the views/pop.jade file to add an extra paragraph after the title:
The EB CLI integrates with git and so update git with the change and then redeploy:
Figure 8: Personalized Version of MongoPop Deployed to AWS EBS
When you’re finished with the application, the environment can be deleted with a single command:
Note that this doesn’t remove the application deployment files that Elastic Beanstalk keeps in AWS S3 storage. To avoid continuing charges, delete those files through the AWS console (Figure 9).
Figure 9: Remove Deployment Files From AWS S3 Storage
The full code for MongoPop can be found in GitHub but this section presents some snippets that are specific to MongoDB and MongoDB Atlas.
Firstly, constructing the final URI to access the database (from views/pop.js):
Connecting to the database and working with the collection (javascripts/db.js):
All of the dependencies (including the MongoDB Node.js driver) are defined in package.json:
IP address whitelisting is a key MongoDB Atlas security feature, adding an extra layer to prevent 3rd parties from accessing your data. Clients are prevented from accessing the database unless their IP address has been added to the IP whitelist for your MongoDB Atlas group.
VPC Peering for MongoDB Atlas is under development and will be available soon, offering a simple, robust solution. It will allow the whitelisting of an entire AWS Security Group within the VPC containing your application servers.
If you need to deploy a robust, scalable application before VPC peering becomes available, some extra steps may be required.
In our example application, the public IP address of the AWS EC2 instance running MongoPop was added to the MongoDB Atlas whitelist for the group.
That works fine but what happens if that EC2 instance fails and is rescheduled – its IP Address changes and so it would not be able to connect to MongoDB Atlas until it was whitelisted. That scenario can be remedied by assigning an Elastic IP address (which survives rescheduling) to the EC2 instance using the AWS Console.
What if demand for your application grows and Elastic Beanstalk determines that it needs to add an extra EC2 instance? Again, that instance will have an IP Address that hasn’t yet been added to the MongoDB Atlas whitelist. To cover that scenario (as well as rescheduling), the AWS NAT Gateway service can be used. Figure 10 illustrates a configuration using a NAT Gateway.
Figure 10: Presenting a Single IP Address Using an AWS NAT Gateway
Two subnets are created within the AWS Virtual Private Cloud (VPC):
Routing tables must be created to route all messages from the private subnet destined for public IP addresses through the NAT Gateway. The NAT Gateway has its own Elastic IP Address which all of the outgoing messages that pass through it appear to originate from – this IP Address must be added to the MongoDB Atlas whitelist.
Messages between the front-end and back-end servers use local IP Addresses and so are routed directly, without passing through the NAT Gateway. Messages from external clients are routed from the IGW to the front-end servers.
Clearly this configuration adds cost and complexity (e.g., the application needs breaking into front and back-end components).
An alternative is to add extra logic to your application so that it automatically adds its IP address to the whitelist using the MongoDB Atlas Public API. If taking that approach, then also consider how to remove redundant IP addresses as the whitelist is limited to 20 entries.
While MongoDB Atlas radically simplifies the operation of MongoDB there are still some decisions to take to ensure the best performance and reliability for your application. The MongoDB Atlas Best Practices white paper provides guidance on best practices for deploying, managing, and optimizing the performance of your database with MongoDB Atlas.
The guide outlines considerations for achieving performance at scale with MongoDB Atlas across a number of key dimensions, including instance size selection, application patterns, schema design and indexing, and disk I/O. While this guide is broad in scope, it is not exhaustive. Following the recommendations in the guide will provide a solid foundation for ensuring optimal application performance.
Learn more about the capabilities of MongoDB Atlas and try it out for yourself here.
Apostrophe is a Content Management Systems that’s designed to build content-driven web sites. Because of their ease of use, Apostrophe is built upon MongoDB and Node.js.
This post explains why MongoDB Atlas is an ideal choice for Apostrophe and then goes on to show how to configure Apostrophe to use it.
MongoDB delivers flexible schemas, rich queries, an idiomatic Node.js driver, and simple to use high availability and scaling. This makes it the go-to database for anyone looking to build applications on Node.js.
MongoDB Atlas provides all of the features of MongoDB, without the operational heavy lifting required for any new application. MongoDB Atlas is available on demand through a pay-as-you-go model and billed on an hourly basis, letting you focus on what you do best.
It’s easy to get started – use a simple GUI to select the instance size, region, and features you need. MongoDB Atlas provides:
Like Apostrophe, MongoDB Atlas is a natural fit for users looking to simplify their development and operations work, letting them focus on what makes their application unique rather than commodity (albeit essential) plumbing.
Before starting with Apostrophe, you should launch your MongoDB cluster using MongoDB Atlas and then (optionally) create a user with read and write privileges for just the database that will be used for this project, as shown in Figure 1. You must also add the IP address of your application server to the IP Whitelist in the MongoDB Atlas security tab.
Figure 1:Creating an Apostrophe user in MongoDB Atlas
If it isn’t already installed on your system, download and install Node.js:
You should then add the bin sub-folder to your .bash_profile file and then install ImageMagick (used by Apostrophe to handle image files); clone the Apostrophe Sandbox project; and then install its dependencies:
Before starting Apostrophe you need to configure it with details on how to connect to your specific MongoDB Atlas cluster. This is done by cloning the configuration file to data/local.js:
You should then edit the data/local.js file and set the uri parameter using the specific connection information provided for your MongoDB Atlas group:
The URI contains these components:
apostrophe_user is the name of the user you created in the MongoDB Atlas UImy_password is the password you chose when creating the user in MongoDB Atlascluster0-shard-00-00-qfovx.mongodb.net, cluster0-shard-00-01-qfovx.mongodb.net, & cluster0-shard-00-02-qfovx.mongodb.net are the hostnames of the instances in your MongoDB Atlas replica set (click on the “CONNECT” button in the MongoDB Atlas UI if you don’t have these)27017 is the standard MongoDB port numberclusterdb is the name of the database (schema) that Apostrophe will use (note that this must match the project name used when installing Apostrophe as well as the database you granted the user access tossl option is usedadmin is the database that’s being used to store the credentials for apostrophe_user
Clients connect to Apostrophe through port 3000 and so you must open that port in your firewall.
You can then create the database and start Apostrophe:
Browse to the application at http://address-of-app-server:3000 as shown in Figure 2 and then login using the username admin and the password demo.
Figure 2: Apostrophe Running on MongoDB Atlas
Now, go ahead and add some content (Figure 3).
Figure 3: Edit Apostrophe Home Page with Data Stored in MongoDB Atlas
Upload some images as shown in Figure 4.
Figure 4: Upload Images to Apostrophe on MongoDB Atlas
Optionally, to confirm that, MongoDB Atlas really is being used by Apostrophe, you can connect using the MongoDB shell:
To visually navigate through the schema and data created by Apostrophe, download and install MongoDB Compass. Use your MongoDB Atlas credentials to connect Compass to your MongoDB database – Figure 5.
Figure 5: Connect MongoDB Compass to MongoDB Atlas
Navigate through the structure of the data in the clusterdb database (Figure 6) and view the JSON documents (Figure 7).
Figure 6: Explore Apostrophe Schema Using MongoDB Compass
Figure 7: View Apostrophe Documents in MongoDB Compass
While MongoDB Atlas radically simplifies the operation of MongoDB there are still some decisions to take to ensure the best performance and reliability for your application. The MongoDB Atlas Best Practices white paper provides guidance on best practices for deploying, managing, and optimizing the performance of your database with MongoDB Atlas.
The guide outlines considerations for achieving performance at scale with MongoDB Atlas across a number of key dimensions, including instance size selection, application patterns, schema design and indexing, and disk I/O. While this guide is broad in scope, it is not exhaustive. Following the recommendations in the guide will provide a solid foundation for ensuring optimal application performance.
Want to try out MongoDB on your laptop? Execute a single command and you have a lightweight, self-contained sandbox; another command removes all traces when you’re done.
Need an identical copy of your application stack in multiple environments? Build your own container image and let your development, test, operations, and support teams launch an identical clone of your environment.
Containers are revolutionizing the entire software lifecycle: from the earliest technical experiments and proofs of concept through development, test, deployment, and support.
Orchestration tools manage how multiple containers are created, upgraded and made highly available. Orchestration also controls how containers are connected to build sophisticated applications from multiple, microservice containers.
The rich functionality, simple tools, and powerful APIs make container and orchestration functionality a favorite for DevOps teams who integrate them into Continuous Integration (CI) and Continuous Delivery (CD) workflows.
This post delves into the extra challenges you face when attempting to run and orchestrate MongoDB in containers and illustrates how these challenges can be overcome.
Running MongoDB with containers and orchestration introduces some additional considerations:
hostname for the service that remains constant through rescheduling.rs.initiate and rs.add commands.As described in the previous section, distributed databases such as MongoDB require a little extra attention when being deployed with orchestration frameworks such as Kubernetes. This section goes to the next level of detail, showing how this can actually be implemented.
This section starts by creating the entire MongoDB replica set in a single Kubernetes cluster (which would normally be within a single data center – that clearly doesn’t provide geographic redundancy. In reality, little has to be changed to run across multiple clusters and those steps are described later.
Each member of the replica set will be run as its own pod with a service exposing an external IP address and port. This ‘fixed’ IP address is important as both external applications and other replica set members can rely on it remaining constant in the event that a pod is rescheduled.
The following diagram illustrates one of these pods and the associated Replication Controller and service.
Figure 1: MongoDB Replica Set member as a Kubernetes Pod
Stepping through the resources described in that configuration we have:
mongo-node1. mongo-node1 includes an image called mongo which is a publicly available MongoDB container image hosted on Docker Hub. The container exposes port 27107 within the cluster./data/db directory within the connector to the persistent storage element named mongo-persistent-storage1; which in turn is mapped to a disk named mongodb-disk1 created in the Google Cloud. This is where MongoDB would store its data so that it is persisted over container rescheduling.mongo-node and provide an (arbitrary) instance name of rod.mongo-rc1 is configured to ensure that a single instance of the mongo-node1 pod is always running.LoadBalancer service named mongo-svc-a exposes an IP Address to the outside world together with the port of 27017 which is mapped to the same port number in the container. The service identifies the correct pod using a selector that matches the pod’s labels. That external IP Address and port will be used by both an application and for communication between the replica set members. There are also local IP addresses for each container, but those change when containers are moved or restarted, and so aren’t of use for the replica set.The next diagram shows the configuration for a second member of the replica set.
Figure 2: Second MongoDB Replica Set member configured as a Kubernetes Pod
90% of the configuration is the same, with just these changes:
mongodb-disk2 and mongo-persistent-storage2 are usedinstance: jane and name: mongo-node2 so that the new service can distinguish it (using a selector) from the rod Pod used in Figure 1.mongo-rc2
mongo-svc-b and gets a unique, external IP Address (in this instance, Kubernetes has assigned 104.1.4.5)The configuration of the third replica set member follows the same pattern and the following figure shows the complete replica set:
Figure3: Full Replica Set member configured as a Kubernetes Service
Note that even if running the configuration shown in Figure 3 on a Kubernetes cluster of three or more nodes, Kubernetes may (and often will) schedule two or more MongoDB replica set members on the same host. This is because Kubernetes views the three pods as belonging to three independent services.
To increase redundancy (within the zone), an additional headless service can be created. The new service provides no capabilities to the outside world (and will not even have an IP address) but it serves to inform Kubernetes that the three MongoDB pods form a service and so Kubernetes will attempt to schedule them on different nodes.
Figure 4: Headless service to avoid co-locating of MongoDB replica set members
The actual configuration files and the commands needed to orchestrate and start the MongoDB replica set can be found in the Enabling Microservices: Containers & Orchestration Explained white paper. In particular, there are some special steps required to combine the three MongoDB instances into a functioning, robust replica set which are described in the paper.
There is risk associated with the replica set created above in that everything is running in the same GCE cluster, and hence in the same availability zone. If there were a major incident that took the availability zone offline, then the MongoDB replica set would be unavailable. If geographic redundancy is required, then the three pods should be run in three different availability zones or regions.
Surprisingly little needs to change in order to create a similar replica set that is split between three zones – which requires three clusters. Each cluster requires its own Kubernetes YAML file that defines just the pod, Replication Controller and service for one member of the replica set. It is then a simple matter to create a cluster, persistent storage, and MongoDB node for each zone.
Figure 5: Replica set running over multiple availability zones/regions
To learn more about containers and orchestration – both the technologies involved and the business benefits they deliver – read the Enabling Microservices: Containers & Orchestration Explained white paper. The same paper provides the complete instructions to get the replica set described in this post up and running on Docker and Kubernetes in the Google Container Engine.
Watch this webinar recording to learn more on this topic and see a live demo putting it all together.
