Using Modules in Node.js
Node.js is an event-driven, server-side JavaScript environment. Node.js runs JS using the V8 engine developed by Google for use in their Chrome web browser. Leveraging V8 allows Node.js to provide a server-side runtime environment that compiles and executes JS at lightning speeds.
The Module System
This article covers the Node’s module system and the different categories of the Node.js modules.
Application Modularization
Like most programming languages, Node.js uses modules as a way of organizing code. The module system allows you to organize your code, hide information, and only expose the public interface of a component using module.exports.
Node.js uses the CommonJS specification for its module system:
- Each file is its own module, for instance, in the following example, jsand math.js are both modules
- Each file has access to the current module definition using the modulevariable
- The export of the current module is determined by the module.exportsvariable
- To import a module, use the globally available requirefunction
Take a look at a simple example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
// math.js file function add(a, b) { return a + b; } … … module.exports = { add, mul, div, }; // index.js file const math = require('./math'); console.log(math.add(30, 20)); // 50 |
To call other functions such as mul and div, use object destructuring as an alternative when requiring the module, for example, const { add } = require(‘./math’);. The code files for the section The Module System are placed at Code/Lesson-1/b-module-system.
Module Categories
You can place Node.js modules into three categories:
- Built-in (native) modules: These are modules that come with Node.js itself; you don’t have to install them separately.
- Third-party modules: These are modules that are often installed from a package repository. npm is a commonly used package repository, but you can still host packages on GitHub, your own private server, and so on.
- Local modules: These are modules that you have created within your application, like the example given previously.
Built-In Modules
These are modules that can be used straight away without any further installation. All you need to do is to require them. There are quite a lot of them, but here are a few that you are likely to come across when building web applications:
- assert: Provides a set of assertion tests to be used during unit testing
- buffer: To handle binary data
- child_process: To run a child process
- crypto: To handle OpenSSL cryptographic functions
- dns: To do DNS lookups and name resolution functions
- events: To handle events
- fs: To handle the filesystem
- httpor https: For creating HTTP(s) servers
- stream: To handle streaming data
- util: To access utility functions like deprecate (for marking functions as deprecated), format (for string formatting), inspect (for object debugging), and so on
For example, the following code reads the content of the lesson-1/temp/sample.txt file using the in-built fs module:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
const fs = require('fs'); let file = `${__dirname}/temp/sample.txt`; fs.readFile(file, 'utf8', (err, data) => { if (err) throw err; console.log(data); }); |
npm – Third-Party Module Registry
Node Package Manager (npm) is the package manager for JavaScript and the world’s largest software registry, enabling developers to discover packages of reusable code. To install an npm package, you only need to run the npm install <package-name> command within your project directory.
Here’s a simple example. If you want to use a package (library) like request in your project, you can run the following command on your Terminal, within your project directory:
|
1 |
npm install request |
To use it in your code, you should require it, like any other module:
|
1 2 3 4 5 6 7 8 9 10 11 |
const request = require('request'); request('http://www.example.com', (error, response, body) => { if (error) console.log('error:', error); // Print the error if one occurred else console.log('body:', body); // Print the HTML for the site. }); |
More details about npm can be found at https://docs.npmjs.com/. When you run the npm install <module-name> command on your project for the first time, the node_modules folder gets created at the root of your project.
Scanning for node_modules
It’s worth noting how Node.js goes about resolving a particular required module. For example, if a /home/tony/projects/foo.js file has a require call require(‘bar’), Node.js scans the filesystem for node_modules in the following order. The first bar.js that is found is returned as follows:
- /home/tony/projects/node_modules/bar.js
- /home/tony/node_modules/bar.js
- /home/node_module/bar.js
- /node_modules/bar.js
Node.js looks for node_moduels/bar in the current folder followed by every parent folder until it reaches the root of the filesystem tree for the current file. Note that the module foo/index.js can be required as foo, without specifying an index and will be picked by default.
Handy npm Commands
Now dive a little deeper into npm, by looking at some of the handy npm commands that you will often need:
- npm init: Initializes a Node.js project. This should be run at the root of your project and will create a respective jsonfile. This file usually has the following parts (keys):
- name: Name of the project.
- version: Version of the project.
- description: Project description.
- main: The entry-point to your project, the main file.
- scripts: This will be a list of other keys whose values will be the scripts to be run, for example, test, dev-server. Therefore, to run this script, you will only need to type commands such as npm run dev-server, npm run test, and so on.
- dependencies: List of third-party packages and their versions used by the project. Whenever you do npm install <package-name> –save, this list is automatically updated.
- devDependencies: List of third-party packages that are not required for production, but only during development. This will usually include packages that help to automate your development workflow, for example, task runners like gulp.js. This list is automatically updated whenever you do npm install <package-name> –save-dev.
- npm install: This will install all the packages, as specified in the jsonfile.
- npm install <package-name> <options>:
- The –saveoption installs the package and saves the details in the json file.
- The –save-devoption installs the package and saves the details in the json, under devDependencies.
- The –globaloption installs the package globally in the whole system, not only in the current system. Due to permissions, this might require running the command with administrator rights, for example, sudo npm install <package-name> –global.
- npm install <package-name>@<version>, installs a specific version of a package. Usually, if a version is not specified, the latest version will be installed.
- npm list: Lists the packages that have been installed for the project, reading from what is installed in node_modules.
- npm uninstall <package-name>: Removes an installed package.
- npm outdated: Lists installed packages that are outdated, that is, newer versions have been released.
Local Modules
You have already looked at how local modules are loaded from the previous example that had math.js and index.js.
Since JavaScript Object Notation (JSON) is such an important part of the web, Node.js has fully embraced it as a data format, even locally. You can load a JSON object from the local filesystem the same way you load a JavaScript module. During the module loading sequence, whenever a file.js is not found, Node.js looks for a file.json.
See the example files in lesson-1/b-module-system/1-basics/load-json.js:
|
1 2 3 |
const config = require('./config/sample'); console.log(config.foo); // bar |
Here, you will notice that once required, the JSON file is transformed into a JavaScript object implicitly. Other languages will have you read the file and perhaps use a different mechanism to convert the content into a data structure such as a map, a dictionary, and so on.
For local files, the extension is optional, but should there be a conflict, it might be necessary to specify the extension. If you have both a sample.js and a sample.json file in the same folder, the .js file will be picked by default; it would be prudent to specify the extension, for example const config = require(‘./config/sample.json’).
When you run npm install, without specifying the module to install, npm will install the list of packages specified (under dependencies and devDependencies in the package.json file in your project). If package.json does not exist, it will give an error indicating that no such file has been found.
Activity: Running Basic Node.js Code
Open the IDE and the Terminal to implement this solution and learn how to write a basic Node.js file and run it. Write a very basic mathematical library with handy mathematical functions using the following steps:
- Create your project directory (folder), where all the code for this will be kept. Inside this directory, create another directory named lesson-1, and inside it, create another directory called activity-a. All this can be done using the following command:
|
1 |
mkdir -p beginning-nodejs/lesson-1/activity-a |
- Inside activity-a, create a file using the touch maths.js
- Inside this file, create the following functions:
- add: This takes any two numbers and returns the sum of both, for example, add(2, 5)returns 7
- sum: Unlike add, sum takes any number of numbers and returns their sum, for example, sum(10, 5, 6)returns 21
- After these functions, write the following code to act as tests for your code:
|
1 2 3 |
console.log(add(10, 6)); // 16 console.log(sum(10, 5, 6)); // 21 |
- Now, on the Terminal, change directory to lesson-1.
- To run the code, run the following command:
|
1 |
node activity-a/math.js |
The 16 and 21 values should be printed out on the Terminal.
Activity: Using a Third-Party Package
This activity will build upon the, Running Basic Node.js activity. If the argument is a single array, sum up the numbers, and if it’s more than one array, first combine the arrays into one before summing up. Use the concat() function from lodash, which is a third-party package that you need to install.
Now create a new function, sumArray, which can sum up numbers from one or more arrays using the following steps:
- Inside Lesson-1, create another folder called activity-b.
- On the Terminal, change directory to activity-band run the following command:
|
1 |
npm init |
- This will take you to an interactive prompt; just press Enter all the way, leaving the answers as suggested defaults. The aim here is to get a json file, which will help organize your installed packages.
- Since you’ll be using lodash, install it. Run the following command:
|
1 |
npm install lodash--save |
Notice that you’re adding the –save option on your command so that the package installed can be tracked in package.json. When you open the package.json file created in step 3, you will see an added dependencies key with the details.
- Create a jsfile in the activity-b directory and copy the math.js code from Activity, Running Basic Node.js into this file.
- Now, add the sumArrayfunction right after the sum
- Start with requiring lodash, which you installed in step 4 since you’re going to use it in the sumArrayfunction:
|
1 |
const _ = require('lodash'); |
- The sumArrayfunction should call the sum function to reuse your code. Use the spread operator on the array. See the following code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
function sumArray() { let arr = arguments[0]; if (arguments.length > 1) { arr = _.concat(...arguments); } // reusing the sum function // using the spread operator (...) since // sum takes an argument of numbers return sum(...arr); } |
- At the end of the file, export the three functions, add, sum, and sumArraywith exports.
- In the same activity-bfolder, create a file, js.
- In jsfile, require ./math.js and go ahead to use sumArray:
|
1 2 3 4 5 |
// testing console.log(math.sumArray([10, 5, 6])); // 21 console.log(math.sumArray([10, 5], [5, 6], [1, 3])) // 30 |
- Run the following code on the Terminal:
|
1 |
node index.js |
You should see 21 and 30 printed out.
If you found this article interesting, you can explore Anthony Nandaa’s Beginning API Development with Node.js to learn everything you need to get up and running with cutting-edge API development using JavaScript and Node.js. Beginning API Development with Node.js begins with the basics of Node.js in the context of backend development and quickly leads you through the creation of an example client that pairs up with a fully authenticated API implementation.
HTTP server applications with Node.js
In this tutorial we’ll learn about HTTP server applications and HTTP sniffing by David Herron, a software engineer in Silicon Valley, who has worked on various enterprise web application projects.
Launching a server with Node.js
Many scripts that you’ll run are server processes. Before you get started, you need to launch a simple HTTP server with Node.js. Borrow the simple server script on the Node.js home page (http://nodejs.org), and create a file named app.js containing the following:
|
1 2 3 4 5 6 7 8 9 10 11 |
const http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello, World!\n'); }).listen(8124, '127.0.0.1'); console.log('Server running at http://127.0.0.1:8124'); |
Run it as follows:
|
1 2 3 |
$ node app.js Server running at http://127.0.0.1:8124 |
This is the simplest of web servers you can build with Node.js. Now, visit http://127.0.0.1:8124 in your browser to see the Hello, World! message:
HTTP server applications
The HTTP server object is the foundation of all Node.js web applications. The object itself is very close to the HTTP protocol, and its use requires knowledge of that protocol. In most cases, you’ll be able to use an application framework such as Express that hides the HTTP protocol details, allowing you to focus on business logic.
The http.createServer function will create an http.Server object because it is an EventEmitter; this can be written in another way to make that fact explicit:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
const http = require('http'); const server = http.createServer(); server.on('request', (req, res) => { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello, World!\n'); }); server.listen(8124, '127.0.0.1'); console.log('Server running at http://127.0.0.1:8124'); |
The request event takes a function, which receives request and response objects. The request object has data from the web browser, while the response object is used to gather the data to be sent in the response. The listen function causes the server to start listening and arranging to dispatch an event for every request arriving from a web browser.
Creating a Node.Js Server
Now, here is something more interesting with different actions based on the URL. Create a new file, named server.js, containing the following code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
const http = require('http'); const util = require('util'); const url = require('url'); const os = require('os'); const server = http.createServer(); server.on('request', (req, res) => { var requrl = url.parse(req.url, true); if (requrl.pathname === '/') { res.writeHead(200, {'Content-Type': 'text/html'}); res.end( `<html><head><title>Hello, world!</title></head> <body><h1>Hello, world!</h1> <p><a href='/osinfo'>OS Info</a></p> </body></html>`); } else if (requrl.pathname === "/osinfo") { res.writeHead(200, {'Content-Type': 'text/html'}); res.end( `<html><head><title>Operating System Info</title></head> <body><h1>Operating System Info</h1> <table> <tr><th>TMP Dir</th><td>${os.tmpdir()}</td></tr> <tr><th>Host Name</th><td>${os.hostname()}</td></tr> <tr><th>OS Type</th><td>${os.type()} ${os.platform()} ${os.arch()} ${os.release()}</td></tr> <tr><th>Uptime</th><td>${os.uptime()} ${util.inspect(os.loadavg())}</td></tr> <tr><th>Memory</th><td>total: ${os.totalmem()} free: ${os.freemem()}</td></tr> <tr><th>CPU's</th><td><pre>${util.inspect(os.cpus())}</pre></td></tr> <tr><th>Network</th><td><pre>${util.inspect(os.networkInterfaces())}</pre></td></tr> </table> </body></html>`); } else { res.writeHead(404, {'Content-Type': 'text/plain'}); res.end("bad URL "+ req.url); } }); server.listen(8124); console.log('listening to http://localhost:8124'); |
To run it, type the following command:
|
1 2 3 |
$ node server.js listening to http://localhost:8124 |
This application is meant to be similar to PHP’s sysinfo function. Node’s os module is consulted to provide information about the server. This example can easily be extended to gather other pieces of data about the server:
A central part of any web application is the method of routing requests to request handlers. The request object has several pieces of data attached to it, two of which are used for routing requests—the request.url and request.method fields.
In server.js, you can consult the request.url data to determine which page to show, after parsing (using url.parse) to ease the digestion process. In this case, you can do a simple comparison of the pathname to determine which handler method to use.
Some web applications care about the HTTP verb (GET, DELETE, POST, and so on) used and must consult the request.method field of the request object. For example, POST is frequently used for FORM submissions.
The pathname portion of the request URL is used to dispatch the request to the correct handler. While this routing method, based on simple string comparison, will work for a small application, it’ll quickly become unwieldy. Larger applications will use pattern matching to use part of the request URL to select the request handler function and other parts to extract request data out of the URL.
A search for a URL match in the npm repository turns up several promising packages that could be used to implement request matching and routing. A framework like Express has this capability already baked in and tested.
If the request URL is not recognized, the server sends back an error page using a 404 result code. The result code informs the browser about the status of the request, where a 200 code means everything is fine, and a 404 code means that the requested page doesn’t exist. There are, of course, many other HTTP response codes, each with their own meaning.
HTTP Sniffer – listening to the HTTP conversation
The events emitted by the HTTPServer object can be used for additional purposes beyond the immediate task of delivering a web application. The following code demonstrates a useful module that listens to all the HTTPServer events. It could be a useful debugging tool, which also demonstrates how HTTPServer objects operate.
Node.js’s HTTPServer object is an EventEmitter and the HTTP Sniffer simply listens to every server event, printing out information pertinent to each event. What you’re about to do is:
- Create a module, httpsniffer that prints information about HTTP requests.
- Add that module to the jsscript you just created.
- Rerun that server to view a trace of HTTP activity.
Create a file named httpsniffer.js containing the following code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
const util = require('util'); const url = require('url'); const timestamp = () => { return new Date().toISOString(); } exports.sniffOn = function(server) { server.on('request', (req, res) => { console.log(`${timestamp()} e_request`); console.log(`${timestamp()} ${reqToString(req)}`); }); server.on('close', errno => { console.log(`${timestamp()} e_close ${errno}`); }); server.on('checkContinue', (req, res) => { console.log(`${timestamp()} e_checkContinue`); console.log(`${timestamp()} ${reqToString(req)}`); res.writeContinue(); }); server.on('upgrade', (req, socket, head) => { console.log(`${timestamp()} e_upgrade`); console.log(`${timestamp()} ${reqToString(req)}`); }); server.on('clientError', () => { console.log(`${timestamp()} e_clientError`); }); }; const reqToString = exports.reqToString = (req) => { var ret=`req ${req.method} ${req.httpVersion} ${req.url}` +'\n'; ret += JSON.stringify(url.parse(req.url, true)) +'\n'; var keys = Object.keys(req.headers); for (var i = 0, l = keys.length; i < l; i++) { var key = keys[i]; ret += `${i} ${key}: ${req.headers[key]}` +'\n'; } if (req.trailers) ret += util.inspect(req.trailers) +'\n'; return ret; }; |
Wow! That was a lot of code! However, the key to it is the sniffOn function. When given an HTTP Server object, it uses the .on function to attach listener functions that print data about each emitted event. It gives a fairly detailed trace of HTTP traffic on an application.
In order to use it, simply insert this code just before the listen function in server.js:
|
1 2 3 4 5 |
require('./httpsniffer').sniffOn(server); server.listen(8124); console.log('listening to http://localhost:8124'); |
With this in place, run the server you launched earlier. You can visit http://localhost:8124/ in your browser and see the following console output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
$ node server.js listening to http://localhost:8124 2017-12-03T19:21:33.162Z request 2017-12-03T19:21:33.162Z request GET 1.1 / {"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/","path":"/","href":"/"} 0 host: localhost:8124 1 upgrade-insecure-requests: 1 2 accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 3 user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5 4 accept-language: en-us 5 accept-encoding: gzip, deflate 6 connection: keep-alive {} 2017-12-03T19:21:42.154Z request 2017-12-03T19:21:42.154Z request GET 1.1 /osinfo {"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"","query":{},"pathname":"/osinfo","path":"/osinfo","href":"/osinfo"} 0 host: localhost:8124 1 connection: keep-alive 2 upgrade-insecure-requests: 1 3 accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 4 user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5 5 referer: http://localhost:8124/ 6 accept-language: en-us 7 accept-encoding: gzip, deflate {} |
You now have a tool for snooping on HTTPServer events. This simple technique prints a detailed log of the event data. The pattern can be used for any EventEmitter object. You can use this technique as a way to inspect the actual behavior of EventEmitter objects in your program.
If you found this article helpful, you can explore David Herron’s Node.js Web Development – Fourth Edition to create real-time applications using Node.js 10, Docker, MySQL, MongoDB, and Socket.IO. With this practical guide, you can go beyond the developer’s laptop to cover live deployment, including HTTPS and hardened security.