Topic 13: Introduction to node.js

node.js (see www.nodejs.org) is an increasingly popular platform which allows you to develop server-side applications in JavaScript. This has the a number of advantages, for example:

You can use the same language to develop both client- and server-side components to your web application, minimising the number of languages you have to learn;
A similar style of programming is used (node.js extensively uses asynchronous development with callbacks);
JavaScript language features and APIs, such as JSON.parse() and bind(), promises (to be covered later), arrow functions, etc. will work with node.js as they do with in-browser JavaScript;
Because node.js is JavaScript it is easy to exchange JSON with the client.

Installing and using node.js

node.js can be downloaded from the website. It is already installed on Edward2 and Neptune. Rather than using an Apache server you write your own custom, lightweight HTTP server with node.js running on a different port to Apache (port 3000 is used in here).

npm

Node also comes with a package manager called npm. This allows you to use add-ons to Node (libraries) which perform additional tasks, not part of the core node.js distribution, such as communicating with a database. To use npm to install new packages, you enter:

npm install <packagename>

at the command-line (e.g. DOS or Linux shell prompt).

Logging onto the Neptune command-line and using the text editor

Because you need to run a node script, you need to log on to Neptune and use the Neptune command-line (Linux shell prompt) as well as using FTP. To do this you should use putty (available on the lab computers, use the search). Find putty and then login with the 'node' account. Enter the following code in Notepad++, save as hello.js and upload to your home directory (the one before public_html) using FileZilla. Or, if you prefer, create the file directly on the Neptune server using nano or a similar text editor.

console.log("Hello world!");

On PuTTY run your Node.js script on the command line by entering:

node hello.js

Hopefully this script is self-explanatory. console.log() simply writes the specified text to the console, so that here, for example, "Hello World!" is displayed.

HTTP Servers

Probably the most useful thing to learn with node.js is how to create a lightweight HTTP server. With node.js, typically you do not use Apache but instead write your own server using the Node.js HTTP APIs. Here is a very basic HTTP server:

const http = require("http");

const server = http.createServer(
    (request, response) =>
    {
        response.write(`Requested with a method of ${request.method}`);
        response.end();
    }
);

server.listen(3000);

Looking at how this works:

First we import the http module. Node comes with modules which are libraries: they provide additional capabilities to the Node environment. Many are provided by third parties. The npm command line tool allows you to install new modules. The http module, however, is provided with Node by default.
Next we create an HTTP server with the createServer() method. This returns a server object. Note how we specify a callback function (arrow function) as a parameter to the createServer() method. This is an example of an anonymous function, which you have seen already.
The callback runs each time we get a request from a client. It takes two parameters, request and response, two objects representing the HTTP request and response respectively. Look at what it's doing: we use the write() method of the request to send some content back to the client, and then we must finish the response by calling response.end()
This simple example just displays what HTTP method (GET or POST) was used to request the server. The request.method property contains the HTTP method.

Exercise 1

Try running this. Rather than 3000, use your own personal port number. This should be a number from 3001-3099 and have the same last two digits as your username, e.g. if you are wad1901, your port number would be 3001, while if you are wad1909 it would be 3009. This is necessary as you are all creating your own personal node servers on Neptune and therefore must each use different ports. You'll find it goes into an infinite loop and does not return you to the command line: this is because it is running the server continuously listening for requests.

Now try sending an HTTP request to the server from your web browser. Enter:

http://ephp.solent.ac.uk:NNNN/

from the browser, where NNNN is your own personal port number. You should see this output appear:

Requested with a method of: GET

because you have sent a GET request to the server.

API documentation

See here for full Node API documentation on the HTTP module.

Using Express

The above example showed how to create a webserver from first principles in Node, using the HTTP module. However, there is a very well-known Node framework which makes the process of developing a web server, particularly one offering REST-compliant web services, much easier - Express. As is typical, Express is offered as a Node module and should be installed with npm. This has been installed on Neptune and Edward2 already. To give an idea of how to install it:

npm install express

This will install it locally to your current project. Or:

sudo npm install -g express

This requires 'sudo' rights (admin privileges) on a Linux/Unix system but will install it globally, to make it available to all projects and all users.

Hello World with Express

The example below is the Hello World with Express. Note that const is an ECMAScript 6 keyword for declaring a constant, in other words a "variable" that cannot change.

const express = require('express');
const app = express();

app.get('/', (req,res)=> {
    res.send('Hello World from Express!');
});

app.listen(3000);

Hopefully you can see it is similar in structure to the non-Express web server, above. We require the express module, then create an Express app object with express().

Next we handle a routes. Routes in Express work in a very similar way to those in Slim. In this example we are simply handling the top level, 'root' route, e.g.:

http://localhost:3000/

without any parameters supplied.

The handler for the route is a function which takes two parameters, req representing the HTTP request and res representing the HTTP response. In this example we call the send() method of the response object to send back Hello World from Express! to the client.

Example

const express = require('express');
const app = express();


app.get('/hello', (req,res)=> {
    res.send('Hello World!');
});


app.get('/time', (req,res)=> {
    res.send(`There have been ${Date.now()} milliseconds since 1/1/70.`);
});
app.listen(3000);

This example sets up two routes '/hello' and '/time'. The first will send back 'Hello World!' and the second will send back the number of milliseconds since January 1st 1970. You could set up POST and PUT API endpoints (an API endpoint is a URL associated with a web service or web API) with app.post() and app.put().

Exercise 2

Try out the example above. Run the server and request each route by typing in the appropriate URL in your web browser.

URL parameters

Express allows us to easily specify URL parameters, though these are typically passed into the URL as part of the route (REST style) rather than via query strings e.g:

http://localhost:3000/artist/Oasis
http://localhost:3000/artist/Madonna

etc. You set up the route to contain named parameters and then you can use these named parameters in your JavaScript. This example counts up to a named parameter:

const express = require('express');
const app = express();


app.get('/countTo/:ntimes', (req,res)=> {
    let response = ''; // let - locally-scoped variable which can be changed
    for(var count=1; count<=req.params.ntimes; count++) {
        response += `${count}<br />`;
    }
    res.send(response);
});


app.listen(3000);

Note how this will setup the route /countTo followed by a parameter :ntimes representing the number of times. So API endpoints such as

http://localhost:3000/countTo/3
http://localhost:3000/countTo/6
http://localhost:3000/countTo/goat

will be matched. (There is no check that ntimes is numeric, so the final URL will be matched. However you can use regular expressions in Express routes to ensure that only numbers are passed in the parameter).

To access the named parameter from the JavaScript code, use req.params.<name of parameter>. So here req.params.ntimes will reference the :ntimes parameter and the output will count from 1 to the specified parameter.

Exercise 3

Create an Express application to set up an artist route. The word artist should be followed by a parameter for the artist name and the application should respond to the request with a message such as
```
You are searching for songs by Oasis
```
(obviously, it shouldn't always show Oasis, but whatever artist was entered in the URL).
Make the application also respond to URLs of this form:
```
http://localhost:3XXX/artist/Oasis/song/Wonderwall
```
which would display:
```
You are searching for Wonderwall by Oasis
```
(obviously, it should respond to any title and artist, not just Wonderwall by Oasis!

Handling POST data

How do we handle POST data in Express? We can do it manually, by examining the request, but it's much easier to use the body-parser module. The body-parser module is a module separate to Express (but designed for it) which can interpret the request body and extract the POST data from it. It needs to be installed with npm separately from Express. Here is an example:

const express = require('express');
const bodyParser = require('body-parser');

const app = express();

app.use(bodyParser.urlencoded({extended:true}));

app.post('/addSong', (req,res) => {
    res.send(`Adding song ${req.body.title} by ${req.body.artist}.`);
});

app.listen(3000);

Note how we tell the app to use the body-parser module with:

app.use(bodyParser.urlencoded({extended:true});

and to refer to the POST data (the equivalent of $_POST in PHP) we use:

req.body.<fieldname>

e.g. req.body.songID here to read in an item of POST data with the name of songID

Exercise 4

Try out the above example. You'll need to create a regular HTML form with an appropriate action and appropriate form fields.

Middleware

Express (and Slim) uses the concept of middleware. Middleware in this context consists of code which runs before a request is handled, which can modify the req and res objects. It is typically used to prepare the request before it's handled. The body-parser is actually middleware. It parses the body of the incoming HTTP request and adds a body property to the req object allowing you to easily access POST data. Without the middleware, the body property of req would not exist.

Middleware is included with the use() method of your app object. If you look at the POST example above you can see how use() has been used to run the body-parser as middleware. Note the urlencoded() method; this means the body-parser is being setup to handle classic POST form data. An increasingly popular alternative approach is to pass POST data as a JSON object instead (rather than named fields); this is the approach the fetch() API for sending AJAX requests (which we will look at soon) takes, for instance. To use body-parser to parse JSON, you would use json() rather than urlencoded().

One possible use of middleware is to add a CORS header to all responses. In an AJAX application you will typically have a front end delivered from a standard server (such as Apache or Nginx) on port 80, yet your Node/Express backend will be listening on port 3000, or a similar port. This violates the same-origin policy and therefore you must add an appropriate CORS header to your response. See the exercise below.

Middleware and routes

We can use routes to specify that a given middleware will only run before handling a particular route. This can be useful as, we might want certain middleware to only run in response to certain requests.

Here is an example of a middleware that will run on any request.

const express = require('express');
const app = express();

app.use( (req,res,next) => {
    console.log(`Received a request at ${Date.now()} milliseconds.`);
    next();
});

app.get('/', (req,res) => {
    res.send(`Hello world!`);
});

app.listen(3000);

Note how use() expects a function which takes request and response as parameters, the same as the route handlers. This example writes a message to the console informing the user of the time of the request.

Notice the call to the next(); function (also passed as a parameter to use()). This is necessary to execute the next middleware (we can set up multiple items of middleware with multiple use() methods) and the route handler.

We can also supply an optional first argument to use() to specify which routes will run this middleware. The middleware below (showing the time of the request once again) will only run with a route of search followed by a search term:

const express = require('express');
const app = express();

app.use( '/search/:query', (req,res,next) => {
    console.log(`Received a request for /search at ${Date.now()} milliseconds.`);
    next();
});

app.get('/', (req,res) => {
    res.send(`Hello world!`);
});

app.get('/search/:query', (req,res) => {
    res.send(`Searching for ${req.params.query}...`);
});

app.listen(3000);

Writing web services

It's easy to send back JSON from an Express app, just replace send() with json(), e.g.:

res.json({'error': 'you did not provide a search query'});

To send back status codes, use status(). These calls can be chained, e.g:

res.status(400).json({'error': 'you did not provide a search query'});

Node and MySQL

In this unit, we will be looking at Node and the NoSQL database MongoDB, as they work particularly well together, and are often used together. However, to allow you to write a useful Express application now (a Node/Express version of your HT-Tracks web service) I'm going to introduce the MySQL Node module. This must be installed as a module as usual (it has been installed on Neptune already). Very good documentation is available here.

To summarise using the MySQL Node module:

Connecting to the database is as follows. Note the callback-based, asynchronous approach; the connect() method takes a callback function as an argument which runs as soon as we are connected. This callback will receive an err parameter if there was an error.

const con = mysql.createConnection({
        host: 'localhost',
        user: 'dftitutorials',
        database: 'dftitutorials',
        password: 'dftitutorials'});

con.connect( err=> {
            if(err) {
                   console.log(`Error connecting to mysql: ${err}`);
            } else { 
                console.log('connected to mysql ok');
                //  now we have a successful connection, query the database..
            }
        });

Performing a SELECT statement is as follows. Note how it's callback-based again: the first argument to query() is the query, the second an array of the bound parameters (using prepared statements), and the third is a callback which runs when the query has returned results. This callback has three parameters: an error (if there is one); the results (an array of objects, one object for each record) and the database fields.
```
con.query(`SELECT * FROM students WHERE lastname=?`,
    [lastname], (error,results,fields) => { 
        if(error) {
            console.log(`Error with mysql statement: details: ${JSON.stringify(error)}`);
        } else {
            results.forEach (result=> {
                console.log(`Name ${result.name} Course ${result.course} Grade: ${result.grade}`); 
            });
        }
    });
```
Note how we loop through the results, add the details of each result to a string variable, and then write that variable to the console.

Performing an UPDATE statement uses the same approach:

con.query(`UPDATE products SET quantity=quantity-1 WHERE id=?`, [productId], (error,results,fields)=> {
    if(results.affectedRows==1) {
        console.log('You bought it successfully.');
    } else {
        console.log('Unable to update, is there a product with that ID?');
    }
} );

Note how we use the affectedRows property of the results object to determine whether any results were updated.

Writing a custom module to easily connect to MySQL

The above examples show a potential issue with using the Node MySQL module: it's callback based, so the code ends up relatively complex. This means that, to integrate MySQL into an Express application, you'd need to connect to MySQL first and then put your Express code in the MySQL connection callback. This makes your code perhaps rather messy and convoluted. A better approach is to create your own Node module and put the connection code in that. Here is an example:

// File - mysqlconn.js
const mysql = require('mysql');

const con = mysql.createConnection({
        host: 'localhost',
        user: 'dftitutorials',
        database: 'dftitutorials',
        password: 'dftitutorials'});

        con.connect( err=> {
            if(err) {
                   console.log(err);
                process.exit(1); // exit the server
            } else { 
                console.log('connected to mysql ok');
            }
        });
    
module.exports = con;

Note how we have created a file mysqlconn.js which connects to the database. Note the final line as this is key:

module.exports=con;

This exports the variable con (representing the connection) to the outside world. This is how we can define a Node module; write some code and then export an object to the outside world.

To use our custom module in our Express application, we use require() in the same way that we imported third-party modules:

const con = require('./mysqlconn');

The variable con will represent our MySQL connection. Note how it's set equal to the return value of require(); this will be whatever was exported from our module. Also note the module name ./mysqlconn and how this corresponds to the module filename mysqlconn.js; the ./ means the current directory.

Using a module to define your routes

In a larger application, you will quickly find that your main Express file (often index.js or app.js) will become very large, handling a large number of routes. What you can do instead is to define a Router. A router allows you to set up a group of routes which match a particular path. For example we might create a router matching /songs which will handle any routes beginning with /songs, e.g: /songs/all, /songs/id/:id and so on. To do this we create a routes directory and place a route handler in there, which would be a Node module. Here is an example router module (song.js) which we would save in our routes directory:

const express = require('express');
// song.js
const song = express.Router();

song.get('/all', (req,res)=> {
    // code to return all songs
});

song.get('/id/:id', (req,res)=> {
    // code to find the song with the given ID
});

module.exports = song; // export the module for external use

Then, in your main Express app, you could use the router as follows:

const express = require('express');
const app = express();

// Import our song router module which we created above
const songRouter = require('./routes/song');

// Tell the app to use songRouter for all routes beginning with /songs
// As we are using "use()", the router is acting as a middleware.
app.use('/songs', songRouter);
app.listen(3000);

So the song router is imported from the routes/song module (corresponding to routes/song.js) and then we call use() (note how the router is acting as a middleware) with two arguments:

/songs, indicating that this router will match all URLs beginning with /songs;
The router object itself.

So, the route /songs/all will be resolved via the song router (as the route begins with /songs) and then finding the /all route within there.

Exercise 5

Write a version of your HT-Tracks search web service using Node.js and Express. The search web service should respond to the route:

/artist/<artist name>

Connect your AJAX search from Topic 6 to your Node/Express search web service. While your Express application will be stored in your folder on the 'node' account, the front-end should be uploaded to your public_html on Neptune (not Edward2!) as normal. You have an Neptune FTP account which has the same login details as your Edward2 account.

The same MySQL database tables are present on Neptune, here are the details:

Username: wad
Password: wad
Database: wad
Table: wadsongs

Output plain text first, and then convert it to output JSON. Note that the 'results' parameter is an array of objects. Use a module to connect to MySQL, as in the example above - it will make your code easier to follow.

If you do that successfully, implement your 'order' web service (Topic 7, DOM) using Express (in the same application as your search service).

Things you'll need to know for this:

The response object res has a status() method which can be used to send back any HTTP status code.
The response object also has a set method which can be used to set a particular HTTP header in the response. You'll need this for CORS as your AJAX front end will be served from a standard Apache server while the back-end is your Node server, on a different port so the same-origin policy applies. You'll need something like: (where should this go?)
```
res.set('Access-Control-Allow-Origin', 'http://ephp.solent.ac.uk');
```
Add your routes to an external router module, as shown above.

Finally - deploying a live Node app

Using a reverse proxy to access your app

What happens when you want to deploy a Node.js application to production? If it's running on port 3000, it will not be accessible to the outside world (for security reasons it's not a good idea to open up non-standard ports), which means you need to provide a 'backdoor' means for the outside world to access it. A common approach is to use a conventional web server (on port 80) as a 'reverse proxy' so that requests which match a particular URL will be routed to your Node application on port 3000. With Apache you can do this using the mod_proxy module. An alternative commonly used web server on Node-based systems is Nginx. DigitalOcean have an article on this: here.

PM2 - Node process manager

The other issue when deploying a Node app to production is stability of your Node server. If you just use the "node" command line to launch your Node app, it will just "die" if it crashes and will not be resurrected. To get round this you can use pm2 (see here; warning, doesn't seem to load on Firefox), which is a process manager typically used with Node applications, though it can also be used to manage other background processes. With pm2, you can launch a Node process in the background, stop it, start it and monitor how it is running. Furthermore if the process crashes, pm2 will restart it.

PM2 has already been installed on Neptune. It can be installed with npm, for example this command will install it globally for all users if you are logged in as a user with admin privileges on a Linux system:

sudo npm install -g pm2

To start a given Node server with PM2 you use this command:

pm2 start name_of_file.js

You can list all processes managed with PM2 with:

pm2 list

Note that each process is allocated a name (the filename minus the .js extension) and an ID (which is useful if two processes with the same name exist). pm2 list also gives information about the CPU and memory usage of each process.

You can stop a running process with:

pm2 stop process_name_or_process_id

This will, however, leave it in the list of managed processes. To completely delete it, you would use:

pm2 delete process_name_or_process_id

Further resources

node.js home page
API documentation
NodeSchool, practical walk-through exercises which provide a detailed introduction to node.js.
Express home page
Node MySQL module