Mobile Application Development - Part 10

Network Communication

In this topic we will:

Examine JSON
Look at how we can parse (interpret) JSON in Android
Look at how we can communicate across the web using Fuel and how we can parse JSON returned from a web server

JSON

Introduction to JSON

JSON - JavaScript Object Notation is a data format which is used to transfer data across the web between server-side scripts (written in technologies such as Node.js or JSP) and clients (such as Android apps). It uses JavaScript syntax to represent data.

In JSON, data is transferred as objects and arrays of objects. We use square brackets [] to represent arrays and curly brackets {} to represent objects. For example, if we had a server side script to look up details of famous people, we might get a result such as:

{
    "name": "Tim Berners-Lee",
    "nationality": "British",
    "dob" : "8/6/1955",
    "comments" : "Inventor of the World Wide Web."
}

What is being sent back is a representation of an object with four attributes: name, nationality, dob and comments. Each attribute has a value. The curly brackets { and } begin and end the object.

It is of course possible that we might get more than one record returned from the server. In this case, we get an array of objects back instead. For example, if we did a search for all people called Tim Smith studying at a university, we might get back something like this:

[
    { "username": "2smitt82", "year": "2", "course": "Computing", "phone": "07962 296229" },
    { "username" : "1smitt71", "year": "3", "course": "Chemistry", "phone": "07713 271317" },
    { "username": "3smitt93", "year": "1", "course" : "Web Design", "phone": "07862 141561" }
]

Note how the JSON contains an array of three student objects. The array is indicated with the square brackets [ and ], and within the array are three objects, each representing an individual student, and each indicated with curly brackets { and }.

Parsing JSON from Android

Android comes with a set of classes to parse JSON. These will load JSON into memory and convert them into Java objects. We make use of the following classes.

JSONObject - represents an individual JSON object (such as the Tim Berners-Lee object above or each of the three individual students in the Tim Smith example)
JSONArray - represents an array of JSON objects, such as the array of the three students named Tim Smith in the second example above.

The general strategy for parsing JSON is to:

Load the string of JSON into a JSONArray or JSONObject (as appropriate)
If an array, loop through each individual JSONObject in the array and extract the individual attributes;
Otherwise, simply extract the attributes.

Here is an example of some code which parses the above example (array of three students) into an output string, parsedData. Imagine the JSON above, containing the three student objects, is stored in a variable called json.

var parsedData = ""
val jsonArray = JSONArray(json) // 'json' contains our JSON (see above)

// JSONArray.length() gives the number of elements inside the array
for(i in 0 until jsonArray.length()) {
    // Get the JSON object at index i within the JSON array
    val curObject = jsonArray.getJSONObject(i)
    
    // Extract the individual fields of the current object with getString()
    val username = curObject.getString("username")
    val year = curObject.getString("year")
    val course = curObject.getString("course")
    val phone = curObject.getString("phone")

    // Add the extracted data to an output string
    parsedData += "Username: $username, Year: $year, Course: $course, Phone: $phone"
}
println(parsedData)

Note the following:

We use getString() with an attribute name to retrieve a particular attribute of the JSONObject.
There are corresponding getInt() and getDouble() methods if the attributes are ints and doubles, rather than strings.
- See the JSONObject API page for full details on the methods of JSONObject.
If the JSON is in an incorrect format, a JSONException will be thrown.

Network communication

We will now examine how to perform communication across a network, with a web API, in Android and Kotlin. The first example will involve sending a basic GET request. (All of you should be familiar with HTTP requests, either via OODD or WAD; if not, please see here and here).

Sending a GET request

@Composable
fun NetworkComm() {
    var responseText by remember { mutableStateOf("") } 
    Column {
        Button( onClick = {
            var response = ""
            // Launch the coroutine using our LifecycleScope - as last week
            lifecycleScope.launch {

                // Switch to the IO context to perform the HTTP request
                withContext(Dispatchers.IO) {
                    // Send a GET request to https://kotlinlang.org
                    response = URL("https://kotlinlang.org").readText()
                }
                responseText = response
            }
        }) {
            Text("Get data from Web!")
        }
        Text(responseText)
    }
}

Note how easy it is to send a simple GET request in Kotlin. We simply use the readText() extension function of Java's URL class. This will send an HTTP request to the given URL and return the response.

Notice again how we use coroutines to send the request asynchronously, just like we did last week for querying an SQLite database. We switch to the Dispatchers.IO context to send the request, so that the UI remains responsive while waiting for the response to be delivered.

What about other requests? - Intro to Fuel

The URL extension function above allows us to send GET requests very easily, however we cannot send other request types (e.g. POST) in the same way. There are, however, many third party libraries which allow us to easily send HTTP request. One of these is Fuel: "the easiest HTTP networking library for Kotlin/Android" in the words of its authors (see here), which I would personally agree with as it is quite intuitive.

There's a lot you can do with Fuel, and we will cover some of the more common aspects, particularly those which are useful for interacting with a JSON web API.

More complex operations with Fuel

You can perform more complex operations with Fuel, in particular with JSON. To perform these operations, you use the httpGet() or httpPost() extension functions of String. This performs the request and then you can use methods such as response() or responseJson() to process the response. Such methods take a callback (typically a lambda function) which takes three parameters, a request object, a response object and a result object. The first two represent the HTTP request and response, while the final object is an object indicating whether the operation was successful or not. This is part of the Result library, written by the same developer as Fuel, which is a general library for error-handling.

We will look at three examples of using Fuel with Result to perform requests. Ensure these two dependencies are in your build.gradle:

implementation("com.github.kittinunf.fuel:fuel:2.3.1")
implementation("com.github.kittinunf.fuel:fuel-android:2.3.1")

The first dependency is Fuel itself. The second is the Android module for Fuel, which allows you to write asynchronous Fuel code without having to worry about creating the coroutines yourself: this is all done for you by the library.

Example 1 - simple case - get a string response

@Composable
fun FuelExample1() {
    var responseText by remember { mutableStateOf("") } 
    Column {
        Button( onClick = {
            // URL which returns JSON describing points of interest
            var url = "https://hikar.org/webapp/tile?x=4079&y=2740&z=13&poi=all"
            url.httpGet().response { request, response, result ->

                when(result) {
                    is Result.Success -> {
                        // result.get() gives ByteArray, decode to string
                        responseText = result.get().decodeToString()
                    }

                    is Result.Failure -> {
                        // is failure if HTTP error
                        responseText = "ERROR ${result.error.message}"
                    }
                }
            }
        }) {
            Text("Get data from Web!")
        }
        Text(responseText)
    }
}

Note how this is working:

We call the httpGet() String extension function on the URL, to send a GET request to it.
The response() method of the object returned by httpGet() takes a lambda as an argument, which runs when the response is received. This lambda takes three parameters:
- request - an object representing the HTTP request;
- response - an object representing the HTTP response. It has a statusCode property containing the HTTP status code, and a data property containing the actual response body, amongst others.
- result - a Result object, part of the Result library. We can use this to get a range of information about the result, for example whether the request was successful or not, and the data returned from the URL (as an alternative to response.data)
We then check the type of the result object. If the operation was successful, its type will be Result.Success, otherwise it will be Result.Failure. We check the type of a variable with the Kotlin is keyword.
If successful, we use the get() method of the result object to get the data. In this example, this will give the response as plain text; we could alternatively use response.data. Note that result.get() returns a ByteArray object, so we have to convert it to a String before updating the state variable. Note how we are not having to switch context here - the Fuel Android module does this for us!
Finally if the request failed, we use the error property of our result to get the exact error, and again update the state variable.

Example 2 - Returning the response as JSON

This next example will automatically parse the JSON returned from a URL. To parse JSON, you need to add Fuel's JSON module to your build.gradle:

implementation("com.github.kittinunf.fuel:fuel-json:2.3.1")

Imagine we have this JSON returned from the URL https://example.com/products/cornflakes:

[
    {
        "id": 10201,
        "name":"Cornflakes",
        "manufacturer":"Organic Products Ltd.",
        "price": 2.49 
    },
    {
        "id": 14641,
        "name":"Cornflakes",
        "manufacturer":"Cockadoodle Cereal Co.",
        "price": 1.79 
    },
    {
        "id": 16384,
        "name":"Cornflakes",
        "manufacturer":"Smith Emporium",
        "price": 0.79 
    },
]

We could use this code to fetch the data from the URL and parse the JSON:

@Composable
fun FuelExampleJson() {
    var responseText by remember { mutableStateOf("") } 
    Column {
        Button( onClick = {
            var url = "https://example.com/products/cornflakes"
            url.httpGet().responseJson { request, response, result ->
                when(result) {
                    is Result.Success -> {
                        val jsonArray = result.get().array()
                        var str = ""
                        for(i in 0 until jsonArray.length()) {
                            val curObj = jsonArray.getJSONObject(i)
                            str += "Manufacturer: ${curObj.getString("manufacturer")} Price: ${curObj.getString("price")}\n" 
                        }
                        responseText = str    
                    }

                    is Result.Failure -> {
                        responseText = "ERROR ${result.error.message}"
                    }
                }
            }
        }) {
            Text("Get data from Web (Fuel/JSON)!")
        }
        Text(responseText)
    }
}

Much of the code is the same as the previous example, but the key differences are:

firstly note how we use responseJson rather than response, this will tell Fuel to treat the response as JSON and automatically parse the data returned. The data can then be processed using the standard JSON API, as seen above.
secondly, as a consequence, note how result.get() returns an object containing parsed JSON rather than plain text. This is why it is a good idea to use result.get() when obtaining the response, because it returns different data depending on what type of response we asked for. We can then use either array() or obj() to obtain a JSONArray or JSONObject representing the data, depending on whether the top-level element in our JSON is an array or an object. It's an array here, as our JSON is an array of objects representing three cornflakes brands.
We can then parse the JSON using the standard JSON API, as before. Note how we parse the JSON, add it to a string, and then use string to update the state.

Example 3 - GSON

The standard JSON API works, but can be a little cumbersome when parsing large amounts of JSON. Consequently, an alternative JSON parsing library exists: GSON. GSON is a JSON serialising/deserialising library, in other words, it automatically converts JSON to objects (deserialisation) and objects to JSON (serialisation). With GSON, you have to create classes which match the structure of your JSON: in Kotlin, data classes are the obvious choice. So when using GSON, you can automatically create objects from your JSON without having to manually parse the JSON, as in the previous example. It makes use of the Java (and Kotlin) feature reflection, in which we can query an object at runtime to find out its class and what methods and attributes it has.

GSON is faily easy to use but with Fuel, it's easier still. We just need to define a data class corresponding to our JSON, so for the previous example we could define a Product class. Note how the properties of the Product class correspond to the fields in the JSON.

data class Product (val id: Int, val name: String, val manufacturer: String, val price: Double)

To use GSON, you need to add Fuel's GSON module to your build.gradle:

implementation("com.github.kittinunf.fuel:fuel-gson:2.3.1")

We can then use code such as the following to parse the JSON:

data class Product (val id: Int, val name: String, val manufacturer: String, val price: Double)

@Composable
fun FuelExampleGson() {
    var products by remember { mutableStateOf(listOf<Product>()) } 
    Column {
        Button( onClick = {
            val url = "https://example.com/products/cornflakes"
            url.httpGet().responseObject<List<Product>> { request, response, result ->
                when(result) {
                    is Result.Success -> {
                        products = result.get()
                    }

                    is Result.Failure -> {
                        responseText = "ERROR ${result.error.message}"
                    }
                }
            }
        }) {
            Text("Get data from Web (Fuel/GSON)!")
        }
        products.forEach {
            Text("${it.name} ${it.manufacturer} ${it.price}")
        }
    }
}

Note the differences from the previous example:

Rather than using responseJson, we use responseObject this time to indicate that we want to parse the JSON into a specific object. We have to specify the type of object the JSON is being parsed into: here it is <List<Product>> as we are parsing our JSON into a list of Products (not a single Product, because the server will return multiple products).
Consequently, this time, result.get() will return a List of Product objects.
This is used to update the state (a list of products).
We then loop through each product in the list and create a Text object with each one.

Example 4 - using POST

This final example shows how to send a POST request with Fuel. Note how we create a list Pair objects representing our POST data. Each item of POST data is a Pair object containing a key/value pair. The key (e.g. name, manufacturer and price) represents which item of data it is, and the value (e.g. "Cherry Jam") represents the data itself. We send the data within our POST request to the server.

@Composable
fun FuelExamplePost() {
    var responseText by remember { mutableStateOf("") } 
    Column {
        Button( onClick = {
            val url = "https://example.com/product/create"
            val postData = listOf("name" to "Cherry Jam", "manufacturer" to "Organic Jams Ltd", "price" to 1.89)
            url.httpPost(postData).response { request, response, result ->
                when (result) {
                    is Result.Success -> {
                        responseText = result.get().decodeToString()
                    }

                    is Result.Failure -> {
                        responseText = "ERROR ${result.error.message}"
                    }
                }
            }
        }) {
            Text("POST data to server!")
        }
        Text(responseText)
    }
}

For more information on Fuel, see the website.

Exercise

Preparation

This exercise will allow you to write an Android client to a web API which looks up all songs by a particular artist and sends them back to the client as JSON. If you are doing COM518 (Web Application Development) you can use the web API you have been developing in that module. If not, you will need to do a little bit of preparation work to set up a server on your own machine:

Download and install Node.js. If you are not aware, Node.js is an environment for writing JavaScript applications outside a browser, and it is frequently used for creating servers.
Install a third-party Node module (library) which you will need: Express. Express is a module to easily create web servers in Node.js. You'll also need the better-sqlite3 module, as the server uses an SQLite database. You should use the package management tool npm to install these dependencies. At the command prompt, enter:
```
npm install
```
If you are studying WAD, you can use your own Node/Express server from that module. If not, download and extract this ZIP file. It contains a simple Node.js server together with an SQLite database containing songs.
Before doing the exercise, run the server:
```
node app.mjs
```

Questions

Ensure you add the imports below to your application. Some of them will not be available via Alt-Enter as they have the same names as inbuilt Kotlin classes.

import com.github.kittinunf.fuel.core.Parameters
import com.github.kittinunf.fuel.httpGet
// import com.github.kittinunf.fuel.json.responseJson // for JSON - uncomment when needed
// import com.github.kittinunf.fuel.gson.responseObject // for GSON - uncomment when needed
import com.github.kittinunf.fuel.httpPost
import com.github.kittinunf.result.Result

If using a server on your local machine, you will have to add android:usesCleartextTraffic="true" to your application tag in the manifest, e.g.:

<application android:usesCleartextTraffic="true"...

This is because by default, Android disallows access to insecure (HTTP) servers (not using HTTPS) and your localhost server is a plain HTTP server, not HTTPS. This parameter disables the HTTPS restriction allowing your app to communicate with your local server.

Create a new Android project, and create a main activity containing a composable with a TextField (allowing the user to enter an artist to search for), a button (which when clicked will send a request to the server) and a Text (to show the results).
Write code to send a request, using Fuel, to your server, so that it looks up all songs by the user's chosen artist. Note that on the emulator, if you are running the server on your own local machine, you can use the special IP address 10.0.2.2 to access it. Your Node server will be running on port 3000, unless you changed the port, so you can access your server via a URL such as:
```
http://10.0.2.2:3000/artist/the artist you are looking for
```
Initially, do not parse the JSON, but just show the JSON unparsed.
Next, parse the JSON so the results are shown in user-friendly way on your interface. You can use either the standard JSON parser or GSON for this (see the second or third Fuel example, respectively). Your data class MUST be placed OUTSIDE your MainActivity.
You need to make your results a scrollable list in order to see all of them if there are many. You'll need to use LazyColumn for this: see topic 3.
Add a second composable to your project (accessible via navigation), to allow the user to add a new song. As you did in earlier topics, link the second screen to the first screen (ideally using a menu, but a button will be fine if you haven't done Topic 8). The second screen should contain text fields for title, artist, and year, and a button. When the button is clicked, send a POST request to your server containing the data. If you are using the madsongserver.js, or running your web server on localhost, the URL will be:
```
http://10.0.2.2:3000/song/create
```
If you are using your own server from COM518, you will need to add this additional line to your server:
```
app.use(express.urlencoded({extended: false}));
```
This is because with Fuel, the POST data is sent to the server a different way. It is not sent as JSON within the request body, but as a series of key-value pairs e.g.
```
title=Wonderwall&artist=Oasis&year=1995
```
The line above specifies a different piece of middleware which can parse POST data in this format (urlencoded).
If you are using madsongserver you will not need to do this, as it is already built-in.
Check that your POST request works. You can do this by searching for songs by the artist you used for the new song or looking inside the provided wadsongs.db SQLite database.