Mobile Development and 3D Graphics - Part 6

Using the Camera

Introduction to using the camera

Many apps need to use the camera on Android. Apps can continuously display a view of the real world through the camera (called a camera preview), take pictures, and save them on the device. Furthermore, the camera is needed in augmented reality, because we need to display a real-world view as seen through the device's camera.

There are many approaches to using the Camera in Android (see here):

Use an implicit intent to launch the inbuilt camera app (this is easy, but not so flexible as you cannot customise the app's behaviour or the interface); see here. You use the ACTION_IMAGE_CAPTURE intent; you can also add a filename as an extra and the photo app will save the photo to that filename if it has been provided. Also, a thumbnail is sent back in the return intent.
Use the Camera2 API. This is the current low-level API used by Android, but it is complex, with example code and detailed explanations hard to find.
Use the traditional Camera API (relatively simple, but deprecated at API level 21 so not guaranteed to work in the future and therefore not advised)
Use a third-party library which wraps the Camera2 API and simplifies development (e.g. CameraKit or FotoApparat). Around three years ago this was the favoured approach by many, but this was before CameraX (described below) was mature.
Use CameraX, a new official Android library, part of Jetpack.. This is the currently-recommended approach. CameraX provides a wrapper to Camera2, and is much simpler to use than Camera2, with many more working examples.

Using CameraX

We will focus on CameraX as it is the currently-recommended approach and reasonably easy to understand, compared to the lower-level Camera2 API.

Asking for permission

The first thing we need to do is ask the user for permission to use the camera. For obvious reasons, the camera (permission name CAMERA) is a sensitive permission, and therefore the user must grant permission explicitly at runtime. The same logic is used as any other sensitive permission, such as location; we check whether the permission has been granted already, and if not, request it from the user.

CameraX dependencies

You need to add these dependencies to your build.gradle:

def cxver="1.1.0"
implementation "androidx.camera:camera-core:${cxver}"
implementation "androidx.camera:camera-camera2:${cxver}"
implementation "androidx.camera:camera-lifecycle:${cxver}"
implementation "androidx.camera:camera-view:${cxver}"

Displaying a preview

The next thing we will consider is displaying a preview. We use the CameraX API for this, as follows:

private fun startCamera() {
    val lifecycleOwner = this // can be any component with a lifecycle
    val preview1 = findViewById<PreviewView>(R.id.preview1)
    val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
    cameraProviderFuture.addListener(
        {
            val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()
            val preview = Preview.Builder().build().also {
                // "preview1" is a UI object representing the preview, we use
                // findViewById() to obtain it (see above)
                it.setSurfaceProvider(preview1.surfaceProvider)
            }
            val cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA
            try {
                cameraProvider.unbindAll()
                cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, preview)
            } catch (e: Exception) {
                Log.d("CAMERAX1", e.stackTraceToString())
            }
        }, ContextCompat.getMainExecutor(this)
    )
}

There's quite a lot of code there: much of the complexity is related to the fact that opening the camera is an asynchronous process so might take some time. So what does it do?

First of all we obtain a camera provider future. A future is a task which will complete at some point in the future; it is rather like a promise.
```
val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
```
We then set up a listener callback which will run when the future completes, i.e. when the camera is ready.
```
cameraProviderFuture.addListener(callback, executor)
```
The callback function (shown in full above) runs when the future has completed. The executor (ContextCompat.getMainExecutor(this) here) is an object used for scheduling tasks. See here.
Within this callback, we first get a camera provider object (ProcessCameraProvider) from the completed future.
Next we create a Preview object using a builder:
```
val preview = Preview.Builder().build()
```
We then link the preview with a particular UI element (preview1 here), to define where the preview will be displayed. The UI element needs to be a PreviewView: see below.
```
it.setSurfaceProvider(preview1.surfaceProvider)
```
We then select the camera we're interested in (here, the back camera, i.e. the one facing away from the user):
```
val cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA
```
...and then use the cameraProvider to bind the camera preview for the selected camera to the activity, or fragment, lifecycle (so that the camera behaves appropriately at different points in the lifecycle, for example it's automatically closed when the activity is destroyed):
```
try {
    cameraProvider.unbindAll()
    cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, preview)
} catch(e: Exception) {
    Log.e("CAMERAX1", e.stackTraceToString())
}
```
That's it! So the key steps, to summarise, are:
- We obtain a camera provider object. This is asynchronous as it might take time to connect to the camera, so we provide a callback function which runs when we have connected.
- We then obtain a camera preview object, and link it to a specific UI element, in which we display the preview.
- Using the camera provider we obtain a specific camera (front or back), and link it with the preview object and with the component (activity or fragment)'s lifecycle so that the camera will behave as we expect according to that lifecycle.

What view should be used for the preview?

We need a particular type of View object to show the preview. This is the PreviewView and is shown in the layout extract below:

<androidx.camera.view.PreviewView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:id="@+id/preview1"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toTopOf="parent" />

Camera use cases

CameraX uses the concept of camera use cases. These are specific uses of the camera, such as displaying a preview, capturing an image or performing analysis on the image. Each use case is created as a separate object, and then bound to the lifecycle using the CameraProvider's bindToLifecycle() method. The previous example created just one use case (the preview) and bound that to the activity lifecycle:

cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, preview)

However, we can create additional use cases, such as image capture, and bind them to the lifecycle by adding them as additional arguments:

cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, preview, imageCapture)

where imageCapture is an ImageCapture object. We will discuss image capture below.

Image capture

Having looked at the camera preview use case, we will now look at the image capture use case. This is easy to create: we use an ImageCapture.Builder() to build an ImageCapture object:

imageCapture = ImageCapture.Builder().build()

and then it can be added to the lifecycle as we have seen above:

cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, preview, imageCapture)

Taking a picture

We then need to take a picture as a separate operation, typically in response to the user pressing a "Capture" button. The same ImageCapture object is used. Here is an example:

private fun takePicture() {

    imageCapture?.apply {
        val name = "captured_image"
        val contentValues = ContentValues().apply {
            put(MediaStore.MediaColumns.DISPLAY_NAME, name)
            put(MediaStore.MediaColumns.MIME_TYPE, "image/jpeg")
            put(MediaStore.Images.Media.RELATIVE_PATH, "Pictures/CameraX1")
        }

        val outputOptions = ImageCapture.OutputFileOptions.Builder(
            contentResolver,
            MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValues
        ).build()

        val imageSavedCallback = object: ImageCapture.OnImageSavedCallback {

            override fun onImageSaved(outputFileResults: ImageCapture.OutputFileResults) {
                AlertDialog.Builder(this@MainActivity).setPositiveButton("OK", null)
                    .setMessage("Saved Successfully").show()
            }

            override fun onError(e: ImageCaptureException) {
                AlertDialog.Builder(this@MainActivity).setPositiveButton("OK", null)
                    .setMessage("Error: ${e.message}").show()
            }
        }

        this.takePicture(
            outputOptions,
            ContextCompat.getMainExecutor(this@MainActivity),
            imageSavedCallback
        )
    }
}

What is this doing?

We are first specifying a location to save the image. This is using the MediaStore API. The MediaStore API is an Android API which allows fast lookup of media by index; see here. Media is saved on the device in particular standard locations (such as the Pictures folder). The first thing we do is specify the properties of the image : its name, its MIME type (content type) and the location we wish to save it. This is created as a ContentValues object:
```
val contentValues = ContentValues().apply {
    put(MediaStore.MediaColumns.DISPLAY_NAME, name)
    put(MediaStore.MediaColumns.MIME_TYPE, "image/jpeg")
    put(MediaStore.Images.Media.RELATIVE_PATH, "Pictures/CameraX1")
}
```
We then create an ImageCapture.OutputFileOptions object from the ContentValues. This provides the image output options in a format the ImageCapture API of CameraX understands:
```
val outputOptions = ImageCapture.OutputFileOptions.Builder(
    contentResolver,
    MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValues
).build()
```

Finally we actually take the picture and save it to the specified location:

val imageSavedCallback = object: ImageCapture.OnImageSavedCallback {

    override fun onImageSaved(outputFileResults: ImageCapture.OutputFileResults) {
        AlertDialog.Builder(this@MainActivity).setPositiveButton("OK", null)
            .setMessage("Saved Successfully").show()
    }

    override fun onError(e: ImageCaptureException) {
        AlertDialog.Builder(this@MainActivity).setPositiveButton("OK", null)
            .setMessage("Error: ${e.message}").show()
    }
}

this.takePicture(
    outputOptions,
    ContextCompat.getMainExecutor(this@MainActivity),
    imageSavedCallback
)

The imageSavedCallback object is an instance of an anonymous class inheriting from ImageCapture.OnImageSavedCallback which provides two overridden methods, onImageSaved() and onError(), which run if the image capture and saving to file was successful, or not successful, respectively.
We then actually take the picture with the takePicture() method of the ImageCapture object, passing the image output options, an Executor again (used for scheduling the picture-taking task), and the OnImageSavedCallback object we just created.

View binding

This topic isn't directly related to the camera (but can be used for the PreviewView), but this week is a convenient point to introduce it.

So far you have been using findViewById(), with an element ID, to access elements from your layout in your Kotlin code. This works, but can become a bit long-winded if you need to access many UI elements. Luckily, a relatively new feature of Android, view binding, now exists to make accessing UI elements more concise. View binding provides a ViewBinding object from your Kotlin code, which contains the UI elements as properties, with property names equal to their IDs in the XML.

To use view binding you must enable it in the android section of your app's build.gradle:

android {
    ...
    buildFeatures {
        viewBinding true
    }
}

In your activity, you then declare a lateinit variable representing the binding object:

private lateinit var binding: ActivityMainBinding

Note the naming here. A binding class called ActivityMainBinding will correspond to activity_main.xml. The class capitalises the words making up the layout file, and removes underscores. So a layout called map_chooser.xml would have a binding class called MapChooserBinding.

In your onCreate(), you initialise it by inflating the layout (layoutInflater is a property of the activity, representing a LayoutInflater object which can be used to inflate XML layouts):

binding = ActivityMainBinding.inflate(layoutInflater)

Then, you set the activity's view to the root of the binding:

setContentView(binding.root)

After this, you will be able to access UI elements as properties of the binding variable. For example, binding.preview1 would reference the UI element with the ID of preview1.

Exercise

Implement an application to display a camera preview. Your layout should have a PreviewView occupying most of the screen, together with a "Take Picture" button. You will need to handle the CAMERA permission; look at your mapping application from week 2 (which used GPS permission) if you're unsure.
Alter your code to use view binding.
Implement the "Take Picture" button so it saves the image to a hard-coded location on your device, which should be a sub-folder of the Pictures folder.
- Note! You will encounter problems with the above code if running on a device running a version of Android below version 10. On older devices you need to use the classic file API rather than the MediaStore, using code such as:
```
val file = "${getExternalFilesDir(Environment.DIRECTORY_PICTURES)}/test.jpg"
val outputOptions: ImageCapture.OutputFileOptions =
	ImageCapture.OutputFileOptions.Builder(File(file)).build()
```
  and then use outputOptions as shown on the original example. You will also need to give your app the WRITE_EXTERNAL_STORAGE permissionin the manifest.
Add a TextView on the UI to allow the user to specify the filename to save the picture to. It should still be in the same folder as the previous question, though.