Issue
I'm investigating the new CameraX API, in relation to how viable it might be to switch over from our current Camera2 system.
In our Camera2 system, we use an OpenGL surface to capture the frames from a PreviewCaptureSession, and we are hitting consistent 30fps image processing speeds across most devices, with some able to hit 60fps with AutoExposure settings enabled.
CameraX isn't giving anything near that speed and I'm not sure if its something I've missed in setup.
I've setup the test examples for CameraX and ImageAnalysis, but I'm getting locked frame rates for the number of images that comes through.
For example, I could set the resolution as low as 320x240 up to 1920x960 and both will come out at a (seemingly capped) 16fps.
When I add a Preview usecase to run along side it, and set enableTorch(true), the ImageAnalysis usecase will suddenly start getting more like 20fps, with it occasionally peaking up to 30ish.
Clearly the Preview usecase alters some of the AutoExposure state of the camera?
Here's a snippet of my current setup...
private fun startCameraAnalysis() {
val metrics = DisplayMetrics().also { viewFinder.display.getRealMetrics(it) }
var resolution = Size(metrics.widthPixels, metrics.heightPixels)
resolution = Size(640, 480) //set to fixed size for testing
val aspectRatio = Rational(resolution.width, resolution.height)
val rotation = viewFinder.display.rotation
// Setup image analysis pipeline
val analyzerConfig = ImageAnalysisConfig.Builder().apply {
val analyzerThread = HandlerThread(
"LuminosityAnalysis").apply { start() }
setCallbackHandler(Handler(analyzerThread.looper))
setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
setTargetRotation(rotation)
setTargetAspectRatio(aspectRatio)
setTargetResolution(resolution)
}.build()
// Setup preview pipeline
val previewConfig = PreviewConfig.Builder().apply {
setTargetRotation(rotation)
setTargetAspectRatio(aspectRatio)
setTargetResolution(resolution)
}.build()
// Build Preview useCase
val preview = Preview(previewConfig)
preview.enableTorch(true)
// Build Analysis useCase
val analyzer = ImageAnalysis(analyzerConfig)
analyzer.analyzer = LuminosityAnalyzer()
CameraX.bindToLifecycle(this, preview, analyzer )
preview.enableTorch(true)
}
Is there anyway to alter the camera settings in CameraX around the ImageAnalysis to get higher frame rates?
Is there anyway at all in fact to alter things like Sensor Duration, ISO, exposure?
Solution
So I've spent some more time investigating and I think I've come up with a solution for now.
It turns out, the ImageAnalysisConfig is not extendable, so you can't alter the camera configuration when just using one of those, so the default camera settings will be used which on my phone I think resulted in AE being on and hitting 16ish FPS.
If you spin up a PreviewConfig to run along side it at the same time, you can then extend this with a Camera2Config.Extender and alter the camera2 properties directly. This can increase the camera preview frame rate, and the Analyser will also start getting frames at the same rate.
So for example, I add this to my PreviewConfig...
// Create Camera2 extender
var camera2Extender = Camera2Config.Extender(previewConfig)
.setCaptureRequestOption(CaptureRequest.CONTROL_MODE, CaptureRequest.CONTROL_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.FLASH_MODE, CaptureRequest.FLASH_MODE_TORCH)
.setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY, 100)
.setCaptureRequestOption(CaptureRequest.SENSOR_FRAME_DURATION, 16666666)
.setCaptureRequestOption(CaptureRequest.SENSOR_EXPOSURE_TIME, 20400000)
So this started hitting 30fps fine in the ImageAnalyser.
If I want to hit 60, I can set...
.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range(60,60))
Obviously assuming the device support (60,60) target FPS range.
So it seems the full Camera2 logic is still available in CameraX, its just a bit clunky that its a little hidden away in a Camera2Config extender, and this only works with Preview use cases.
Answered By - Iain Stanford
Answer Checked By - Mildred Charles (JavaFixing Admin)