Joint Optimization of Camera Model
and Deep Neural Network for Image Recognition

Youta Noboru¹, Yuko Ozasa¹, Masayuki Tanaka²

¹Tokyo Denki University, ²Institute of Science Tokyo
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2026

Paper Code

Visual comparison of images generated by different camera models.

Abstract

In this paper, we propose joint optimization of a camera model and a deep neural network (DNN) for image classification and object detection tasks. The camera model consists of an image sensor model parameterized by the camera spectral sensitivity (CSS) and an image signal processing (ISP) model. We assume the camera model is composed of a three-sensor imager without a demosaicing process and an ISP with simple color correction and gamma correction. The DNNs then classify or detect objects. A key contribution of this paper is the joint optimization of not only the ISP model and DNN but also the image sensor model. For stable joint optimization, we have implemented a fully differentiable camera model. Therefore, we can jointly optimize the camera model and the DNN. Experimental comparisons with the flower and leaf datasets show that our approach outperforms existing approaches. Furthermore, we demonstrate that our approach is also effective for the object detection task.

Three optimization approaches for image classification.

Overview of the proposed joint optimization approach. A hyperspectral (HS) image is input into a camera model comprising a learnable camera spectral sensitivity (CSS) and an image signal processing (ISP) module. The ISP includes a color correction matrix (CCM) and a gamma correction component that incorporates our gradient constraint method. The resulting image is then passed to a classification network. The entire system is trained jointly via backpropagation of the task loss.

Visual comparison of images generated by different camera models.

Classification accuracy on the leaf dataset across different camera models and optimization methods, where values represent average ± standard deviation. Bold and underlined values indicate the best and second-best results, respectively.

Ablation study on the contribution of optimizing the CSS, the CCM, the gamma correction, and the network model of ResNet-18 with the leaf dataset. Bold and underlined values indicate the best and second-best results, respectively.

BibTeX

@InProceedings{Noboru_2026_WACV,
    author    = {Noboru, Youta and Ozasa, Yuko and Tanaka, Masayuki},
    title     = {Joint Optimization of Camera Model and Deep Neural Network for Image Recognition},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {March},
    year      = {2026},
    pages     = {7626-7635}
}