Naoto Yokoya is a lecturer with the Department of Complexity Science and Engineering, the Department of Computer Science, and the Department of Information Science at the University of Tokyo, running Sugiyama-Yokoya-Ishida Laboratory. He is the unit leader of the Geoinformatics Unit at the RIKEN Center for Advanced Intelligence Project (AIP).
His research interests include image processing, data fusion, and machine learning for understanding remote sensing images, with applications to disaster management and environmental monitoring.
He is an associate editor of IEEE Transactions on Geoscience and Remote Sensing (TGRS). He is an organizer of CVPR EarthVision workshop and IJCAI CDCEO workshop.
|X. Dong, N. Yokoya, L. Wang, and T. Uezato, ”Learning mutual modulation for self-supervised cross-modal super-resolution,” Proc. ECCV, 2022. |
PDF Code Quick Abstract
Abstract: Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In these modulations, we develop cross-domain adaptive filters to fully exploit cross-modal spatial dependency and help induce the source to emulate the resolution of the guide and induce the guide to mimic the modality characteristics of the source. Moreover, we adopt a cycle consistency constraint to train MMSR in a fully self-supervised manner. Experiments on various tasks demonstrate the state-of-the-art performance of our MMSR.
|G. Baier, A. Deschemps, M. Schmitt, and N. Yokoya, ”Synthesizing optical and SAR imagery from land cover maps and auxiliary raster data,” IEEE Transactions on Geoscience and Remote Sensing, 2022. |
PDF Code Quick Abstract
Abstract: We synthesize both optical RGB and SAR remote sensing images from land cover maps and auxiliary raster data using GANs. In remote sensing many types of data, such as digital elevation models or precipitation maps, are often not reflected in land cover maps but still influence image content or structure. Including such data in the synthesis process increases the quality of the generated images and exerts more control on their characteristics. Spatially adaptive normalization layers fuse both inputs and are applied to a full-blown generator architecture consisting of encoder and decoder, to take full advantage of the information content in the auxiliary raster data. Our method successfully synthesizes medium (10m) and high (1m) resolution images, when trained with the corresponding dataset. We show the advantage of data fusion of land cover maps and auxiliary information using mean intersection over union, pixel accuracy and Fréchet inception distance using pre-trained U-Net segmentation models. Handpicked images exemplify how fusing information avoids ambiguities in the synthesized images. By slightly editing the input our method can be used to synthesize realistic changes, i.e., raising the water levels. The source code is available at this https URL and we published the newly created high-resolution dataset at this https URL.
N. Yokoya, K. Yamanoi, W. He, G. Baier, B. Adriano, H. Miura, and S. Oishi, ”Breaking limits of remote sensing by deep learning from simulated data for flood and debris flow mapping,” IEEE Transactions on Geoscience and Remote Sensing, 2022.|
PDF Quick Abstract
Abstract: We propose a framework that estimates inundation depth (maximum water level) and debris-flow-induced topographic deformation from remote sensing imagery by integrating deep learning and numerical simulation. A water and debris flow simulator generates training data for various artificial disaster scenarios. We show that regression models based on Attention U-Net and LinkNet architectures trained on such synthetic data can predict the maximum water level and topographic deformation from a remote sensing-derived change detection map and a digital elevation model. The proposed framework has an inpainting capability, thus mitigating the false negatives that are inevitable in remote sensing image analysis. Our framework breaks limits of remote sensing and enables rapid estimation of inundation depth and topographic deformation, essential information for emergency response, including rescue and relief activities. We conduct experiments with both synthetic and real data for two disaster events that caused simultaneous flooding and debris flows and demonstrate the effectiveness of our approach quantitatively and qualitatively. Our code and datasets are available at https://github.com/nyokoya/dlsim.
C. Robinson, K. Malkin, N. Jojic, H. Chen, R. Qin, C. Xiao, M. Schmitt, P. Ghamisi, R. Haensch, and N. Yokoya, ”Global land cover mapping with weak supervision: Outcome of the 2020 IEEE GRSS Data Fusion Contest,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 2021.|
PDF Quick Abstract
Abstract: This paper presents the scientific outcomes of the 2020 Data Fusion Contest organized by the Image Analysis and Data Fusion Technical Committee of the IEEE Geoscience and Remote Sensing Society. The 2020 Contest addressed the problem of automatic global land-cover mapping with weak supervision, i.e. estimating high-resolution semantic maps while only low-resolution reference data is available during training. Two separate competitions were organized to assess two different scenarios: 1) high-resolution labels are not available at all and 2) a small amount of high-resolution labels are available additionally to low-resolution reference data. In this paper we describe the DFC2020 dataset that remains available for further evaluation of corresponding approaches and report the results of the best-performing methods during the contest.
T. Uezato, D. Hong, N. Yokoya, and W. He, “Guided deep decoder: Unsupervised image pair fusion,” Proc. ECCV (spotlight), 2020.|
PDF Supmat Code Quick Abstract
Abstract: The fusion of input and guidance images that have a tradeoff in their information (e.g., hyperspectral and RGB image fusion or pansharpening) can be interpreted as one general problem. However, previous studies applied a task-specific handcrafted prior and did not address the problems with a unified approach. To address this limitation, in this study, we propose a guided deep decoder network as a general prior. The proposed network is composed of an encoder-decoder network that exploits multi-scale features of a guidance image and a deep decoder network that generates an output image. The two networks are connected by feature refinement units to embed the multi-scale features of the guidance image into the deep decoder network. The proposed network allows the network parameters to be optimized in an unsupervised way without training data. Our results show that the proposed network can achieve state-of-the-art performance in various image fusion problems.
|2013 Mar.||D.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan|
|2010 Sep.||M.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan|
|2008 Mar.||B.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan|
|2020 May||-||present||Lecturer | The University of Tokyo | Japan|
|2018 Jan.||-||present||Unit Leader | RIKEN | Japan|
|2019 Apr.||-||2020 Mar.||Visiting Associate Professor | Tokyo University of Agriculture and Technology | Japan|
|2015 Dec.||-||2017 Nov.||Alexander von Hunboldt Research Fellow | DLR and TUM | Germany|
|2013 Jul.||-||2017 Dec.||Assistant Professor | The University of Tokyo | Japan|
|2013 Aug.||-||2014 Jul.||Visiting Scholar | National Food Research Institute (NFRI) | Japan|
|2012 Apr.||-||2013 Jun.||JSPS Research Fellow | The University of Tokyo | Japan|
|1st place in the 2017 IEEE GRSS Data Fusion Contest.|
|Alexander von Humboldt research fellowship for postdoctoral researchers (2015).|
|Best presentation award of the Remote Sensing Society of Japan (2011, 2012, 2019).|
|2022 Apr.||-||2026 Mar.||PI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS)|
|2021 Apr.||-||2028 Mar.|| PI, FOREST (Fusion Oriented REsearch for disruptive Science and Technology),|
Japan Science and Technology Agency (JST)
|2021 Apr.||-||2024 Mar.||CI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS)|
|2019 Apr.||-||2022 Mar.||CI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS)|
|2018 Apr.||-||2021 Mar.||PI, Grant-in-Aid for Young Scientists, Japan Society for the Promotion of Science (JSPS)|
|2015 Apr.||-||2018 Mar.||PI, Grant-in-Aid for Young Scientists (B), Japan Society for the Promotion of Science (JSPS)|
|2015 Jan.||-||2016 Dec.||PI, Research Grant Program, Kayamori Foundation of Informational Science Advancement|
|2013 Aug.||-||2014 Mar.|| PI, Adaptable and Seamless Technology Transfer Program through Target-driven R&D (A-STEP),|
Japan Science and Technology Agency (JST)
|2012 Apr.||-||2013 Jun.||PI, Grant-in-Aid for JSPS Fellows, Japan Society for the Promotion of Science (JSPS)|
|Mathematics for Information Science||The University of Tokyo||since 2020|
|Computer Vision||The University of Tokyo||since 2021|
|Remote Sensing Image Analysis||The University of Tokyo||since 2021|
|2015 Sep.||-||2017 Nov.||Visiting scholar at DLR and TUM, München, Germany.|
|2011 Oct.||-||2012 Mar.||Visiting student at the Grenoble Institute of Technology, Grenoble, France.|
|Organizer||IJCAI CDCEO Workshop 2022|
|Organizer||CVPR EarthVision Workshop 2019, 2020, 2021, 2022|
|Chair & Co-Chair||IEEE GRSS Image Analysis and Data Fusion Technical Committee (2017-2021)|
|Secretary||IEEE GRSS All Japan Joint Chapter (2018-2021)|
|Organizer||IEEE GRSS Data Fusion Contest 2018, 2019, 2020, 2021|
|Student Activity & TIE Event Chair||IEEE IGARSS 2019|
|Program Chair||IEEE WHISPERS 2015|
|Associate Editor||IEEE Transactions on Geoscience and Remote Sensing, since 2021.|
|Associate Editor||IEEE Journal on Selected Topics in Applied Earth Observations and Remote Sensing, 2018-2021.|
|Editorial Board Member||Remote Sensing, since 2018.|
List of Special Issues
"Remote Sensing on Land Surface Albedo"
IEEE Journal on Selected Topics in Applied Earth Observations and Remote Sensing|
List of Special Issues
"Benchmarking in Remote Sensing Data Science"
|Guest Editor|| IEEE Geoscience and Remote Sensing Letters|
Special Issue on "Advanced Processing for Multimodal Optical Remote Sensing Imagery"
Reviewers for various journals and conferences:
- IEEE Transactions on Geoscience and Remote Sensing
- IEEE Transactions on Image Processing
- IEEE Transactions on Signal Processing
- IEEE Transactions on Computational Imaging
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- IEEE Journal of Selected Topics on Applied Remote Sensing
- IEEE Journal of Selected Topics in Signal Processing
- IEEE Geoscience and Remote Sensing Letters
- IEEE Geoscience and Remote Sensing Magazine
- Proceedings of the IEEE
- Remote Sensing
- Remote Sensing of Environment
- International Journal of Remote Sensing
- Pattern Recognition
- Pattern Recognition Letters