Yokoya

Naoto

Naoto Yokoya

Yokoya Naoto

横矢

横矢直人

University of Tokyo

東京大学

hyperspectral

ハイパースペクトル

remote sensing

リモートセンシング

pattern recognition

パターン認識

data fusion

データ融合

Current position

Naoto Yokoya is a lecturer with the Department of Complexity Science and Engineering, the Department of Computer Science, and the Department of Information Science at the University of Tokyo, running Sugiyama-Yokoya-Ishida Laboratory. He is the unit leader of the Geoinformatics Unit at the RIKEN Center for Advanced Intelligence Project (AIP).

His research interests include image processing, data fusion, and machine learning for understanding remote sensing images, with applications to disaster management and environmental monitoring.

He is an associate editor of IEEE Transactions on Geoscience and Remote Sensing (TGRS). He is an organizer of CVPR EarthVision workshop and IJCAI CDCEO workshop.

Featured publications

X. Dong, N. Yokoya, L. Wang, and T. Uezato, ”Learning mutual modulation for self-supervised cross-modal super-resolution,” Proc. ECCV, 2022.
PDF    Code    Quick Abstract

Abstract: Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In these modulations, we develop cross-domain adaptive filters to fully exploit cross-modal spatial dependency and help induce the source to emulate the resolution of the guide and induce the guide to mimic the modality characteristics of the source. Moreover, we adopt a cycle consistency constraint to train MMSR in a fully self-supervised manner. Experiments on various tasks demonstrate the state-of-the-art performance of our MMSR.

G. Baier, A. Deschemps, M. Schmitt, and N. Yokoya, ”Synthesizing optical and SAR imagery from land cover maps and auxiliary raster data,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
PDF    Code    Quick Abstract

Abstract: We synthesize both optical RGB and SAR remote sensing images from land cover maps and auxiliary raster data using GANs. In remote sensing many types of data, such as digital elevation models or precipitation maps, are often not reflected in land cover maps but still influence image content or structure. Including such data in the synthesis process increases the quality of the generated images and exerts more control on their characteristics. Spatially adaptive normalization layers fuse both inputs and are applied to a full-blown generator architecture consisting of encoder and decoder, to take full advantage of the information content in the auxiliary raster data. Our method successfully synthesizes medium (10m) and high (1m) resolution images, when trained with the corresponding dataset. We show the advantage of data fusion of land cover maps and auxiliary information using mean intersection over union, pixel accuracy and Fréchet inception distance using pre-trained U-Net segmentation models. Handpicked images exemplify how fusing information avoids ambiguities in the synthesized images. By slightly editing the input our method can be used to synthesize realistic changes, i.e., raising the water levels. The source code is available at this https URL and we published the newly created high-resolution dataset at this https URL.

N. Yokoya, K. Yamanoi, W. He, G. Baier, B. Adriano, H. Miura, and S. Oishi, ”Breaking limits of remote sensing by deep learning from simulated data for flood and debris flow mapping,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
PDF    Quick Abstract

Abstract: We propose a framework that estimates inundation depth (maximum water level) and debris-flow-induced topographic deformation from remote sensing imagery by integrating deep learning and numerical simulation. A water and debris flow simulator generates training data for various artificial disaster scenarios. We show that regression models based on Attention U-Net and LinkNet architectures trained on such synthetic data can predict the maximum water level and topographic deformation from a remote sensing-derived change detection map and a digital elevation model. The proposed framework has an inpainting capability, thus mitigating the false negatives that are inevitable in remote sensing image analysis. Our framework breaks limits of remote sensing and enables rapid estimation of inundation depth and topographic deformation, essential information for emergency response, including rescue and relief activities. We conduct experiments with both synthetic and real data for two disaster events that caused simultaneous flooding and debris flows and demonstrate the effectiveness of our approach quantitatively and qualitatively. Our code and datasets are available at https://github.com/nyokoya/dlsim.

C. Robinson, K. Malkin, N. Jojic, H. Chen, R. Qin, C. Xiao, M. Schmitt, P. Ghamisi, R. Haensch, and N. Yokoya, ”Global land cover mapping with weak supervision: Outcome of the 2020 IEEE GRSS Data Fusion Contest,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 2021.
PDF    Quick Abstract

Abstract: This paper presents the scientific outcomes of the 2020 Data Fusion Contest organized by the Image Analysis and Data Fusion Technical Committee of the IEEE Geoscience and Remote Sensing Society. The 2020 Contest addressed the problem of automatic global land-cover mapping with weak supervision, i.e. estimating high-resolution semantic maps while only low-resolution reference data is available during training. Two separate competitions were organized to assess two different scenarios: 1) high-resolution labels are not available at all and 2) a small amount of high-resolution labels are available additionally to low-resolution reference data. In this paper we describe the DFC2020 dataset that remains available for further evaluation of corresponding approaches and report the results of the best-performing methods during the contest.

T. Uezato, D. Hong, N. Yokoya, and W. He, “Guided deep decoder: Unsupervised image pair fusion,” Proc. ECCV (spotlight), 2020.
PDF    Supmat    Code    Quick Abstract

Abstract: The fusion of input and guidance images that have a tradeoff in their information (e.g., hyperspectral and RGB image fusion or pansharpening) can be interpreted as one general problem. However, previous studies applied a task-specific handcrafted prior and did not address the problems with a unified approach. To address this limitation, in this study, we propose a guided deep decoder network as a general prior. The proposed network is composed of an encoder-decoder network that exploits multi-scale features of a guidance image and a deep decoder network that generates an output image. The two networks are connected by feature refinement units to embed the multi-scale features of the guidance image into the deep decoder network. The proposed network allows the network parameters to be optimized in an unsupervised way without training data. Our results show that the proposed network can achieve state-of-the-art performance in various image fusion problems.

Education

2013 Mar. D.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan
2010 Sep. M.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan
2008 Mar. B.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan

Experience

2020 May-present Lecturer | The University of Tokyo | Japan
2018 Jan.-present Unit Leader | RIKEN | Japan
2019 Apr.-2020 Mar. Visiting Associate Professor | Tokyo University of Agriculture and Technology | Japan
2015 Dec.-2017 Nov. Alexander von Hunboldt Research Fellow | DLR and TUM | Germany
2013 Jul.-2017 Dec. Assistant Professor | The University of Tokyo | Japan
2013 Aug.-2014 Jul. Visiting Scholar | National Food Research Institute (NFRI) | Japan
2012 Apr.-2013 Jun. JSPS Research Fellow | The University of Tokyo | Japan

Award

1st place in the 2017 IEEE GRSS Data Fusion Contest.
Alexander von Humboldt research fellowship for postdoctoral researchers (2015).
Best presentation award of the Remote Sensing Society of Japan (2011, 2012, 2019).

Research grants

2022 Apr.-2026 Mar. PI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS)
2021 Apr.-2028 Mar. PI, FOREST (Fusion Oriented REsearch for disruptive Science and Technology),
 Japan Science and Technology Agency (JST)
2021 Apr.-2024 Mar. CI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS)
2019 Apr.-2022 Mar. CI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS)
2018 Apr.-2021 Mar. PI, Grant-in-Aid for Young Scientists, Japan Society for the Promotion of Science (JSPS)
2015 Apr.-2018 Mar. PI, Grant-in-Aid for Young Scientists (B), Japan Society for the Promotion of Science (JSPS)
2015 Jan.-2016 Dec. PI, Research Grant Program, Kayamori Foundation of Informational Science Advancement
2013 Aug.-2014 Mar. PI, Adaptable and Seamless Technology Transfer Program through Target-driven R&D (A-STEP),
 Japan Science and Technology Agency (JST)
2012 Apr.-2013 Jun. PI, Grant-in-Aid for JSPS Fellows, Japan Society for the Promotion of Science (JSPS)

Teaching

Mathematics for Information ScienceThe University of Tokyosince 2020
Computer VisionThe University of Tokyosince 2021
Remote Sensing Image AnalysisThe University of Tokyosince 2021

International mobility

2015 Sep.-2017 Nov. Visiting scholar at DLR and TUM, München, Germany.
2011 Oct.-2012 Mar. Visiting student at the Grenoble Institute of Technology, Grenoble, France.

Service

Organizer IJCAI CDCEO Workshop 2022
Organizer CVPR EarthVision Workshop 2019, 2020, 2021, 2022
Chair & Co-Chair IEEE GRSS Image Analysis and Data Fusion Technical Committee (2017-2021)
Secretary IEEE GRSS All Japan Joint Chapter (2018-2021)
Organizer IEEE GRSS Data Fusion Contest 2018, 2019, 2020, 2021
Student Activity & TIE Event Chair IEEE IGARSS 2019
Program Chair IEEE WHISPERS 2015

Editorial activity

Associate Editor IEEE Transactions on Geoscience and Remote Sensing, since 2021.
Associate Editor IEEE Journal on Selected Topics in Applied Earth Observations and Remote Sensing, 2018-2021.
Editorial Board Member Remote Sensing, since 2018.
Guest Editor  Remote Sensing
 List of Special Issues

 "Remote Sensing on Land Surface Albedo"
 "Advanced Machine Learning Techniques for High-Resolution Remote Sensing Data Analysis"
 "Data Fusion for Urban Applications"
 "Spectral Data Meets Machine Learning: From Datasets to Algorithms and Applications"
 "Deep Learning and Feature Mining Using Hyperspectral Imagery"
 "Point Cloud Processing in Remote Sensing"
 "Multisensor Data Fusion in Remote Sensing"
 "Spatial Enhancement of Hyperspectral Data and Applications"

Guest Editor  IEEE Journal on Selected Topics in Applied Earth Observations and Remote Sensing
 List of Special Issues

 "Benchmarking in Remote Sensing Data Science"
 "Semantic Extraction and Fusion of Multimodal Remote Sensing Data: Algorithms and Applications"
 "2020 Gaofen Challenge on Automated High-Resolution Earth Observation Image Interpretation"
 "Integrating Physics and Artificial Intelligence for Remote Sensing Applications"
 "Computer Vision-based Approaches for Earth Observation"
 "Hyperspectral Remote Sensing and Imaging Spectroscopy"

Guest Editor IEEE Geoscience and Remote Sensing Letters
 Special Issue on "Advanced Processing for Multimodal Optical Remote Sensing Imagery"

Reviewer

Reviewers for various journals and conferences:
- IEEE Transactions on Geoscience and Remote Sensing
- IEEE Transactions on Image Processing
- IEEE Transactions on Signal Processing
- IEEE Transactions on Computational Imaging
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- IEEE Journal of Selected Topics on Applied Remote Sensing
- IEEE Journal of Selected Topics in Signal Processing
- IEEE Geoscience and Remote Sensing Letters
- IEEE Geoscience and Remote Sensing Magazine
- Proceedings of the IEEE
- Remote Sensing
- Remote Sensing of Environment
- International Journal of Remote Sensing
- Pattern Recognition
- Pattern Recognition Letters
- Neurocomputing
- CVPR
- ECCV
- AAAI
- WACV
- ICME