Naoto Yokoya is a lecturer with the Department of Complexity Science and Engineering, the Department of Computer Science, and the Department of Information Science at the University of Tokyo, running Sugiyama-Yokoya-Ishida Laboratory. He is the unit leader of the Geoinformatics Unit at the RIKEN Center for Advanced Intelligence Project (AIP).
His research interests include image processing, data fusion, and machine learning for understanding remote sensing images, with applications to disaster management and environmental monitoring.
He is an associate editor of IEEE Transactions on Geoscience and Remote Sensing (TGRS). He is an organizer of CVPR EarthVision workshop and IJCAI CDCEO workshop.
![]() | G. Baier, A. Deschemps, M. Schmitt, and N. Yokoya, ”Synthesizing optical and SAR imagery from land cover maps and auxiliary raster data,” IEEE Transactions on Geoscience and Remote Sensing, (in press), 2021. PDF Code Quick Abstract Abstract: We synthesize both optical RGB and SAR remote sensing images from land cover maps and auxiliary raster data using GANs. In remote sensing many types of data, such as digital elevation models or precipitation maps, are often not reflected in land cover maps but still influence image content or structure. Including such data in the synthesis process increases the quality of the generated images and exerts more control on their characteristics. Spatially adaptive normalization layers fuse both inputs and are applied to a full-blown generator architecture consisting of encoder and decoder, to take full advantage of the information content in the auxiliary raster data. Our method successfully synthesizes medium (10m) and high (1m) resolution images, when trained with the corresponding dataset. We show the advantage of data fusion of land cover maps and auxiliary information using mean intersection over union, pixel accuracy and Fréchet inception distance using pre-trained U-Net segmentation models. Handpicked images exemplify how fusing information avoids ambiguities in the synthesized images. By slightly editing the input our method can be used to synthesize realistic changes, i.e., raising the water levels. The source code is available at this https URL and we published the newly created high-resolution dataset at this https URL. |
![]() |
C. Robinson, K. Malkin, N. Jojic, H. Chen, R. Qin, C. Xiao, M. Schmitt, P. Ghamisi, R. Haensch, and N. Yokoya, ”Global land cover mapping with weak supervision: Outcome of the 2020 IEEE GRSS Data Fusion Contest,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. (early access), 2021. PDF Quick Abstract Abstract: This paper presents the scientific outcomes of the 2020 Data Fusion Contest organized by the Image Analysis and Data Fusion Technical Committee of the IEEE Geoscience and Remote Sensing Society. The 2020 Contest addressed the problem of automatic global land-cover mapping with weak supervision, i.e. estimating high-resolution semantic maps while only low-resolution reference data is available during training. Two separate competitions were organized to assess two different scenarios: 1) high-resolution labels are not available at all and 2) a small amount of high-resolution labels are available additionally to low-resolution reference data. In this paper we describe the DFC2020 dataset that remains available for further evaluation of corresponding approaches and report the results of the best-performing methods during the contest. |
![]() |
B. Adriano, N. Yokoya, J. Xia, H. Miura, W. Liu, M. Matsuoka, and S. Koshimura, ”Learning from multimodal and multitemporal earth observation data for building damage mapping,” ISPRS Journal of Photogrammetry and Remote Sensing (in press), 2021. PDF Quick Abstract Abstract: Earth observation technologies, such as optical imaging and synthetic aperture radar (SAR), provide excellent means to monitor ever-growing urban environments continuously. Notably, in the case of large-scale disasters (e.g., tsunamis and earthquakes), in which a response is highly time-critical, images from both data modalities can complement each other to accurately convey the full damage condition in the disaster's aftermath. However, due to several factors, such as weather and satellite coverage, it is often uncertain which data modality will be the first available for rapid disaster response efforts. Hence, novel methodologies that can utilize all accessible EO datasets are essential for disaster management. In this study, we have developed a global multisensor and multitemporal dataset for building damage mapping. We included building damage characteristics from three disaster types, namely, earthquakes, tsunamis, and typhoons, and considered three building damage categories. The global dataset contains high-resolution optical imagery and high-to-moderate-resolution multiband SAR data acquired before and after each disaster. Using this comprehensive dataset, we analyzed five data modality scenarios for damage mapping: single-mode (optical and SAR datasets), cross-modal (pre-disaster optical and post-disaster SAR datasets), and mode fusion scenarios. We defined a damage mapping framework for the semantic segmentation of damaged buildings based on a deep convolutional neural network algorithm. We compare our approach to another state-of-the-art baseline model for damage mapping. The results indicated that our dataset, together with a deep learning network, enabled acceptable predictions for all the data modality scenarios. |
![]() |
N. Yokoya, K. Yamanoi, W. He, G. Baier, B. Adriano, H. Miura, and S. Oishi, ”Breaking limits of remote sensing by deep learning from simulated data for flood and debris flow mapping,” IEEE Transactions on Geoscience and Remote Sensing, (early access), 2020. PDF Code Quick Abstract Abstract: We propose a framework that estimates inundation depth (maximum water level) and debris-flow-induced topographic deformation from remote sensing imagery by integrating deep learning and numerical simulation. A water and debris flow simulator generates training data for various artificial disaster scenarios. We show that regression models based on Attention U-Net and LinkNet architectures trained on such synthetic data can predict the maximum water level and topographic deformation from a remote sensing-derived change detection map and a digital elevation model. The proposed framework has an inpainting capability, thus mitigating the false negatives that are inevitable in remote sensing image analysis. Our framework breaks limits of remote sensing and enables rapid estimation of inundation depth and topographic deformation, essential information for emergency response, including rescue and relief activities. We conduct experiments with both synthetic and real data for two disaster events that caused simultaneous flooding and debris flows and demonstrate the effectiveness of our approach quantitatively and qualitatively. Our code and datasets are available at https://github.com/nyokoya/dlsim. |
![]() |
T. Uezato, D. Hong, N. Yokoya, and W. He, “Guided deep decoder: Unsupervised image pair fusion,” Proc. ECCV (spotlight), 2020. PDF Supmat Code Quick Abstract Abstract: The fusion of input and guidance images that have a tradeoff in their information (e.g., hyperspectral and RGB image fusion or pansharpening) can be interpreted as one general problem. However, previous studies applied a task-specific handcrafted prior and did not address the problems with a unified approach. To address this limitation, in this study, we propose a guided deep decoder network as a general prior. The proposed network is composed of an encoder-decoder network that exploits multi-scale features of a guidance image and a deep decoder network that generates an output image. The two networks are connected by feature refinement units to embed the multi-scale features of the guidance image into the deep decoder network. The proposed network allows the network parameters to be optimized in an unsupervised way without training data. Our results show that the proposed network can achieve state-of-the-art performance in various image fusion problems. |
2013 Mar. | D.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan |
2010 Sep. | M.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan |
2008 Mar. | B.Eng. | Department of Aeronautics and Astronautics | The University of Tokyo | Japan |
2020 May | - | present | Lecturer | The University of Tokyo | Japan |
2018 Jan. | - | present | Unit Leader | RIKEN | Japan |
2019 Apr. | - | 2020 Mar. | Visiting Associate Professor | Tokyo University of Agriculture and Technology | Japan |
2015 Dec. | - | 2017 Nov. | Alexander von Hunboldt Research Fellow | DLR and TUM | Germany |
2013 Jul. | - | 2017 Dec. | Assistant Professor | The University of Tokyo | Japan |
2013 Aug. | - | 2014 Jul. | Visiting Scholar | National Food Research Institute (NFRI) | Japan |
2012 Apr. | - | 2013 Jun. | JSPS Research Fellow | The University of Tokyo | Japan |
1st place in the 2017 IEEE GRSS Data Fusion Contest. |
Alexander von Humboldt research fellowship for postdoctoral researchers (2015). |
Best presentation award of the Remote Sensing Society of Japan (2011, 2012, 2019). |
2021 Apr. | - | 2024 Mar. | PI, FOREST (Fusion Oriented REsearch for disruptive Science and Technology), Japan Science and Technology Agency (JST) |
2021 Apr. | - | 2024 Mar. | CI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS) |
2019 Apr. | - | 2022 Mar. | CI, Grant-in-Aid for Scientific Research B, Japan Society for the Promotion of Science (JSPS) |
2018 Apr. | - | 2021 Mar. | PI, Grant-in-Aid for Young Scientists, Japan Society for the Promotion of Science (JSPS) |
2015 Apr. | - | 2018 Mar. | PI, Grant-in-Aid for Young Scientists (B), Japan Society for the Promotion of Science (JSPS) |
2015 Jan. | - | 2016 Dec. | PI, Research Grant Program, Kayamori Foundation of Informational Science Advancement |
2013 Aug. | - | 2014 Mar. | PI, Adaptable and Seamless Technology Transfer Program through Target-driven R&D (A-STEP), Japan Science and Technology Agency (JST) |
2012 Apr. | - | 2013 Jun. | PI, Grant-in-Aid for JSPS Fellows, Japan Society for the Promotion of Science (JSPS) |
Mathematics for Information Science | The University of Tokyo | since 2020 |
Computer Vision | The University of Tokyo | since 2021 |
Remote Sensing Image Analysis | The University of Tokyo | since 2021 |
2015 Sep. | - | 2017 Nov. | Visiting scholar at DLR and TUM, München, Germany. |
2011 Oct. | - | 2012 Mar. | Visiting student at the Grenoble Institute of Technology, Grenoble, France. |
Organizer | IJCAI CDCEO Workshop 2022 |
Organizer | CVPR EarthVision Workshop 2019, 2020, 2021, 2022 |
Chair & Co-Chair | IEEE GRSS Image Analysis and Data Fusion Technical Committee (2017-2021) |
Secretary | IEEE GRSS All Japan Joint Chapter (2018-2021) |
Organizer | IEEE GRSS Data Fusion Contest 2018, 2019, 2020, 2021 |
Student Activity & TIE Event Chair | IEEE IGARSS 2019 |
Program Chair | IEEE WHISPERS 2015 |
Reviewers for various journals and conferences:
- IEEE Transactions on Geoscience and Remote Sensing
- IEEE Transactions on Image Processing
- IEEE Transactions on Signal Processing
- IEEE Transactions on Computational Imaging
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- IEEE Journal of Selected Topics on Applied Remote Sensing
- IEEE Journal of Selected Topics in Signal Processing
- IEEE Geoscience and Remote Sensing Letters
- IEEE Geoscience and Remote Sensing Magazine
- Proceedings of the IEEE
- Remote Sensing
- Remote Sensing of Environment
- International Journal of Remote Sensing
- Pattern Recognition
- Pattern Recognition Letters
- Neurocomputing
- CVPR
- AAAI
- WACV
- ICME