Abstract
Neural coding, including encoding and decoding, is one of the key problems in neuroscience for understanding how the brain uses neural signals to relate sensory perception and motor behaviors with neural systems. However, most of the existed studies only aim at dealing with the continuous signal of neural systems, while lacking a unique feature of biological neurons, termed spike, which is the fundamental information unit for neural computation as well as a building block for brain–machine interface. Aiming at these limitations, we propose a transcoding framework to encode multi-modal sensory information into neural spikes and then reconstruct stimuli from spikes. Sensory information can be compressed into 10% in terms of neural spikes, yet re-extract 100% of information by reconstruction. Our framework can not only feasibly and accurately reconstruct dynamical visual and auditory scenes, but also rebuild the stimulus patterns from functional magnetic resonance imaging (fMRI) brain activities. More importantly, it has a superb ability of noise immunity for various types of artificial noises and background signals. The proposed framework provides efficient ways to perform multimodal feature representation and reconstruction in a high-throughput fashion, with potential usage for efficient neuromorphic computing in a noisy environment.
Original language | English |
---|---|
Pages (from-to) | 1935-1946 |
Number of pages | 12 |
Journal | IEEE Transactions on Neural Networks and Learning Systems |
Volume | 33 |
Issue number | 5 |
Early online date | 19 Oct 2021 |
DOIs | |
Publication status | Published - May 2022 |
Bibliographical note
Funding:This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFB1002503 in part by the National Science Fund for Distinguished Young Scholars under Grant 61925603 in part by the Zhejiang Laboratory under Grant 2019KC0AD02 and Grant 2019KC0AB03 in part by the Ten Thousand Talent Program of Zhejiang Province under Grant 2018R52039 in part by the Royal Society Newton Advanced Fellowship of U.K. under Grant NAFR1-191082 in part by the CAAI-Huawei Mindspore Open Fund under Grant CAAIXSJLJJ-2020-024A and in part by the Fundamental Research Funds for Central Universities under Grant DUT21RC(3)091.
Keywords
- Cross-multimodal
- decoding
- denoising
- neural spikes
- reconstruction
- spatio-temporal representations