NeuroTalk: Voice Reconstruction from Brain Signals


Contents


Section 1: Video - Experimental Paradigm <Top>




Section 2:Audio Demo<Top>

Reconstruction - Participant 1 <Top>

Word : "Hello"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Help me"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Pain"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Water"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Yes"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Reconstruction - Participant 2 <Top>

Word : "Hello"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Help me"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Pain"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Water"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Word : "Yes"

Original Audio Mel + Vocoder
Spoken EEG Imagined EEG

Failure cases<Top>

Word : "Thank you"

Original Audio Mel + Vocoder Spoken EEG
Imagined EEG (CER=0)
"THANK YOU" 		Imagined EEG (CER=50)
"THANKEE" 		Imagined EEG (CER=90)
"AVE"

Section 3: Model <Top>

Overall NeuroTalk frameworks


Overall frameworks.


The architecture details


The architecture details in (a) generator, (b) discriminator, and (c) MRF module.


Section 4: Code <Top>

Code is available at https://github.com/youngeun1209/NeuroTalk

Section 5: Data and materials availability<Top>

Sample dataset is available at https://github.com/youngeun1209/NeuroTalk