Table of contents
Abstract
Previous works have found that generating coherent raw audio waveforms with GANs is challenging. In this paper, we show that it is possible to train GANs reliably to generate high quality coherent waveforms by introducing a set of architectural changes and simple training techniques. Subjective evaluation metric (Mean Opinion Score, or MOS) shows the effectiveness of the proposed approach for high quality mel-spectrogram inversion. To establish the generality of the proposed techniques, we show qualitative results of our model in speech synthesis, music domain translation and unconditional music synthesis. We evaluate the various components of the model through ablation studies and suggest a set of guidelines to design general purpose discriminators and generators for conditional sequence synthesis tasks. Our model is non-autoregressive, fully convolutional, with significantly fewer parameters than competing models and generalizes to unseen speakers for mel-spectrogram inversion. Our pytorch implementation runs at more than 100x faster than realtime on GTX 1080Ti GPU and more than 2x faster than real-time on CPU, without any hardware specific optimization tricks. Blog post with samples and accompanying code coming soon.
Results
Spectrogram Inversion on Unseen Speakers
Original
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Reconstructed
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
End-to-end text-to-speech examples
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Unconditional Music Synthesis
Original
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Reconstructed
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Sampled
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Music Translation
Example for source domain: Bach Solo Cello
Beethoven accompanied violin
Beethoven solo piano
Original
Mor et al. 2019
Ours
Mor et al. 2019
Ours
1
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
2
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
3
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
4
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
5
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Your browser does not support the audio element.
Samples along Training
50 epochs - 1.35 hours
Your browser does not support the audio element.
100 epochs - 2.71 hours
Your browser does not support the audio element.
200 epochs - 5.42 hours
Your browser does not support the audio element.
400 epochs - 10.84 hours
Your browser does not support the audio element.
800 epochs - 21.68 hours
Your browser does not support the audio element.
1600 epochs - 43.36 hours
Your browser does not support the audio element.
3200 epochs - 86.72 hours
Your browser does not support the audio element.
Ablation
original
Your browser does not support the audio element.
Your browser does not support the audio element.
Baseline
Your browser does not support the audio element.
Your browser does not support the audio element.
l1_observed_no_feat_match
Your browser does not support the audio element.
Your browser does not support the audio element.
l1_observed_space
Your browser does not support the audio element.
Your browser does not support the audio element.
no_dilations
Your browser does not support the audio element.
Your browser does not support the audio element.
no_group_disc
Your browser does not support the audio element.
Your browser does not support the audio element.
no_multiscale_disc
Your browser does not support the audio element.
Your browser does not support the audio element.
no_patch_gan
Your browser does not support the audio element.
Your browser does not support the audio element.
no_weight_norm
Your browser does not support the audio element.
Your browser does not support the audio element.
spectral_norm
Your browser does not support the audio element.
Your browser does not support the audio element.