Can StarGANv2-VC be utilized as an intelligent audiofilter for upscaling phone conversations to be less distorted?
The StarGANv2-VC model is able to convert speech of an unrecognized input speaker. As such, we tested whether the StarGAN-v2 VC model could clean distorted audio out of the box. The utterances of one of the speakers can be heard below.
| Original Audio Recording | Same Clip with Distortion | StarGANv2-VC, Mapping (Distortion → Original) |
|---|---|---|
As can be heard, the StarGANv2-VC model cannot remove the introduced distortion, when it has not been trained on distorted noise.
We added the constructed, distorted versions of the data, and trained the StarGANv2-VC model on this in addition to the original training data. We then replaced standard convolutions in the StarGANv2-VC with depthwise separable convolutions which resulted in a 63 % total reduction in the number of trainable model parameters, to conform with the aims of augmentedhearing.io to have a model better fit for mobile devices. Listen below to see if you can hear a difference! This second model version is known as the “reduced” model.
| StarGANv2-VC, Mapping (Distortion → Original) |
StarGANv2-VC trained on distorted data, Mapping (Distortion → Original) |
StarGANv2-VC (reduced) trained on distorted data, Mapping (Distortion → Original) |
|---|---|---|