universal interpretation service fairbluePlus
With the experience of providing an HTML-5 client with translation features for BBB, and with the new plugin option for BBB 3+ #160 we are sketching here the option of building a generic translation service, that could be combined also with other video or on-site conferencing systems. Let's call it fairbluePlus
and describe the user stories here briefly:
Translator UX
The interpreter would simply watch and listen to the video conference or the live stream by using a headset. The microphone would connect to fairbluePlus
, which offers its own interface (to be opened in a second browser window or as an app). Here the interpreter would enter a unique code (related to the event) and then choose which language s:he will translate to.
Listener UX
A participant would open fairbluePlus
, enter the event code and choose the language. If the same device / browser is being used, the floor (original stream / conference) audio needs to be muted (we need to check how that could be achieved). Alternatively the listener connects to fairbluePlus
via a second device (e.g. smartphone) and simply uses its headset.
Add translations to recording
A recorded conference contains the original audio track. We would offer a feature, to upload that to fairbluePlus
, which then would add additional tracks of the translated languages (could likely be achieved with ffmpeg, we need to check start timestamps to be in sync). The enhanced recording will be then made available for download or publish e.g. on a PeerTube instance (for which we have a transposer plugin for automatic subtitling + translations).
AI translation
Live captioning could be achieved with an AI model (Whisper, kaldi) through a transposer queue, which could be then translated with fairtranslate and then even converted to separate language tracks with text-to-speech. This may require beefy hardware if done live. The quality of translation may not be the same as with human translators, however it might fit well for some sort of events or some additional languages.