Microsoft Patent Filing Describes AI That Rewrites Livestreams in Real Time

A new Microsoft patent filing describes an artificial intelligence system that can listen to a livestream and rewrite what speakers are saying as it happens. Patents do not always become real products, but filings show strong industry interest in real-time AI editing.

The patent filing was published in late October and uncovered by Patentlyze. It outlines a system that listens for specific words in a broadcast audio track and swaps them out using details from a viewer's profile, such as their age, location, and native language. Microsoft states that the tool is meant to clean up presentations by removing pauses and filler words, but it could also censor bad language and swap words based on web searches.

The filing suggests the system could adjust lip movements in the video frames between edits to hide the changes. This feature means viewers might watch altered livestreams without knowing the video was changed. Because scammers already use real-time face-swapping tools, altering spoken words in a live broadcast could create new risks for fraud and misinformation.

A separate part of the patent describes using the technology to translate live video. While platforms like YouTube already offer AI audio translation, Microsoft's method would translate livestreams while matching the speaker's lip sync. Unlike the word-swapping feature, this translation tool keeps the original recording as a true reference.

This filing follows other recent industry experiments with real-time generative AI. Microsoft filed a claim in July to rewrite video game code on the fly based on player behavior. Sony has also filed patents for AI that predicts upcoming video game frames to reduce lag or controls game difficulty.

Summary

Microsoft's patent highlights how companies are exploring real-time AI editing for broadcasts and games. While the technology could improve translation and clean up audio, it also raises concerns about hidden edits and misinformation.