PATENT DROP: Amazon shares its smarts

Plus: Microsoft’s sustainability play; Google may drop hotwords

Happy Thursday and welcome to Patent Drop! 

Today, we’re looking at a filing from Amazon to help AI models study together; Microsoft’s tech that keeps an eye on how much energy its computers are sucking up; and Google’s plan to get rid of hotwords on its voice assistants. 

Let's dive in. 

Amazon shares the love 

Knowledge is power. According to Amazon, that extends to machine learning models, too. 

The company wants to patent a system that transfers the knowledge of one machine learning model to another. Essentially, Amazon’s patent allows for the knowledge of one machine learning model that’s completed its training to be transferred to another model that’s undergoing training, thereby speeding up the training of AI models. 

“Generally, training a machine learning model, such as a reinforcement learning model, from scratch requires a huge amount of time and computing resources,” the filing notes. “Thus, it is desirable to have techniques to improve the learning speed of a machine learning model.” 

Amazon outlines two kinds of transfers: a representation transfer and an instance transfer. In a representation transfer, the knowledge that’s given (such as how to perform a task or follow a policy) is based on the similarities between the “teacher model” and the “student model.” Essentially, this system will distill the knowledge transferred to only the relevant information depending on those similarities. 

Alternatively, in an instance transfer, the conveyed knowledge comes from “sampled trajectories” of the teacher model, or basically exact examples of its own inputs and outputs. The system automatically switches between doing representation transfers and instance transfers depending on the circumstance. 

Amazon noted that this method could be applied to various AI training processes, including neural networks used for image processing and speech recognition. 

Photo via the U.S. Patent and Trademark Office.

Knowledge transfer is far from a new technique in the machine learning community, said Rijul Gupta, co-founder of AI communications company Deep Media. Gupta gave the example of an AI model that can play chess versus one that can play checkers: The games are somewhat similar, so transferring knowledge between them wouldn’t be difficult. 

Most of Amazon’s patent is quite broad, Gupta said, so it’s unlikely that it would get approved without a decent amount of pushback and narrowing. In its current form, granting it would “essentially cripple everyone else's machine learning development,” Gupta said.  

But the filing isn’t entirely old news. One piece that’s novel is its ability to automatically choose which teacher model is the best fit for training the student model without having to actually go through training, said Gupta. Typically, this process is a costly and tedious process that’s done by hand. 

Secondly, Amazon’s patent discusses a way to bridge the gap between teacher and student models that are more distinct from each other. For example, rather than making the chess algorithm play checkers, Gupta said, it could “take an algorithm that's built to play chess and make it play Fortnite.”  

Using transfer techniques, he said, could speed up machine learning development by as much as 95%, helping it catch companies like Google, Microsoft and OpenAI. Meanwhile, securing the patent in a more narrow form could send the company’s AI development soaring, making it “very difficult for any of Amazon's competitors to catch up to them.”

Amazon has thrown its hat into the AI ring in more ways than one: The company pledged $100 million toward a AWS Generative AI Innovation Center, put together a new team focused on large language models, and has filed for multiple AI-related patents. As it stands, though, it remains a step behind the AI heavyweights. 

That said, Amazon’s real strength lies in AWS, said Gupta, with tools like SageMaker and Bedrock, as well as Inferentia and Trainium, its custom in-house chips for training AI. Plus, as AWS CEO Adam Selipsky told The Verge in a podcast last week, “Cloud and AI are not two different things. They’re really just two of the many faces of the same thing.” 

“There's an old saying that when there's a gold rush, you should sell shovels,” said Gupta. “And Amazon, right now, is selling shovels.” 

Microsoft micromanages your energy

Microsoft wants its computers to stop blowing through so much energy. 

The company filed a patent application for a “sustainability-aware” system for device behavior management. The system would determine the actions taken by a computing device by considering their impact on the energy grid. 

Microsoft’s system first obtains what it calls “sustainability information” associated with an energy grid, such as how carbon-intense the action will be at a given time, and a variety of environmental factors, such as weather, temperature, and energy demand on the grid. It also collects and stores historical sustainability information. 

With this information (and in some examples using an AI model), it churns out a sustainability forecast, and uses that forecast to manage device functionality by holding off on performing certain actions until they’d have a “comparatively lower environmental impact.” The system also takes into account the tasks that need to be done and their priority when deciding what actions to put off. Microsoft said this avoids energy consumption during peak times which would have a higher environmental impact. 

In practice, this could show up on the user end as a task scheduler requesting to put off downloads, software updates, backups or charging a device’s battery. The sustainability forecast is updated periodically, and may change if the location of your device or IP address changes. 

Along with reducing strain on the grid, Microsoft said its tech can manage behavior in a way that lengthens hardware lifespan, thereby avoiding e-waste and reducing carbon emitted in the manufacturing of a new device.

Photo via the U.S. Patent and Trademark Office.

Microsoft is no stranger to environmental goals. On the patent side, the company filed an application for a carbon capture system that works to reduce emissions from its data centers. And carbon capture tech is just a piece of the company’s ambitious goal to go carbon-negative by 2030, and remove the equivalent of its historical emissions by 2050. 

While the patent details delaying actions such as downloading, charging and system updates that are typically associated with personal computers, inventions in patents are often intended to cover a far wider scope than the examples within them. With that in mind, Microsoft’s tech could be put into practice in data centers, said Dr. Dan Stein, founder and director of climate giving consultancy Giving Green.

Microsoft’s tech could hold off certain data center operations until times of day when clean energy is readily available, said Stein, whether it be from the energy grid or from its own sources of renewable energy. Given that data centers accounted for roughly 1.3% of energy consumption globally in 2022, according to the International Energy Agency, Microsoft keeping its own data centers in check in this way could help it make progress toward its broader goals. 

“The idea is, if you could anticipate when there's going to be clean energy, and there are certain highly computational intensive tasks that you have a choice of when you're going to them, you would just do them at the right time,” said Stein. 

The question of how much this could help is still up in the air, said Stein, as some actions likely can’t be delayed depending on safety and user demand times. However, Microsoft’s filing is one example of a type of energy conservation that’s being discussed and implemented far beyond tech companies, extending to household appliances, utilities and basically every other operation that uses up energy, said Stein. 

“The energy nerds call it demand shaping,” said Stein. “It’s getting the demand to be when we have access to clean electricity.” 

Google drops it like it’s hot(words) 

Google may want to get rid of hotwords once and for all. 

The company wants to patent a system for “hot-word free adaptation” of automated assistant functions. Rather than relying on a wake-up phrase such as “Hey Google,” the company’s system basically attempts to pick up on social cues to determine whether or not a user is addressing it, relying on lots of user activity monitoring to do so. 

Here’s how it works: First, this system periodically collects a whole bunch of sensor data, including: a user’s gaze direction; distance from the device; mouth movement; voice activity; facial, speaker and presence recognition; body pose relative to the device; and gestures. If no user activity is detected, the system may stop collecting data and go dormant to save on computational resources. 

Using this data, the system comes up with “occurrence and/or confidence metric(s)” that determine whether or not a user intends to interact with the device. These metrics will then control the system’s functionality. 

For example: If you look at the device and speak a command, such as “turn on the lights,” the device will complete that request. But if you’re having a conversation and happen to glance toward the device, it may say something along the lines of “Looks like you're talking to the Assistant, look away if you don't want to.”  In another example, gestures can indicate certain actions, such as a thumbs up to increase volume, or a “stop” gesture to stop an alarm or timer.  

The metrics of occurrence and confidence are determined using neural network-based models, so they improve over time as they correctly (or incorrectly) determine that a user is, in fact, talking to the device. 

The company argues in the filing that hotwords like “OK Google” are “unnatural prescribed phrases that are awkward to speak,” and getting rid of them will cut down friction. 

Photo via the U.S. Patent and Trademark Office.

Google has long been working on ways to make its voice assistant technology better at listening. The company has sought plenty of patents related to speech recognition, including tech to enable natural conversations between users and their speakers, a system to correct what a device mishears, and a way to permit (or restrict) device access depending on who’s talking. 

Patents aside, Google also introduced a feature called Quick Phrases to certain devices, including Pixel smartphones and some voice assistant devices, which allows users to assign specific phrases to certain actions. For example, a user can command its device to “stop” or “snooze” when an alarm is going off, or can say “accept” or “decline” on an incoming call, without having to yell “Hey Google” beforehand. This filing, however, suggests expanding these capabilities far beyond a few fixed phrases. 

Google’s patent also brings up a quandary that comes up with almost all new consumer tech devices this day in age: How much privacy are users willing to give up for the promise of convenience? 

In lieu of hotwords, Google’s proposed system collects and uses a whole lot of user data, monitoring its owner’s environment nearly constantly whenever they’re in the room. The filing itself brings up one benefit of hotwords: “to preserve user privacy … a user must often explicitly invoke an automated assistant before the automated assistant will fully process a spoken utterance.” 

Whether or not users will trust a system like this depends on consumer confidence in the devices and the tech companies themselves. And according to NPR’s Smart Audio Report from last year, 47% of survey respondents were uncomfortable with the fact that their smart speakers were “always listening,” and 52% were concerned that bad actors could gain access to these devices. Around 58% surveyed trusted the company that makes their smart speaker. 

Another consideration: Google is not the only company looking to cut down hotwords. Apple recently announced plans to get rid of the “Hey” in “Hey Siri,” and Amazon allows users to control their speakers with either “Hey Alexa” or just “Alexa.” Getting to that point, though, was likely not an easy engineering feat, so whether or not Google’s hotword-less technology will actually work is to be determined. 

---

Extra Drops

We‘re not done yet. 

• EBay wants to make sure you’re styled properly. The company filed a patent application for a system to modify “three-dimensional garments using gestures.” 

• Apple wants to know when you’re going in for a hug. The company is seeking to patent a system for “skin-to-skin contact detection,” which uses sense circuitry to detect touching or movement “between a first body part and a second body part.” Talk about too close for comfort. 

• Sony wants to make racing games more accurate. The company is seeking to patent an AI-based system for creating “driving lines in racing games,” in order to provide players with adaptive “driving aides.”

What else is new? 

• Global smartphone shipments could drop to their lowest in a decade, with Apple potentially notching the top spot among competitors for the first time, according to Counterpoint Research. 

• New York City banned TikTok on government-owned devices, arguing that the app “posed a security threat to the city’s technical networks.”

• Microsoft is still struggling to make Bing a thing, despite adding AI capabilities. The company’s search engine reportedly holds just 3% of the search market worldwide.  

Have any comments, tips or suggestions? Drop us a line! Email at admin@patentdrop.xyz or shoot us a DM on Twitter @patentdrop.