Microsoft yesterday launched a range of Copilot Plus PCs that include new AI features built right into Windows 11. Behind the scenes, the company now has more than 40 AI models running on Windows 11 thanks to the new Windows Copilot Runtime Which will also allow developers to use these models for their applications.
In today’s Microsoft Build, the company provides a lot of details on exactly how the Windows Copilot Runtime works. The runtime includes a library of APIs that developers can leverage for their own applications, with AI frameworks and toolchains designed for developers to ship their own on-device models on Windows.
“The Windows Copilot library consists of ready-to-use AI APIs such as Studio Effects, Live Captions Translations, OCR, Recall with User Activity, and Phi Silica, which will be available to developers in June,” explains Pavan Davuluri, head of Windows and Surface.
Developers will be able to use the Windows Copilot library to integrate things like Studio Effects, filters, image blur, and other features into their apps. Meta is adding Windows Studio effects to WhatsApp, so you’ll get features like background blur and eye contact during video calls. Even Live Captions and the new AI-powered translation feature can be used by developers with little to no code.
Microsoft showed off its Recall AI feature yesterday, allowing Copilot Plus computers to document and store everything you do on your computer so you can recall memories and search through a timeline. All of this is powered by a new Windows semantic index that stores this data locally, and Microsoft plans to let developers build something similar.
“We will make this capability available to developers who use the Vector Embeddings API to create their own vector store and RAG within their applications and with their application data,” says Davuluri.
Photo: Alison Johnson/The Verge
Developers will also be able to enhance Windows’ new paging feature by adding contextual information to their applications that feeds the database that powers the feature. “This integration helps users pick up where they left off in your app, resulting in improved app interaction and seamless flow for users between Windows and your app,” Davuluri says.
All of these improvements within Windows for Developers are the first building blocks for more AI-powered applications as well as new Arm-powered systems and NPUs coming from AMD and Intel soon. As Microsoft builds the platform for developers to create AI applications for Windows, it is now banking on this being an important part of the next decade of Windows development. On stage at Build today, Davuluri stood in front of a slide that read “Windows is the most open platform for AI,” indicating how important this moment is for Microsoft.
“Friendly food geek. Communicator. Hipster-friendly creator. Bacon evangelist. Zombie nerd. Pop culture advocate. Beer aficionado.”