iPad AI Tutor Demo Opens the Way to an Amazing New World for Students

If you haven’t yet watched yesterday’s OpenAI event, I highly recommend you do so. The big news was that the latest GPT-4o model works seamlessly with any combination of text, audio and video.

This includes the ability to “show” the GPT-4o app a screen recording that you take from another app – and it’s this capability that the company showed off with a fairly large iPad AI Tutor demo. crazy…

GPT-4o

OpenAI stated that the “o” stands for “omni”.

GPT-4o (“o” for “omni”) is a step towards much more natural human-machine interaction: it accepts as input any combination of text, audio and image and generates any combination of text, audio and image output.

It can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time(opens in new window) in a conversation (…) GPT-4o is particularly better in terms vision and audio. understanding compared to existing models.

Even the vocal aspect is a big problem. Previously, ChatGPT could accept voice input, but it converted it to text before using it. GPT-4o, on the other hand, actually understands speech and therefore skips the conversion step completely.

As we noted yesterday, free users also benefit from many features previously reserved for paid subscribers.

AI Tutor Demo on iPad

One of the capabilities demonstrated by OpenAI was GPT-4o’s ability to watch what you are doing on your iPad screen (in split-screen mode).

The example shows the AI ​​giving private lessons to a student with a math problem. You can hear that initially GPT-4o understood the problem and wanted to fix it immediately. But the new pattern can be interrupted, and in this case it was asked to help the student solve it himself.

Another capability observed here is that the model claims to detect emotions in speech and can also express emotions itself. For my taste, it was rather overdone in the demo version, and that’s reflected here – the AI ​​is perhaps a little condescending. But everything is adjustable.

Indeed, every student in the world could have a private tutor with these kinds of capabilities.

To what extent will Apple integrate it?

We know that AI is the main focus of iOS 18 and that it is finalizing a deal to bring OpenAI features to Apple devices. While at the time this was described as being for ChatGPT, it now seems quite likely that the actual deal is for access to GPT-4o.

But we also know that Apple is working on its own AI models, with its own data centers running its own chips. For example, Apple is working on its own way to let Siri make sense of app screens.

So we don’t know exactly what GPT-4o capabilities the company will bring to its devices, but this one seems so perfectly Apple that I have to believe it will be included. It’s really about using technology to empower people.

Image: OpenAI. Benjamin Mayo contributed to this report.