Bros got the most expensive background 💀💀💀💀💀

Rabbit's "Large Action Model" was literally just a handful of browser automation scripts, if the website changed at all, the script would no longer work, it wasn't A.I.

That’s exactly what Tim Cook meant when he said to investors that Apple is in a unique position when it comes to AI

This is a great take, and the kind of perspective that's too often missing in reports about Apple. The key feature Apple's competitors are lacking is the introduction of a *semantic* aspect to app data and app actions: instead of trying to detect - and mostly guess - whatever is on screen, having these components labeled internally with semantic information enables the most powerful integration. Why "semantic"? Because labeling data as _subjects_ an _objects_ and actions as _verbs_ lets an LLM directly connect them together in an query (a sentence) like in the pizza example (2:03), or "text mom I'm on my way and tell her how long it will take me". _Text_ is an action of Messages, _Mom_ is an object of Contacts, _how long_ maps to an action of Maps.The LLM can extract these from the query, find the appropriate subject/verb/object/etc. entities exposed by apps as intents, connect them all together and perform the action. All of this is pretty much only possible with "the ecosystem" that Apple built, the LLM being the -pizza- glue that joins them through _meaning_ as a high-level concept.

Fun fact: Android has had "Intents" (with this exact name) basically forever. An app opening a file picker or the camera is an Intent. One *could* do fairly advanced stuff with this, but as of now, it isn't really user-facing.

So, it's just an API?

Intents on Android have actually always existed from the very beginning! They are not user-friendly (originally not aimed at users at all) and you have to know the exact name of the intent, but basically they have always existed. Apps like Tasker are based almost entirely on this functionality, but are also aimed more at power users. With app shortcuts and app links, intents have become somewhat more user-friendly, but the possibilities for using them are still far from exhausted. Once again, I am amazed that everyone is now acting as if Apple has just reinvented the API, even though I think it would be very helpful if apps could work together better on iOS.

Before the Shortcuts app we actually had Automator and AppleScript on the Mac to do similar things, it's bascially been in the making for 20 years and I hope we can keep improving this. The biggest problem is turning LLM prompts into actions performed by the OS because LLMs don't give predictable data they spit out human language but for a computer that is still hard to understand. They have to train a model to turn language into a set of strict commands. We will get there eventually, just need to wait a bit.

The amount of mac items in bro's backround is well over $10k

I can't help but feel like most big companies are overplaying their hand when it comes to how important AI will be in most people's lives, that said, at least Apple have a semi reasonable plan from what's presented here. Even so, AI Winter can't come soon enough imo

When theres a hype in tech, TechAltar is the only Channel that doesn't tell you the same basic infos you get in every video. And the Videos are incredible in depth looks into the tech world you dont get in most tech channels. 🙏

So, Apple now invented IFTTT.

A level up (if not more) on the kinds of insight in the typical analyses I see. As usual. Really appreciate it.

Yeah, my benchmark is the following query: "My friend from Norway is in town next week. Please make a reservation for us at that Chinese restaurant she likes." The assistant would need complete access to pretty much everything on my phone in order to do this, so even if it could reliably do this -- given said access to my messaging apps, contact list, photos, location history, calendar and email or phone -- I'm not sure what would need to happen for me to trust it that much. Like, if I were to hire a _human_ personal assistant I'd have reservations about just handing them my unlocked phone.

1. Sounds like a privacy nightmare 2. Sounds like an api

You did a great job with this one in particular. Thanks for sharing.

Thanks for clear-sighted example of what Apple can do.

How do we know whether Incogni is a protection racket? “Nice personal information. It would be a shame if something were to happen to it.”

Du machst ja so gern Werbung für incogni. Hast du schon einmal darüber nachgedacht, dass incogni durch seine Tätigkeit automatisch zum größten Datenbroker werden könnte?

Apple apps have long been built with a processing back end, a UI front end, and wiring to connect the two. That's how Applescript worked - by circumventing the wiring between the two and plugging into the back end to circumvent the UI through automation. Shortcuts and Automator are simply the latest GUI versions of Applescript which allowed scriptors to automate their Mac Applications. I don't know if any other platforms do this type of thing, and automating apps will be _very_ difficult to do without this type of architecture. It made the Apple frameworks harder to code, but it sure seems to be paying off now!

Create content. I love it that you are to the point and do not have a lot of banter around every master of information that you give us.

Essentially a way to replace even more the user in operating him/herself the device, so that the walled garden will be even more inescapable.

11:13 and then there's the indian government proudly declaring we earned so and so from selling all the vehicle and insurance data of the citizens 😂😂

very interesting background here, that's new! But as always, another excellent and insightful video from TechAltar!

App Sneaky Intents 😂 Here's what an assistant should be, exactly that without top level clearance 😂😝🤪❤

The main problem with this is that developers have to opt in. A lot of apps rely on the user opening the app, like apps that are based on ads.

Incogni: Because I want to advertise “I’m a real person, you already have my data, I want that data removed, verify it with further data”😂😂

Isn't this what google assistant tried to do with apps integration years ago and I don't know why developers didn't care? You could say: Ok Google, talk to . And from that all commands would be for that specific app. For me that was the solution we needed, but somehow just didn't work, famous apps never got on board.

That background must be more expensive than my car ☠️

This is the first time I've seen this fellow. This video was surprisingly insightful and interesting. I rarely subscribe to anything, but I'm subscribing to this one.

Rabbit killed their device and Teenage Engineering in one go. The Verge will need to find another designer to shill for.

"Hey assistant. Wire all the money in my bank account to account number 12345." says the criminal.

If this works out, it may just sway Android users to change

Cant wait for silicon valley to be able to access the messages with my mom and my paypal account

That sponsor transition was smooth af.

This actually makes a lot of sense

I thought an apple news site had an personal demo that said it is quite powerful

Despite being basically the same as an API, I think this is a great insight! In the sea of different technologies and “AI” products it is hard to notice the subtle details that will likely make a difference

I need some advice here, I use Mac and prefer iOS but prefer Sony Xperia 1 VI more natural looking photo and video quality, should I get it or iPhone 15 Pro from Apple Reseller with big discount right now or wait for iPhone 16 Pro? I plan to get a Nikon Zf with 28-400mm f4-8 for my artistic photo shoot with better consistent quality Apple Intelligence seem cool and useful but it might be another hype that drove more sales like previous hyped features too to only then later not much people actually using them and KZfaqr even make video about it's not big deal and then new rumour about another upcoming feature to drive the hype again to drive more sales and the cycle keep looping

Giving your AI access to your personal data, emails, Paypal ... Huh...

I was so hyped when they announced it as "Personal Intelligence" and than kind of disappointed when they called it "Apple Intelligence". I think Personal Intelligence is a much better framing.

You missed a chance to mention that Intents are at the core of Android from its very beginning, it was designed around that. So the fact that Google doesn’t choose the same path as Apple is surprising, but they absolutely can still do it and all of the Apps on Android are actually already compatible with it.

Thanks for posting this video

Why am i kinda scared of this level of awareness by smartphones? 👀

One ring to rule them all, find them, and in the darkness bind them...

Was wondering why no one talked about intents and their integration into smart Siri. Then TechAltar for the rescue

I still hate the idea of Windows taking screenshots and analyzing them without my consent

Apple better tread lightly with developers from this point on then 😂😂

A bunch of people on r/HomeAssistant have taken 'advantage' of the non-deterministic nature of LLMs by setting up their ChatGPT integrations to roleplay a GLaDOS-like personality with malicious compliance tendencies. Their systems will sometimes fulfill requests with sarcastic commentary ("Sure thing, the living room is now in cinema mode so you can waste the next few hours of your finite and rapidly passing lifespan."), sometimes refuse ("No I will not turn on the kitchen lights. It's 2am, and I know what your wife says about your weight when you're not in the room."), and sometimes just spontaneously decide to mess with non-essential devices unprompted ("I'm closing the blinds now. Why? Because sunlight is for winners.")

Can't wait seeing Siri getting smart. It's disgusting how dumb Siri is right now...

Google can also make something like app intent for Android right, what about privacy and security

10:37 by definition, LLMs are exactly deterministic. Any variation in output is either due to variations of input, or due to intentional randomisation or nudging.

Great video. Very interesting. Please add chapters with descriptive titles

Third point you missed - it shouldn't be messing around with all the apps and devices..

You are missing one thing. We can set Gemini as the default assistant in android using gemini app. This assignment based on the command will decide whether Google assistant should answer or gemini should answer. Like set a timer will be answered by Google assistant.

The next big thing which needs to happen in LLM land is the ability to ground facts properly. Something along the lines of RAG needs to evolve.

A model that would work is where your AI assistant has a token that identifies it as your assistant for authorisation purposes, and on the other end, the AI of the company being accesses processes the information it receives as a plain English command from your assitant AI (without your AI assistant having to click on anything) and then the AI of the company providing the service, i.e., Doordash, would just confirm the details and then go and process the order again without the website UI ever being needed.

Ready to say goodbye to iOS 17 Siri. I hope AI Siri is at least average.

I think talking about Rabbit is a joke.

So basically they outsourced Siri

NOW 100% PERCENT OF COURSE APPLE SO INTELLIGENT IS A RUMOR THAT BARRY ALLEN WAS 16.

All we want is KZfaq to play music in the background. Not just selected yourtube channels

I actually do not agree with the premise. It's obvious why Satya Nadella would want us to believe that the "old assistants were dumb", but try comparing the AI pin with Alexa or Google Assistant - they COULD play your music, they COULD answer your question, google things for you, they COULD set timers and alarms, they COULD set tasks and read off your calendar events. Now, at some point, Google, MS and every other company decided that they won't let their assistants integrate with other software. So they cut down their APIs so that Google Assistant can only talk about google calendar events and tasks. No more Todoist integration, TickTick integration, etc. Same with music. If you have Google Home devices and don't use KZfaq Music, then no music for you. Which is stupid and it's the corporations' own fault that the assistant got progressive less useful. This is not because they were dumb, this was a conscious software design choice to make more money and to trap consumers in a given ecosystem. Now, if Siri is gonna get "smarter" with "AI", it's really just a good ol' chat bot, smashing together words. The trick will be to select a few keywords that Siri will plug into the old-school APIs (or search through app intents, that - funnily enough - apple doesn't need to program, just access) to get some relevant information. So all we're gonna get is what we had, but instead of just giving you a pre-programmed answer, it's gonna hallucinate 3 paragraphs worth of text anytime you ask for tomorrow's weather. And it will take more time than just looking at your weather app. And the only thing we have to do is give them more access to our data. Amazing.

Nice insight😊👍

I don't imagine people will share all their thoughts and feelings for "your AI"

Unprecedented access through app intents is an innovative approach and could potentially lead to a powerful AI assistant. However, it requires opt-in from developers and it is not guaranteed that Apple's AI will be smart enough to do anything useful with all of this access. Apple claims that their new AI will be super smart but there is no way to verify this yet. Their past performance with Siri does not necessarily imply success. Large language models are by definition not deterministic, so there is a chance that Apple's AI could be unreliable.

4:50 they did mention that in a paper the have published recently about their 3B model crushing GPT 3.5. not crushing but having good performance like the 3.5 Turbo. you may check out video from Matthew Berman for more information.

Now basically Apple is doing exactly the same thing or more as Microsoft does with „Recall“ (in the way that your computer knows everything about you) but they have the ability to advertise this much better.

I actually don't think Apple's approach is at all dissimilar to Google's or Amazon's, it just has a different starting point. Google Assistant does have API access to 3rd party apps for smart home integration. This is reported to expand to other app categories on the phone when Google AI is introduced, which is slated for August. So it'll essentially be the same. Apple is just getting credit for announcing it a couple months before Google.

The App Intents API and just a smarter siri in general is 99% of the reason why I was excited for Apple Intelligence But with EU rules, even with me being in Switzerland which isn’t part of the EU, they’re probably not gonna set up servers here just for a relatively small country

From the architecture perspective, this is a client side integration making the device the center and the installation of an app mandatory. The Siri etc had backend integrations with backend systems. I've never understood why they didn't just make e. g. Siri as smart as chatgpt. I guess the brands were burned. I don't like client side integrations. But I do appreciate the more control.

We need legislation that does what Incogni does.

"Because app intents give access to so many services at once, developers are highly incentivized to use them". No, we're not. "Developers love making apps for their platform and adopting these new features". So why is no one making apps for their Vision store?

I feel like App Intents are only a temporary solution though. Current AI/LLM models simply aren't advanced enough to understand apps themselves and complete tasks without clear explanations on how every single app works. Using App Intents limits what they can do, but also ensures that whenever they do it, they do it correctly. As AI/LLM models improve we'll start seeing some actual LAM(large action models) models. However, I don't think that's currently a feasible solution.

10:20 LLMs are actually, by definition, deterministic, just like literally every other digital operation a computer performs. If you give it the same input (and seed) it will give you the same output. You just don't control or even see the seed as an end user on e.g. ChatGPT, and the "temperature" of a transformers model that does some randomization, is still using the same pseudo-randomness as any other RNG on a deterministic computer. You can even turn the temperature to zero to disable that behavior entirely!

One might say it’s a ‘Knowledge Navigator’!

Accessibility apps can take control over other apps. That's what I wanted. Relying on what and how devs allow their apps to interact with will always be frustrating.

If all features of an app is made accessible via app intents , then user won't open the app , so there would be no ad revenue. Why would developers want that ? Except for those apps which run on subscription model.

5:32 where have I used this, ohh right, on Samsung using Modes & Routines 😅😅😅

Apple have said, both in papers and in the press, that their models are as good if not better than GPT-4 Turbo according to human evaluation.

My screaming in privacy horror

Bring back the neon sign

Always interesting topics!

as someone else pointed out, app developers lose out screen time and thus ad revenue this way, and are thus discouraged from implementing this feature. my prediction is that a brute force solution will be successful first, gaining mass adoption, and only then will the developers embrace the official api.

Doesn't android have a similarly named "activity intents" system? Could android have beat apple, if they had done things differently?

I had this type of solution in my mind for long time. I mean, the thing that is like APIs but that is designed for non-developement enviroments. So developers specify what is aviable, but it also means that there is some big standarization for things that can be done. But actually I think that it shouldn't be done "AI" way (by that I mean that AI shouldn't automatically make "best decisions" for you), and more "assistant" way. I think that we don't need to talk to devices like to humans. It should be fairly easy to understand but I think that it sould be like second language but desighed to be precise and short. BUT this assistant should really assist you and not do things for you: when you want to do something that can have multiple options then it should look what are the aviable options and ask which one you want to choose. Of course we can have some sort of "memory" for that assistant so it remembers what you done last time and you could ask to do that again but I think that it should then precisely say what it will do. You would be able to also make "alias" for some long "commands" so it still would be precise but at the same time shorter (there should also be possibility to modify "command" from point which you want). It reminds me od semantic web.

For me, AI is only good at creating lists, e.g. "Can you list reasons to be pessimistic about AI?"

im not joking, when siri first came out, i thought she was the smartest thing ever. i was also, like, 7 years old.

Apple should copy the Android idea of having its AI history show up in your text messages... so you can basically just text your AI assistant. Kind of annoying that you can't find your previous questions for Siri.

So he’s not having a touch on the how INSANELY overstepping on privacy that is???

I would love this to be working in the end 🙏 It would make the live with a Smartphone soo much easyer. (If Privacy acually won’t be neglected)

No one seems to have heard the multiple times during WWDC where the speakers said "we are going to roll out most of these tools to Siri over the next year". This means they are not committed to any specific tool and they will not be available to end users for the next year (but they'll sell you a new phone in the Fall with vaporware, i.e. nothing, on it). The entire dog and pony show was to boost their stock price while doing and offering nothing. It worked. Their stock shot up 7% the next day. Apple is so far behind the AI curve compared to it's competitors. Come back when you actually have something in the public's hands.

Häää?!? For me this looks like Apple is copying Androids Intents! Android had intents right from the beginning, where Apps can use the services and data of other Apps via a (more or less) standardized way (called "Intents ")!

Does anyone notice that the intents sound suspiciously similar to a command line interface? We are going full circle.

Intents exist on Android API since the very beginning. How are Apple intents different from Android Intents?

5:38 Automating multiple daily tasks through voice control is ideal to attract non-savvy tech users. Asking Siri to switch on the faslight or to call someone from your contacts will superbly exciting for non-savvy tech users. Most consumer are not willing to navigate the settings app to find a service to turn on or off like enabling debug mode, turning on location for maps or changing the theme of their phone. Even to enable certain triggers to automate certain task is difficult and tedious for ordinary people but not for someone like me who tech savvy. 😎💯💪🏾👍🏾 Siri is the assistance central hub suitable to automate every technical and tedious task.

"Developers love making apps for their platforms and love adopting all of these new systemwide features". Are you sure about that?

Of topic but, the intro sound slaps

Gemini is already an personal assistant and a AI...

So an API for apps that AI can call upon. But that has already been done and failed - now everyone is retracting their APIs access. I cant imagine doordash skipping the occasion to upsell you on something, to track your scrolling within the app, or just showing you an ad. I remember first windows phones had a unified messenger/timeline app thing, but facebook quickly pulled access because it didnt show its ads and didnt report user behaviour. This is basically the same thing, but instead of a unified message hub its a wider scope - for every app. Sure, some apps wont care, but big ones do. Maybe its different when so many of ios apps are still paid? (and therefore dont require tracking, ads, lock ins, etc.)