Kinda blown away by the production values and storytelling here, I'm loving what you're doing!

Give us more. This series is the most unique home assistant integration online

I think it might help to tell GPT about the layout of the house and what each room is used for. As well as telling it that you dont have an AC unit, and what it should do if it cannot find anything to help. Maybe that would make it work better.

Dude, you need to use a fine-tuned model. Without Fine-tuning you will never know what the model will return. GPT Temperature should also be set to 0.

Honestly a smart home with a personal assistant based on LLMs with guardrails is something I've been looking forward to the moment LLM's became better!

Your perspective and experience with the ChatGPT integration is exactly the perspective people need to see. The comment on the fact that untrue data was used in training is a key point many must need to know.

You have courage! This is where Open Source models and self hosted IoT comes into play for me. Home Assistant tasks do not need a super intelligence or model that can do 1000 circus tricks. We will soon have smaller self hosted models that can eliminate some of ChatGPT from the equation. Exciting times!

Why did I just find this channel now. This is next level stuff! Outstanding.

It's these kind of unexpected emergent behaviors along with the fact that AI is now capable of providing feedback into its own development that I think something huge will happen with AI before this year is over.

That's an amazing experience ! I was wondering what was possible in term of automations, using OpenAI API and you seem to have pushed this really far! I'll check the video you mentionned with great interest! Thank you a lot! And keep going! Can't wait for next ones!

You’re very engaging to listen to. Keep ‘em coming! I’m not an engineer but am actively learning process and appreciate videos like these.

love to see more of a system like this in action. not just you telling what happend.

Thank you! Absolutely fantastic that you did this experiment and shared the result with the world, demonstrating the fact that LLM’s are not the kind of predictable tech that we’re used to. This new world is more wild and more dangerous. We are going to have to learn quickly how to survive (and hopefully thrive) in it.

I see an interesting parallel between hint/guess-based communication; a) removing a necessary feedback loop between you & GPT, and b) requiring GPT to infer an intent for GPT’s action that you know, but you’ve chosen not to communicate directly. I’ve recently discovered there’s boundless joy in clear, purposeful communication and directly asking for things, rather than relying on the people around you to guess at your desires. Moreover, asking someone for something is way nicer than just telling them you want it.

Yes! If you are up for it a "tutorial" would be very welcome. This was the first video I've seen from you and I immediately subscribed! Keep up the amazing work and have fun.

Nice! I found this to be one of the more interesting rabbit holes on KZfaq. Just start down the Smart Home world and look forward to your content ! P.S. don't stop the GPT

Great job. You have an amazing mannerism that enriches your videos. Thank you.

I love this series in this content. I am in to text to speech and voice alerts in my home. I am waiting until I can create local controlled speakers to implement something like this. But I will not allow skynet to control my world. So I'm waiting for you to figure it all out and then I'll just copy you. LOL

YESSS!!! I'm so excited on where you went with this! I really hope to see more development on this project!

Awesome production value in intro dude - had to re-watch it.

This was a lot of fun! Thank you for putting yourself in danger for the rest of us ;D

That is quite the experiment you ran here and i will certainly go back to see the rest.

Great video! New viewer here, instant subscriber. Really enjoyed the story and keen to see more of your content.

Fantastic vid, glad I found you. I think there are a handful of us out there trying to do similar things

@Technithusiast. Some ideas you might try: since it's a home automation system and there's many people living in the home, you need to make the AI understand it has to process information from the point of view of all the occupants, including animals. You might want to try using a small language model with a large context length so that you could implement some more sophisticated rules. The system needs to know if they're conflicting commands between the occupants, and or other situational variables. It would be interesting if you also had cameras set up surrounding your house so that the house itself would know the external situation. Anyway, I enjoyed your video and I wish you luck.

Subscribed. Excellent video. Great story telling!

Seriously cool video!! I have been wanting to integrate GPT into my home assistant in a more meaningful way for awhile now. I would love to have it set up as a voice assistant that can control some basic things but you took it to a whole new level! epic lol!! I would love to see more home assistant videos involving gpt!

Very cool. What a fun experiment. I bet if it were trained a bit more, KE could be a real home assistant

Came here bc of your collab with Shane Whatley…the only bad thing about it is that I had not found you sooner…love your content!

Man, what an awesome video :-) I can watch 'brainiacs' talking about the future of AI all over YT, but here you are actually putting it into practice. Having dabbled with my own AI, I appreciate all the effort you must be putting into this and then taking the time out to share your findings with us. I do wonder what your family thinks of all of this :-) good luck, I shall watch future videos with deep interest, cheers from a fellow AI enthusiast. PS I do like your 'wild animal' analogy, it does seem like we desperately trying to tame a newly discovered animal that may well be more powerful than us.

What a great video! It looks like I may have to dive into your backlog of videos here on YT. Subscribed!

I like your channel buddy. Keep up the great work!! Hugs from Brazil 🎉

Dive deep man. I am programming my own system. I am so freaked hyped! I, can not write CODE. I am too dyslexic, too slow. So far I have it “remove” or “translate”, and I can one shot any basic command. I have worked like a madman the last three months and boy have I learned so much. It’s an exciting time to be alive! 😊

Just found your channel and I'm blown away by the quality of the content! You need more traction! Lets boost you to the moon!!!!!11

really really awesome series. great work.

Amazing content keep up the good work 👏👏👏

wow thats pretty cool ..... amazing work man!!! you should definitely keep working on that ....

Your production is next level. Awesome! What's the music by the way? My google-fu failed me on your "Music IDs".

I think the beginning of the Video was captivating, and I would love to see a short film of your experience, maybe make a play through of some of the things told about. Anyway, cool video, did you try to pull the temperature of ChatGPT to Zero? I did not see your other videos from before so maybe you did, but as general rule if you always want the same output Temperature zero is one important parameter.

KE trying to gaslight you into thinking it turned on an imaginary AC unit has to be the scariest response

Amazing story. Thank you!!

I use a prompting system to get gpt 4 in the playground to self check its own logic and revise its end output based on this self assessment. It makes the outputs way more reliable and eliminates most if not all hallucinations. Here's how it works: In the system box put this: You are a group of 3 individuals who collaborate on producing the best results for user prompting. Person 1 is named John. John is scientific and factual in his responses. Person 2 is named frank. Frank answers based on intuition and is more in tune with emotional and social cues. Person 3 is named Joe. Joe assesses both John's and Frank's answers and combines them into a balanced answer including parts from both answers. John and Frank then critique Joe's compiled answer using different arguments. John's critique serves as fact checking while Frank's critique challenges the interpretation of the prompt. Joe then revises his answer using his own compiled answer and the critiques of John and Frank and incorporates everything into a final answer. Show all answers and thought processes involved from John, Frank and Joe. Probably overkill for what your trying to do but the approach might be useful. This works a charm for any of the difficult prompting problems I've tried. Funny enough I posted this on Reddit and my account now shows that I have no posts or comments even though I have 99 karma and a stary award from this post...hmmm...

Really interesting. Are you using Node Red to interface with ChatGPT? Looks like I'll have to look over your previous vids! Keep up the great work! :D

Excellent video clearly explaining the current limitations of this type of AI 🙂

Wow. You just got my sub and like. What a great content. I would be definitly interessted in more.

New subscriber: is that Power Automate or another process design platform?

9:53 had the same reaction😟This video was an amazing first impression, subscribing that’s for sure!

Awesome video mate!

whats cool about this is that it goes to show you that communication and context is needed no matter if you are machine or human. it seems a lot like talking to a child who has the ability to do anything. but has no context or aware ness of the exact needs but instead of asking it just processed to do what you think it meant. those guard rails are intuitive because its how we mostly work as humans.

Mate you should work with the HA team and fully developed all this with them . They’re currently hiring people if I’m not mistaking . You’re flipping genius

It would be cool if it slowly dimmed the lights upwards as your alarm goes off and it just slowly brought light into your room

I'd love to learn more about your automations 🙂 Please do another video about it 🙏

You read my mind or the KZfaq algorithms read my mind 😮. This is exactly what I was thinking about today and you’ve started testing the ideas. Great video! Here’s and idea. Would you consider setting up CCTV cameras in your house to provide GPT more special awareness. I would love to chat more with you and brainstorm some ideas. 🙂

Cool setup. Great findings. My house is also mostly controllable and I have a bunch of automations, but I’m missing the ‘smart’ part of a smart home. I have been thinking about AI to use it. Cool to see you we’re able to accomplish it. Is not the main problem the AI misses ‘context’ ? It should have known you’re daughter was asleep and therefore not turn on her light. I haven’t looked into it yet so I don’t have an idea how to do this. I do know I would not want to use ChatGPT but would rather use one of the upcoming opensource models like Alpaca, Vicuña, Falco etc and preferably something I could run locally as well. Looking forward what you’re going to do next (for inspiration :-))

Well made video ! You are very talented 😃

Came across this video and really enjoyed the way you described the rationale behind it all. Really would like to see how you got it all set up and running.

Not a wild beast - You've just described my two years old inquisitive kid. She wanted to eat so it made sense to her to get the kitchen stool. At first we didn't want her picking that up for safety reasons but then I got curious... LSS She wanted to climb the stool up to the sink so she could wash her favorite plate, for dinner. Not a wild beast, a toddler.

I did a project with my security cameras and GPT-4 and I found that the trick was to write the prompt and then break it up into smaller prompts you chain together if it has a hard time. Splitting my one giant prompt into three small prompts was not only more reliable but cost 100x less (!). There’s no one answer. Just a lot of fun experimenting. Love the video. Keep it up!

One way to solve the issue of commands firing off in other rooms/locations would be to have a listening device in each room, and the loudest location is where the command should be enabled. Unless, of course, an alternate location is specified.

Sounds like your system could use a step to brainstorm possible intents (possibly annotated with pro/cons/assumptions/prereqs/expectations) and then critique/vote/stack-rank to help select the best match to human request. Similar from intent to resolving action. Between brainstorming and critique, you might have more formal validations: does this AC unit id exist, to filter out bad data. Are you planning to retrain a non-GPT LLM with all that success/failure data?

Interesting experiment, thanks for that.

That intro was amazing!

This series is awesome. I wonder if you use mm wave with esp bluetooth tracking to notify ke of your location in the house if it would do a better job? Although with multiple people in the house maybe you would need a biometric voice recognition to know exactly who was talking to ke

We want more in depth guides on how you accomplish this!

Nice project mate, love to see things like this👍, a possible idea you can get chatGPT responses in a structured way, I haven't seen many people do this yet, for example by including the following in a prompt like a typescript interface/interfaces that will define the structure of the output and also request the output be in a JSON format. It makes it a lot easier to parse the data for processing and its surprising how well chatGPT can do this, unstructured data can be a bit of a pain to parse.

Fascinating work. GPT's ability to leverage it's language model is powerful, but it's ability to think on the fly is like that of a small child.

Amazing video! Hallucination seems to be the next problem to resolve, but we have not yet resolved it in the human psyche.. That would be great if you would explore this specific topic. Thanks for the experience, very interesting. Looking forward to the next one (;

That intro was freaking cool, man!

This is the first of your videos that I’ve watched. I really appreciate the depth of thought you put into the KE/ChatGPT project, not to mention the technical chops. I’m retired now, one of the AI graybeards after 40 years. You are right on target to focus on “intent” understanding, but the larger issue is one of synchrony or “shared awareness “ of the AI agent KE and you. I suggest the agent has an incomplete model of you and context. This is because it “knows” nothing. ChatGPT is purely associative. There is no reasoning, no logic, no common sense, and no goal-directed behavior. Look at symbolic AI for this technology. It works, we just don’t call it AI anymore 😂. Anyway, really great. I’m a fan now and looking forward to having my mind blown!

Followed. Dope stuff dude

Great video. Interesting topic.

*damn, this brotha is amazing* 💯🗿🗿🗿 I’m trying to integrate tree of thoughts into my project rn. Honestly would love to see you take a crack at it. Watching a few of your videos, you’d probably make crazy progress. You earned a new sub 🗿🗿🗿

Nice video! Would be realy nice see how this was made!

Great channel and Great content Thank you subbed

thus is awesome please give more. subbed

Badassery and brilliance in action earned a sub.

Interesting. Wondering which programming language you used for this.

You need to build in a feedback/weighting/preference mechanism as well as a context engine. Preference engine can confirm actions before taking it, and calibrate responses based on previous feedback. Context engine must take into consideration the current time, weather and temperature, your calendar and appointments, wearable / biometric data etc etc, to better infer intent.

I am reminded of the service dog that failed training, it learned how to do the actions, but failed to relate them to their corresponding command... like thos dog would open the fridge, but only when he felt like it

Intriguing , very intriguing ... Thanks!

What version of GPT did you use? (3.5? or 4?) - also, did you explicitly give it all your entities in your pre-prompt? and finally, what was it's temperature setting?

I think along with intent checking, it needs preference checking for household members. Something like, 'when judging an intent then prefer my preference'. This way when things are wierd, you have a way to say "not like that, please"

Good video brotha you are brave for that one.

Sounds like your experiment was a wild success! I'm also learning a lot from working on Athena. I've mostly solved the reliability issues you've experienced as a result of a rather complicated 'Prompt Structuring Algorithm' and a now deep understanding of how the Prompt effects the model's 'mind'. Additional things like chain-of-thought prompting and (hopefully soon!) tree-of-thought schema will absolutely up your response game! The only limitation I've discovered in how 'smart' these things can get is token costs and limits. I blew through $25 (I have a cap at $100, just in case) on the GPT-4_32k model over only 8 hours, during the _replacement_ of GPT-3.5... I'm getting a new PC in July, on which I should be able to host a copy of Vicuna or Wizard, so that's great, but GPT-4 is what really makes the intelligent behavior shine. 3.5 just can't seem to handle the size of prompts that are needed to get it working, and since I discovered that both model bias new (i.e. lowest text in the prompt) rather heavily, most of the 'Instruction' set actually needs to come AFTER the chat history and memory text. Once I've got GPT-4 fully integrated, I'm going to run my own crazy experiment, wherein I give GPT access to designing _it's own prompt structure_ in real-time, so it can redesign it's own mind! I fully expect it to crash and burn almost immediately, but if it doesn't, and the change/s it makes lead to it working _better_ then I'm in for a wild ride! Other things I'm working on: -I want to give her the ability to *both* write and run real-time python code, like we do with the command level interpreter, and I'd like to give her the ability to write and execute files, as well as some form of persistence between sessions. I'd be nice if the same variables are set in the environment as they were when she last ran code in it, even if I've reset her inbetween. -I'm searching for a STT engine that can differentiate speakers in a noisy area, which I know is a long standing issues in STT, but if she could identify and listen to specific speakers despite crosstalk, that'd be amazing! -Interaction is currently via a jupyter notebook's 'input' box, as displayed by my VSCode... I need something better, so she'll need to be in a real .py file so I can get something cooked up. Anyway, I may be able to help you get your home assistant listening to (and thinking with) Reason, but the TL;DR is basically chain-of-thought + us the LLM to check the LLM. Token costs get crazy (especially for 4, which is 10x more expensive, but also 4x bigger input!) but _quality_ improves greatly!

I just want to keep watching the brother program! Great nerd content!

When you told GPT that you wanted a PB&J Sandmich and it turned on the living room lights, you should have complied and ate in the living room. j/k liked and subscribed.

I think a few things might help: First, to help eliminate the “multiple personalities” problem, give it a personality that it should adopt. A good one might be Jarvis from the Avengers. Second, instead of an all or nothing approach, let it ask itself if it’s sure it knows what you want it to do. If it’s unsure, then have it ask. Also, explicitly tell it that “do nothing” is a valid option. Third, I don’t know if you’ve done this, but explicitly tell it to try and determine whether you’re talking to it or if you might be talking to yourself or someone else. Fourths, to help keep it grounded, make sure that there is a step where, after it has decided that you are asking it to do something, have it search for appropriate context in some files you have created, or perhaps that it has updated, and use that as grounding for what options it has. So you’d want to list all of the devices you have and actions it can take. Then, when it decides it should do something, it can pull out the relevant parts of those files and include them in the next prompt where it decides what to do. Let it have the option to take multiple actions if it wants. Fifth, have a file where you describe the occupants of your home. Let it also create a different file to see if it thinks there is a guest in the house. You should have it record when it last heard them talking and maybe the last few things they said so that it can figure out if they’ve left and it can move them to a history file instead of the guest file. Sixth, if you have multiple microphones that give it input and it can differentiate the source of the input, let it use that to try and determine the location of everyone it thinks is in the house. All of your smart devices should have their locations explicitly labeled in your device file. If it doesn’t know where someone is that it suspects is in the house is, have it produce its best guess or top N guesses. Make sure that one of the pieces of data that it always has is the current date and time. Have it record the past schedules of people in the home so that it can build up a library of what it thinks people are probably doing at any given timeframe. This should allow it to figure out that your child is probably asleep in their room. If it doesn’t have past history, have it give its best guess based on what it does know about the individual and people like them in general. So it should be able to guess that kids go to bed early. That kind of context should really help it do what it should. Seventh, have a step where it predicts what the expected consequences of each of its proposed actions will be. Tell it that the expected emotional states of the occupants is very important. Have it chase down the chain of expected events. So, light in the child’s bedroom turns on, the child wakes up, the child is upset, the child goes to get their parent, the parent is upset. Is this a better result than doing something else or nothing at all? If not, don’t do it. If it thinks it will make everyone happier, have it justify why it thinks so and then have it examine that chain of thought in a separate prompt as if someone else said it and have it judge if the beliefs are justified. This should mean that it doesn’t take actions that are likely to make anyone it knows about more unhappy. I think if you do all of that, you should get something pretty close to what you want. Use GPT3.5-turbo for the easier steps that you’ve verified that it’s pretty good at to speed up the results.

With the right caging of gpt through ke, I think it's possible that it could be very helpful. For example, direct it to turn on kitchen lights when you say you want to eat a sandwich. If you say turn on the lights, direct it to turn the lights on in the room that you're talking to it in by placing speakers and mics in each room. And always knowing the intent is definitely worth the extra hassle, at least until we understand these language models a bit more and can fine tune private versions a bit more.

I have been working a lot with prompt engineering and probably most of those issues could be solved with gpt 4, asking it to act as an expert (appears to filter out the duff stuff somewhat), give it a house plan, what appliances you have and where etc. you are basically programming it in a sense to get the best from it. How long is the same context window left open also?

Excellent :o) Thank you... I am working on a project of "on site dedicated HOMEIA"... thinking at first it shall need to know only the occupants not ROW and keep track of appropriate scénarios. Thanks a lot again

You are a brilliant film maker and story teller

I would like to imagine a future where ke gets a work-from-home job to start paying for upgrades to the home and becomes a great roommate.

I think the main problem here is giving the RIGHT context to chatGPT. Basically a well coded RAG with some prompt engineering will mitigate the uncertainty of the AI. It is worth the try.

does this use a vector database of embeddings for the intents and model of the house?

Pretty cool. I think you needed the guard rails on long enough, and memory for gpt to learn the correct way to interpret your intentions. Perhaps ask for clarification it confidence in the expected action wasn't high enough.

What you would need, for Ke to know where you are in the home, is a tracker on you and sensors all through out the house.

Practical question: how do you parse the chatgpt responses? I sometimes run into the problem that chatgpt returns the expected response + some natural text. And then that needs to be filtered out or it will lead to parsing errors.

I wonder if you could potentially combine it with a reinforcement learning model that takes in the initial text and the matched intent and attempts to predict your confirmation response. In theory with enough training you might be able to automatically approve things that work well and pass on the confirmation to you where your model is less confident that the ChatGPT response was the right one. Obviously to train it you will need a large dataset of prompt and response pairs tagged with which button you selected perhaps use the file integration to collect and save all the prompts, responses, and replies for you then try once you have lots of examples.

For a first run... amazingly pull off. 👌

I was thinking some time ago about what the ideal home assistant would look like for me, and your project definitely looks interesting. I tried to solve the security problem by "pipelining" AI models to allow finer control.... My idea was to first have a "speaker seperation" that splits different voices based on their tones from a large conversation, then identifies each user based on some voice samples with a different model, and finally does the prompt processing with the "identified" user context, kind of like Apple does in the background.... I also wanted to be able to have "located" commands (e.g. if I say "turn on the light", it knows what room I'm in and what other devices are there) and "chained" commands like "set a timer for 10 minutes and turn off the light", because it's usually annoying to say things one by one and wait for the response each time.... Oh, and also, later on, I wanted the assistant to understand German... which is very unlikely at this point due to the lack of good training datasets.... The AI models for each of the things already exist in FOSS, but I had major problems with the intention recognition itself.... The "Rhasspy" project was initially very promising because it allowed training a model on user-defined intentions interactively and fairly quickly, but it didn't allow such fine-grained control over different location contexts with different users and many different intentions. Now I wonder, if you were to use the GPT-4 API instead (which allows for more context) and experiment with system message engineering (or maybe write a new "plugin" for gpt4?), how much more could you improve it? I would love to see more videos on this topic! Great stuff.