Uncover GPT-4o Potential: Custom Prompt Automation for OCR Success | Power Automate | Azure
Рет қаралды 8,600
Күн бұрынWelcome to today's tutorial where I will demonstrate how you can learn how to extract values / entities from an image using a single api call and the new ChatGPT-4o Omni model. Using Power Automate I will demo how you can call the API on Azure. You will learn how to deploy the model to Azure and how to fine tune a prompt for a GPT use case. I will then demonstrate how you can run your automation on files uploaded to SharePoint, saving key metadata straight back to the file properties in SharePoint, all using Power Automate.
Whilst there are off the shelf models for OCR extractions in the form of AI Builder and Azure Document Intelligence, this new GPT-4o model is remarkably cheap to run and extremely flexible due to the ability for you to formulate a natural language requirement. By insisting the model returns structured JSON, we have a consistent output and can use this consistently in further stages of our automation.
Timestamps:
00:00 Introduction
00:45 Power Automate Flow and GPT-4o
03:42 Upload a file and trigger the automation
06:00 Deploy a model to Azure
07:25 Azure AI Playground
10:30 Devising a prompt with ChatGPT.com
12:06 Modify the flow to trigger from SharePoint
17:28 Testing the flow from SharePoint
18:40 Outro
By the end of this tutorial, you'll have a comprehensive understanding of how to deploy ChatGPT-4o to Azure and integrate with low code automation tools such as Power Automate. Please do let me know in the comments your ideas for automation.
Want to explore pre-built models? AI Builder or Azure Document Intelligence • Power Automate Invoice...
Want to build a canvas app gpt bot? Deploy Mistral Large, Integrate with Power Platform #Mistral #MistralAI #PowerPlatform
• Canvas App Chat Bot vi...
Got an idea? Or video request? Drop me a note here forms.office.com/r/4EqE7VHVfH 👍
Want to buy me a coffee www.buymeacoffee.com/DamoBird365 ☕
#PowerAutomate #PowerPlatform #gpt4o #azureopenai
@mannymorales977 10 күн бұрын
Damo, nice work - thank you for producing and sharing this excellent information. Kudos for throwing in a bit of debugging insight as well! Looking to continued content!
@mannymorales7913 10 күн бұрын
Great work, Damien! Thank you for producing and sharing the awesome video. It inspires me to keep pressing on with Power Automate integration with AI.
@DamoBird365 5 күн бұрын
Fantastic to hear 👍
@toluvictor 2 ай бұрын
Thanks Damo, this is awesome. This is something I wanted to get started with and I was just looking for an ideal use case
@ChandraChandra-wx9em Ай бұрын
This is just awesome. Thank you
@Emre-yx6vw 2 ай бұрын
Hi Damo, thanks for the content. It is highly professional with chapters, description, extra links and the way you explain how to do it is really amazing.
@DamoBird365 2 ай бұрын
Thank you very much for kind words 👍
@JonDoesFlow 2 ай бұрын
Damien is the master.
2 ай бұрын
Multimodal LLMs just made the OCR simpler... Great use cases Damien and as always superb demo, great you kept the base64 error as well 🔥🚀😍
@DamoBird365 2 ай бұрын
Yeah, pretty cool stuff isn’t it. Glad you liked the inclusion of the error. The amount of errors that I edit out too, you wouldn’t believe it.
2 ай бұрын
@@DamoBird365 No doubt. Respect for your time and effort to create such engaging content 😍
@XEQUTE 2 ай бұрын
@@DamoBird365 haha
@GeorgeBrown-gi2ku 2 ай бұрын
Thanks!
@DamoBird365 2 ай бұрын
Thank you George 😍
@JonDoesFlow 2 ай бұрын
This is weapons grade brilliant.
@DamoBird365 2 ай бұрын
Cheers Jon, really enjoyed exploring this and the capabilities are impressive. Can’t wait for audio to come to the api too.
@JonDoesFlow 2 ай бұрын
@@DamoBird365 Hey Damo, so I did initially get an error using preview API end point, "Request headers too large", I changed to the api-version=2024-02-01 and I no longer got that message
@DamoBird365 2 ай бұрын
@@JonDoesFlow exactly the same. Hope the video helped. I am glad it wasn’t just me. Now what are you creating 😍
@JonDoesFlow 2 ай бұрын
@@DamoBird365 I saw another post somewhere about that being a problem. Well at the moment I’ve built similar to you. It would be interesting to check the difference between AI credit cost and the cost of these calls to the API. Could be cheaper on higher volumes going this way.
@geralddahl9159 2 ай бұрын
Thanks for introducing me to the concept of developing a model in Azure that allows me to harness powerful chat gpt abilities. Can bing ai be used to refine the prompt for the model or must one use chat gpt proper for best results? PS - if my question misses the mark it’s because I have a sloppy understanding of large language model learning etc. PPS - 0.008 seems like 0.01 aka one hundredth of a dollar aka one penny but perhaps I’m getting that wrong too? (No need to spell out any long math remediation) - bottom line, I’m thrilled that you take the time to cost out approximate Azure pay as you go costs. Knowing that I can test something 500 times and only incur as little as 50 cents or as much as $5 is helpful. Thank you.
@DamoBird365 2 ай бұрын
I believe you’re right. It’s about a cent a run 👍 clearly my math brain wasn’t working this morning 😂 thanks for the spot.
@peterpetrou1904 2 ай бұрын
Hello Damien, thanks for great video. Can I ask does this solution require premium license in terms of the connector you are using? Thank you so much.
@DamoBird365 2 ай бұрын
Yes it would. Either via the http action or as a custom connector.
@tylerkolota 2 ай бұрын
I started exploring upgrading some flows based on my OCR & GPT3.5Turbo template to GPT4o and I'm finding there are some limitations not mentioned in the video. First, GPT4o requires the upload documents to be images, so png or jpg. That I expected. But the built-in Microsoft convert PDF to jpg action only converts the 1st page of a PDF to an image. So to use GPT4o on multi-page invoices or other documents would require a subscription to a 3rd party service like Encodian or Adobe in order to convert each multi-page PDF into a set of several images that could be fed to GPT4o. This is likely a no-go as my org doesn't want all our documents with sensitive data going through a 3rd party service. Second, GPT4o takes longer to process images than the OCR & GPT3.5Turbo set-up. So if an application has a user waiting in real-time on the document to be processed before proceeding, then it may be a hinderance to user experience. Overall, I think I'd only switch over to GPT-4o if going from like 96% to 99% field extract accuracy was necessary or if the use case requires processing something not typed text related like whether the document has a signature/stamp on it or not. Otherwise I may wait to see if later models enable uploading PDFs directly.
@DamoBird365 2 ай бұрын
Noted Tyler, feel free to explore other options. I’ve also covered AI Builder and Azure Doc Intelligence kzfaq.info/get/bejne/nLJ4oKioyM2ceqM.html. Hopefully you gleamed what’s currently possible from my exploration of the use case, but realise it’s a proof of concept, the model is all but 2 weeks live. This video and others will have inevitably given many new ideas of what’s possible now or in the future and I’m sure it won’t be long before agents use these models to orchestrate what we had previously considered to be possible with digital process automation.
@freshy1097 Ай бұрын
Can't you just ask chagpt to convert the pdf to jpeg?
@tylerkolota Ай бұрын
@@freshy1097 You’d have to feed in the base64 string or something of the PDF & ask it to output an array of base64 strings of images for each page. I’m not sure it can handle that type of request as it requires a decent chunk of code & libraries to do. Also base64 strings are really long. I doubt GPT could ever output then in it’s limited output tokens. But I actually just completed a prototype that allows me to do this in Azure without 3rd party connectors. So I’ll put up a community blog post on how to add that to this template.
@freshy1097 Ай бұрын
@@tylerkolota Thank you for the response. Looking forward to it!
@DamoBird365 Ай бұрын
@@freshy1097you could probably use a low or pro code OCR model and then sent the text to an LLM. I covered structured options here: Two options for Invoice Processing in Power Platform | AI Builder or Azure Document Intelligence kzfaq.info/get/bejne/nLJ4oKioyM2ceqM.html.
@bmassimo1966 2 ай бұрын
Hi, very nice video. An OCR recognition would be possible with Mistral Large LLM too?
@DamoBird365 2 ай бұрын
I’m not sure to be honest. I’ve used Mistral for text here 👉 Deploy Mistral Large, Integrate with Power Platform #Mistral #MistralAI #PowerPlatform kzfaq.info/get/bejne/Zt5xo9Gg2piroGw.html
@bmassimo1966 2 ай бұрын
@@DamoBird365 because for Azure OpenAI models are requested company subscriptions
@rishikeshmishal6986 2 ай бұрын
Hi Damo, Thank you for this great video. I implemented this in my environment, but facing issue "Request Header Fields Too Large" for a small image file content. Do you know how to solve this?
@DamoBird365 2 ай бұрын
Yes, it’s the date version in the url. If you jump to earlier in the video you’ll see the date. I got the same error.
@S0ulH0und Ай бұрын
Would it be possible to feed chatgpt inside power automate both jpeg and pdf for it to extract data from?
@DamoBird365 Ай бұрын
I don’t believe pdf can be ingested directly but you could convert to text via ocr first.
@TrueSpeaker-gx5tw 2 ай бұрын
A video request: Hi Damien, In our organisation we work in teams. And for new projects we create a channel with an predefined file structure, with a onenote for that specific project and a planner board for that project.... But thats a lot of manuell work... Is there a possibility to automate all these processes?
@DamoBird365 2 ай бұрын
Definitely doable. If you drop me more details via dm, I can consider it as a video. damobird365.com/contact-me/
@TrueSpeaker-gx5tw 2 ай бұрын
@@DamoBird365 I did just now😊
@theaunsyed 2 ай бұрын
Amazing. Can you give it as a subscription? I have clients that want to use it. What would be accuracy for Arabic receipts?
@DamoBird365 2 ай бұрын
Interesting 🤔 it does appear that Arabic is supported. openai.com/index/hello-gpt-4o/ whilst my demo is hosted in Azure, you/customer could use the OpenAI api.
@theaunsyed 2 ай бұрын
@@DamoBird365 CanI get access to your demo so I can play around with it? Would that be viable for you?
@johnfromireland7551 2 ай бұрын
@@DamoBird365 That said, the Model still needs to be hosted in the US.
@cbau0809 2 ай бұрын
You have referred to Chatgpt 4o as optima, and the o stands for omni.
@DamoBird365 2 ай бұрын
Very good shout 👍 openai.com/index/hello-gpt-4o thanks for pointing that out. Noted.
@hammadyounas2688 Ай бұрын
How to use it for pdf data extraction?
@DamoBird365 Ай бұрын
I’m not sure right now but you could check Two options for Invoice Processing in Power Platform | AI Builder or Azure Document Intelligence kzfaq.info/get/bejne/nLJ4oKioyM2ceqM.html 👍
@hammadyounas2688 Ай бұрын
@@DamoBird365 But i have the pdf which is basically a resume. I want to give the pdf file to chatgpt to give the summary of skills and years of experience and degree. Can you provide a tutorial on it.
@hammadyounas2688 Ай бұрын
@@DamoBird365 do you have any idea or code for this idea?
@DamoBird365 Ай бұрын
You could use OCR with either low code/pro code and then send the text to an LLM. But I’ll note your request for the multimodal LLM 👍
@hammadyounas2688 Ай бұрын
@@DamoBird365 how to code in power automate? I means do you have flow for that?
@stevedaregmailcom 2 ай бұрын
This is a gamechanger! thx for sharing! have you tried to make it work for PDF files? I tried replacing image/png;base64 with application/pdf;base64 but it doens't work, I think the type 'image url' has to be changed in that case?
@DamoBird365 2 ай бұрын
I’ve not yet. I couldn’t find it in the docs yet. If you do get it working, let us know 👍
@johnfromireland7551 2 ай бұрын
Try using Power Automate to, first, convert your pdf into an image and run from there. Though, for multi page PDfs will it convert to one long image? Or perhaps you will have to manage the individual PDF pages as separate images. It's starting to look tricky!. :-(
Lisa Crosbie
Рет қаралды 17 М.
Helena Liu
Рет қаралды 487 М.
Practical AI through Prototypes
Рет қаралды 10 М.
DamoBird365
Рет қаралды 7 М.
Leila Gharani
Рет қаралды 1,3 МЛН
Immersed
Рет қаралды 7 МЛН
LED Screen Factory-EagerLED
Рет қаралды 9 МЛН