No video

Power Automate - Fast Data Aggregation - Group By, Sum, Count

  Рет қаралды 7,385

DamoBird365

DamoBird365

Күн бұрын

In this video, I will show you how to group by, count and sum (add large arrays of numbers) values from a data source, such as SharePoint, using the most efficient method possible. I will use select to get distinct values, then convert the JSON data into XML. I will use select again to sum and count the values with XPath expressions. I will also demonstrate how to use xpather.com, a free online tool that helps you write and test XPath expressions. You will learn how to build a complete solution that can handle complex data aggregation scenarios in Power Automate Cloud Flow. No apply to each actions were harmed during the recording of this video. Only 7 actions and therefore 7 api calls are used in this flow to do all of the above.
[00:00:00] The purpose and benefits of data aggregation
Helps with writing, researching, and organizing data
Reduces the amount of API calls and avoids throttling
Enhances the performance and functionality of flows
[00:01:02] The example scenario and data source
A SharePoint list of employee expenses with 50 items
The goal is to get the total sum, the group by employee, and the sum and count per employee
A comparison with the built-in group by and sum features in SharePoint
[00:02:14] The steps of building the flow
Using a manual trigger and a get items action
Using a select action to get an array of employee IDs
Using a compose action with a union expression to get the distinct employee IDs
Using another compose action with an object to create a root for XML conversion
[00:05:02] The techniques of using XML and XPath
Using another compose action with an XML expression to convert the JSON into XML
Using another compose action with an XPath expression to sum up the total cost of all expenses
Using a select action with a concat expression to create dynamic XPath expressions for each employee
Using XPath expressions to get the name, email, total cost, and count of transactions for each employee
[00:08:14] The final output and verification
Using a compose action to display the final output as an array of objects
Comparing the output with the SharePoint list and the group by view
Explaining the advantages and limitations of this approach
[00:10:12] The setup and structure of the flow
The flow has a manual trigger and a get items action
The flow uses select, compose, and xpath actions to manipulate data
The flow outputs the total sum, the group by, and the individual sums and counts
[00:14:02] The use of select and compose to get distinct values
Select allows to select values from an array
Compose allows to create an object with a root and values
Union and sort expressions can be used to get distinct and sorted values
[00:17:15] The use of compose and xpath to convert JSON to XML
XML allows to use xpath expressions to query data
Compose allows to convert JSON to XML using the xml function
XML can be tested and validated using the Xpaather tool
[00:20:10] The use of xpath to perform data aggregation
Xpath allows to use expressions such as sum, count, and text
Xpath can be used to filter data by employee number
Xpath can be used to get the name, email, cost, and count for each employee
[00:24:22] The final output and review of the flow
The flow runs fast and returns the expected results
The flow uses only seven API calls and no apply to each
The flow can be modified and improved as needed
Check out my efficiency playlist for more great videos • Power Automate Efficiency
#PowerAutomate #Tutorial #demo
Please buy me a coffee www.buymeacoffee.com/DamoBird365 ☕

Пікірлер: 58
@jamesclark2020
@jamesclark2020 5 ай бұрын
I applied this technique today at work. What a thrill it was when it did exactly what it was supposed to do! Thank you again Damien ❤
@DamoBird365
@DamoBird365 5 ай бұрын
Cheers James 😍
@neilthackeray181
@neilthackeray181 2 ай бұрын
Simply AMAZING! very easy to follow what could've otherwise been a complicated process. Thank You Damo
@DamoBird365
@DamoBird365 2 ай бұрын
Glad you enjoyed it!
@om0729
@om0729 6 ай бұрын
I really love 💖 your videos. please keep sharing the knowledge. this is another great example where we can perform group by with in the Power Automate.👏👏👏
@ukm365
@ukm365 6 ай бұрын
It's good that you've split it out like this, Damo. I often did this with solutions in the community. Whilst we know that we can smoosh a lot of these into one expression, it doesn't provide the easy path to learn that.
@wendylumsden2224
@wendylumsden2224 3 ай бұрын
Thank you so much! I have learnt heaps and will apply to many of my flows which create custom reports from Dataverse tables. Greatly appreciate your generously sharing your knowledge to assist others
@DamoBird365
@DamoBird365 3 ай бұрын
Thanks Wendy 👍
@jamesclark2020
@jamesclark2020 6 ай бұрын
Wow. Excellent video, Damien! You made xpath seem more approachable
@cvkealey
@cvkealey 6 ай бұрын
Thank you for the reminder that I REALLY need to sit down and learn/get comfortable with xpath. I've been working with more flows that need to "post-process" data pulled in from SharePoint or Excel and the "filter array" action doesn't always do what I want.
@emmanuelmaceda2475
@emmanuelmaceda2475 6 ай бұрын
I was just reading your old post from 2021 on this topic with a hack using length function. Excellent vid as usual. Thank you for sharing!
@thanura.m
@thanura.m 5 ай бұрын
thank you for the demo. i constructed a similar usecase using a loads of loops. i should use this from now on!
@filipwinski4219
@filipwinski4219 6 ай бұрын
Awesome!!!
@dessyducasse3003
@dessyducasse3003 3 ай бұрын
Thanks this method is what I was looking
@temidayolonge4566
@temidayolonge4566 6 ай бұрын
Great Video as always.
@mrZerg00s
@mrZerg00s 6 ай бұрын
Thank you! I had no idea xpath was so powerful. 😲
@DamoBird365
@DamoBird365 6 ай бұрын
Glad it was helpful!
@user-hp8vy2co2j
@user-hp8vy2co2j 26 күн бұрын
Hi Damien. Your video is super-informative and pretty easy for a layman like me to understand. I followed it step-by-step, except in my case I am running a query against a Power BI dataset. I'm certainly not an expert and I mimicked your flow as closely as possible but, I think that my attempt is failing because I can't seem to get the grouped categories into the same structure as your in the XML output. My output looks like this: [ { "Category": "" }, { "Category": "- ACC CLOSED" }, { "Category": "- ADD INCOME" }, { "Category": "- INTERNAL" }, { "Category": "* CASH 1 _DEL Mon - INV Fri - CALL Thu" }, { "Category": "* CASH 2 _DEL Tue - INV Mon - CALL Fri" }, { "Category": "* CASH 3 _DEL Wed - INV Tue - CALL Mon" }, { "Category": "* CASH 4 _DEL Thu - INV Wed - CALL Tue" }, { "Category": "* CASH 5 _DEL Fri - INV Thu - CALL Wed" }, { "Category": "* EFT 1 _DEL Mon - INV Fri - CALL Thu" }, { "Category": "* EFT 2 _DEL Tue - INV Mon - CALL Fri" }, { "Category": "* EFT 3 _DEL Wed - INV Tue - CALL Mon" }, { "Category": "* EFT 4 _DEL Thu - INV Wed - CALL Tue" } ] As you can see, my output contains all of the squiggly brackets { } and I can only guess that it is this that is the stumbling block for later flow steps. I get this message at the Compose XML step "This XML file does not appear to have any style information associated with it. The document tree is shown below." 72 DEGREES BAKERY & FROZEN FOODS 0 0 2314.5 0 0 2314.5 I'm very inexperienced and not entirely sure that I have identified the actual error but it is the one thing that stands out for me. I'd really appreciate it if you could point me in the right direction. It will save me bucket loads of time if I can get the desired result. I've spent so much time on this and I'm super keen to get a positive result. Many thanks for your assistance, Andrew
6 ай бұрын
"Old friend classic designer" FTW :) Great demo Damo again. 👍💡🥳
@DamoBird365
@DamoBird365 6 ай бұрын
Ah yes. I’m working hard to feedback my findings with PG. It will get there.
@DamoBird365
@DamoBird365 6 ай бұрын
😂😂😂
@arturmusialak8704
@arturmusialak8704 6 ай бұрын
Great video! Thanks a lot 👍
@DamoBird365
@DamoBird365 6 ай бұрын
Cheers Artur 👍
@ukm365
@ukm365 6 ай бұрын
OOOOOH ... xpather is LUSH.
@wjameswilliams
@wjameswilliams 6 ай бұрын
I really feel for newcomers to Power Automate who will have no idea about the classic designer but also that will have no chance at figuring their way around all the bugs in the new editor. It's difficult enough to determine the issue when you know what you're doing! That said, this video was excellent. Avoiding unnecessary loops is so important, not least because of consumption but also because it's so much faster! Scheduling overheads can add minutes to a flow because of a loop before any actions even take place, which can even be exacerbated by other Microsoft customers' use of the multi-tenant environment 😞
@DamoBird365
@DamoBird365 6 ай бұрын
It’s definitely not an easy time to jump into Power Automate with both classic and new designer on the go and new having so many quirky bugs - agreed. I’m not a personal fan of Copilot in Power Automate either with so much of the automation incomplete but it will improve no doubt. I’ll keep sharing ideas on the new designer and hopefully folk can return to these videos. The concept won’t change and hopefully the video stands the test of time. I am planning on some newbie content 🤞 Just got to find the time as this is really just a hobby, albeit one I enjoy.
@chritoni
@chritoni 5 ай бұрын
Thanks! This provided a great way to handle large invoicing and accounting materials with efficiency. I´d really like to see how you would handle something that is large enough to require paging.
@DamoBird365
@DamoBird365 5 ай бұрын
Paging from the data source? Or something else? I’ve got graph api paging here kzfaq.info/get/bejne/aq-gm7WVpqqucpc.htmlsi=jTbBHRjpUY68veTW
@chritoni
@chritoni 5 ай бұрын
@@DamoBird365 Yes, I suppose I meant paging a source and in general I´m especially interested in handling a large number of rows in just about any setting. Did not find paging behind that link, but if you have such a video, I´ll find it eventually 🙈 Thanks for you blog and videos 👍👍👍
@DamoBird365
@DamoBird365 5 ай бұрын
Oooops, wrong link apologies 👉 Boost Your Power Automate Skills with Complex Arrays, Select, XML, and Join Techniques kzfaq.info/get/bejne/l8yhqapyx5iUdKs.html
@humayonsalim1877
@humayonsalim1877 2 ай бұрын
This is very useful. Can I ask how you would build the XPath expression with the sum() inside this xpath(outputs('Compose_XML), concat('(//values[EmployeeNumber=',item(),']/Email)[1]/text()'))). ?
@aaronalbers7280
@aaronalbers7280 Ай бұрын
Does xpath in this context support string lookups instead of integers? For example, lets say you wanted to to use John Smith instead. "//values[Name="John Smith"]/Expense[1]/text()"
@DamoBird365
@DamoBird365 Ай бұрын
It does indeed. 👍
@sankhlayashraj
@sankhlayashraj 5 ай бұрын
Thank you for the lovely video Damien. Leart a lot. I had one question for you on the xpath expression - The name in the map of is the name that you have given; however in the expression, are those the internal names of the columns. Also want to know would that be same if I am trying to get the count of data from a choice column in SP.
@DamoBird365
@DamoBird365 5 ай бұрын
The easiest way to find the name is to look at the output history. SharePoint might have field1,2,3 etc if you’ve created a list from an import. Good luck 👍
@sankhlayashraj
@sankhlayashraj 5 ай бұрын
Thank you so much. I was able to create a flow which is working perfectly fine. Appreciate your help.
@Xwodsarecool
@Xwodsarecool 4 ай бұрын
I have a similar list, with a column called County and Status and I want to 1.) go through the list and count the number of items grouped by the county name and be provided totals and then 2.) I have another column called status that’s a choice column and then I want to count the number of Active, Closed, and Pending statuses also by the county and be given totals. Would your demonstration work with what I am looking to do as well?
@DamoBird365
@DamoBird365 4 ай бұрын
It would certainly help you with most of your requirements 👍
@klebermedeiros8742
@klebermedeiros8742 3 ай бұрын
Very good. Is it possible to get a max value, using a expression like sum?
@DamoBird365
@DamoBird365 3 ай бұрын
There is a max expression learn.microsoft.com/en-us/azure/logic-apps/workflow-definition-language-functions-reference#max
@abbeyesval
@abbeyesval 5 ай бұрын
thanks for this excellent video, I just have a question, how can I add number format on the xpath function in order to summarize the costs? I tried with formatNumber(Item()?['AmountLocalCurrency'],'N2') on the initial table, however the sum result that I receive is NAN, hope you can help me
@DamoBird365
@DamoBird365 5 ай бұрын
I wonder if your currency is a string? You could try float() around the item()?[‘currency’]. The history will let you see if number is in quotes (a string) or not (a number).
@ma.teresatabaco5398
@ma.teresatabaco5398 2 ай бұрын
Hi Damo, thank you for sharing this great video. This is exactly what I'm looking for. However, I want to ask how about the calculated columns? Is it possible to use this also? I'm sorry I am new to Power Automate.
@DamoBird365
@DamoBird365 2 ай бұрын
In theory yes, but what you need to look at is the history of get items. Run the flow, check the history, can you see the calculated column value in output history. If yes, the logic is the same.
@ma.teresatabaco5398
@ma.teresatabaco5398 2 ай бұрын
@@DamoBird365 Hi Damo, yes, I can see the calculated columns now. For many days I did a lot of searching online for this. I'm happy I found your channel.
@ma.teresatabaco5398
@ma.teresatabaco5398 2 ай бұрын
By the way, can you help me on how to call this flow from another Sharepoint List? I have second SP List with a column for the sum (by employee) of the calculated column from first SP list. I am not sure if you have video regarding this, but I'm trying to watch all your videos one by one. If you have, please share with me here the link... Thank you so much.
@McIlravyInc
@McIlravyInc 6 ай бұрын
I use power query and VBA to do this, but i can see that this might be useful in conjunction with power apps to return some data for the user. Any other use cases where this technique is better than power query/bi/vba ? (VBA is no bigger lift than flows or apps IMO).
@DamoBird365
@DamoBird365 6 ай бұрын
There are definitely other products / platforms that can do this but some folk work in the low code space and have this requirement. An apply to each and a variable is not an efficient way of doing this. Office Scripts could achieve this easily I bet. But some orgs don’t have Office Scripts. Good to have options.
@McIlravyInc
@McIlravyInc 6 ай бұрын
Yes this technique has a lot of gems in it for other processes too. If they have power platform they have Excel also, and i find greater trust and acceptance with excel solutions in my area (especially because they can edit the email before sending, decide not to send one or more, etc) than just having emails come automagically (as one friend calls it). But I can see a use in a power app I'm thinking of for sure!
@DamoBird365
@DamoBird365 6 ай бұрын
@@McIlravyInc nice 👍 let me know how you get on.
@mk15minut64
@mk15minut64 6 ай бұрын
Can you make the same video in classic ui? Or maybe provide a zip copy of the flow?
@DamoBird365
@DamoBird365 6 ай бұрын
I’ve not got plans to do videos in both designer. But the video should give you everything you need.
@mikepsychles
@mikepsychles 21 күн бұрын
Why does this PA editor work for you and not for me. It does not save the parameters I add (like xpath strings, etc.) so I have to save them out to notepad every time I write one.
@klebermedeiros8742
@klebermedeiros8742 3 ай бұрын
My flow gives an error when the dynamic item has letters and numbers together. What can it be?
@klebermedeiros8742
@klebermedeiros8742 3 ай бұрын
It worked when I inserting " before '.
@DamoBird365
@DamoBird365 3 ай бұрын
Glad you worked it out ok 👍
@joao0790
@joao0790 2 ай бұрын
@@klebermedeiros8742 Hi Kleber, what do you mean by inserting "before"? I am having the same issue. The field I am using to find the xml value is a string (very long, indeed), and I noticed using the xpather, it cannot find elements. Do you have any clue?
Power Automate | Apply to Each | Why? | Arrays and Objects
19:05
DamoBird365
Рет қаралды 36 М.
50 YouTubers Fight For $1,000,000
41:27
MrBeast
Рет қаралды 210 МЛН
How to group and calculate sum of Power Automate array items
24:54
Alireza Aliabadi
Рет қаралды 15 М.
IT'S HERE: Work Queues in Power Automate - Full Tutorial
42:19
Anders Jensen
Рет қаралды 20 М.
How to write expressions in Power Automate
18:31
DamoBird365
Рет қаралды 10 М.