How to process multiple folders with a single Power Query script

  Рет қаралды 13,656

Exceed Learning

Exceed Learning

Күн бұрын

In this video, we show a way to solve a specific problem that occurs when we need to process multiple folders. Instead of processing folders one by one, we created a single script that will process multiple folders.
Processing a whole folder is one of the most powerful Power Query features, but with the use of nested custom functions, we can make it even more powerful!
If you are interested in learning more about Power Query, check out our Power Query course!
exceed.hr/en/academy/powerque...
#powerquery #powerbi #etltools #data #dataanalysis

Пікірлер: 51
@sourabhjadhav4621
@sourabhjadhav4621 2 жыл бұрын
I love you man!! The level of simplicity in teaching and amount of orderliness in your thinking is a gift
@ThepExcel
@ThepExcel Жыл бұрын
This one is cool!!
@dangtuannguyen948
@dangtuannguyen948 2 жыл бұрын
Man, amazing. It blows my mind. Thank you so much . It saves ton of working hours. 6 stars
@felipesignorellireis7839
@felipesignorellireis7839 3 жыл бұрын
Amazing. It is that what I'm looking for. Because, usualy I put all the files into one folder. I will training that. You are a Master from PQ. Cheers from Brazil.
@andrewsegawa7221
@andrewsegawa7221 Жыл бұрын
Best thing I’ve watched ever on Excel data Manipulation!! Thanks for sharing this knowledge!! Let me follow you INSTANTLY!! ❤️❤️❤️❤️
@tonydreisenstock
@tonydreisenstock 10 ай бұрын
I think your PQ videos are amazing. I have learnt so much from you thanks
@weeyewtiong9622
@weeyewtiong9622 Жыл бұрын
Thank you, Sir. What you have demo is very practical in most aspect of my work for data transformation. Best of all, with these methods, I am enable to allocate designated file paths & speedup the runtime in PQ. Great work. Love your teaching.
@1qtaz
@1qtaz 2 жыл бұрын
Wow! I know I will feel competent using functions now. Thanks so much.
@nguoibenle4479
@nguoibenle4479 2 жыл бұрын
Awesome! Thank you! Found & learned PQ a few years ago when it wasn't popular at all and when PQ was still an add-in to Excel. Gave up after a few months due to lack of learning resources for M language - I needed to make things dynamic since I work with data from 50+ states and there are so many variations. Recently I found some nice use of PQ for some tasks at work - easier to create and maintain compared to writing codes in R or Python. But the need for M language is still valid. Your video is really helpful! Hope to have time to watch more videos of yours. Thanks for sharing your knowledge!
@ExceedLearning
@ExceedLearning 2 жыл бұрын
Thank you NguoiBenLe! If you are interested, you can check our new PowerQuery course with lots of advanced M topics covered! exceed.hr/en/courses/powerquery-fundaments/
@larmondoflairallen4705
@larmondoflairallen4705 3 жыл бұрын
Fascinating! I have never understood custom functions, but I'm starting to understand their usefulness thanks to your videos.
@ExceedLearning
@ExceedLearning 3 жыл бұрын
Happy to hear that! :)
@silverfunnel6819
@silverfunnel6819 3 жыл бұрын
Congratulations, sir! Always great content!
@naveedyousaf1657
@naveedyousaf1657 2 жыл бұрын
Absolutely Brilliant. Thanks for Sharing. 👍
@zaydarendse2812
@zaydarendse2812 3 жыл бұрын
I'm officially a fan :) Thanks again!
@aronnordon4275
@aronnordon4275 Жыл бұрын
This video is the best i've ever seen. Helps me a lot. Thank you so much!!
@Hans_Olo
@Hans_Olo Жыл бұрын
Thank you!!!! This is a huge time-saver
@balakrishnaj1713
@balakrishnaj1713 2 жыл бұрын
Thanks, this is crystal clear for me to understand each point of code. Waiting for more videos to learn more :)
@user-gs2uf5gb6b
@user-gs2uf5gb6b 11 ай бұрын
Thankyou, very useful and very well presented. One of the best Power Query videos i have seen
@FRANKWHITE1996
@FRANKWHITE1996 Жыл бұрын
Thanks for sharing ❤
@tekylyfasharing9995
@tekylyfasharing9995 Жыл бұрын
Thanks for sharing, it's very useful for me. i have learned something on query.
@kennethstephani692
@kennethstephani692 2 жыл бұрын
Great video!
@kebincui
@kebincui 9 ай бұрын
Excellent tutorial❤
@ImranHussain-xv4se
@ImranHussain-xv4se Жыл бұрын
Really helping
@juansepowerplatform
@juansepowerplatform 3 жыл бұрын
Thank you, I was able to use this but with CSV files. Great content
@ExceedLearning
@ExceedLearning 3 жыл бұрын
Thanks for the info Juan. This technique is versatile and can be used in many different scenarios. The borderline idea is to understand that you can nest custom functions!
@dirkstaszak4838
@dirkstaszak4838 2 жыл бұрын
Good video. Interesting approach, that I used in a today's task :) Thanks
@ExceedLearning
@ExceedLearning 2 жыл бұрын
Glad to help!
@malchicken
@malchicken 3 жыл бұрын
Excellent, much appreciated! I’ll have to try this, along with your shared blog to solve ‘table redlining’, in time. Some follow-on thoughts: I tend to want to share these solutions across different groups within the business, I wonder if you have any tips for sharing power query solution “templates”. So far I’ve been trying out ‘Export Connection File’ for functions, which seems to let people pick and choose which saved function template they want to use at any given time. It seems a bit more versatile so far vs sharing an Excel Template document with all the functions pre-built in. I think there is some way to enable a “load organizational shared connection files” but, our org doesn’t seem to have that, so I just ask people to load the connection files from a shared folder. Additionally, I think for this solution I would try to ‘hide’ the first function in the second one by using a second Let-In, that way people only have to import 1 shared connection file to get this to work. Also I appreciate in the video how you highlighted "this is the main take-away". To further make it easy to find the key take-away later, please consider adding a "chapter" to the video which highlights this key take-away (this is a youtube video feature). I also wanted to recommend adding 'Recommended' channels to your KZfaq channel. It seems like channels with high views tend to have lots of featured other channels. I wonder if KZfaq's algorithm uses those channel recommendations to figure out which videos to recommend I watch next. Thank you for the tricks :)
@ExceedLearning
@ExceedLearning 3 жыл бұрын
Thank you Hendrick! Haven't thought about sharing Excel PQ scripts as a template. My approach would be to create a parameter table in which each department would paste their location/s before refreshing the data. And thank you for your insights on how to improve reach on KZfaq! We are still fresh in this area, so every feedback is much appreciated! Cheers
@AStepIntoSeclusion
@AStepIntoSeclusion Жыл бұрын
This is great, and a good intro for me for functions. However, when I do the manual approach (what I was doing before functions), I get the filepath as a column which is very good for doing certain joins (i can use the filepath for dates folders were created, and then use a master date table for current week, or quarters, etc). Is there any way to get the filepath to be included with the approach in the video as well?
@ExceedLearning
@ExceedLearning Жыл бұрын
Hi, you can get the folder path column by including it in Removed Other Columns step of the second function (the second step of CleanAllRegions function): #"Removed Other Columns" = Table.SelectColumns(Source, {"Content", "Name", "Folder Path"} )
@nlopedebarrios
@nlopedebarrios 2 жыл бұрын
Hi, great tutorial. However, I would like to load the data into an existing sheet, but the option is greyed out. What could it be? The source is a Folder.
@felipesignorellireis7839
@felipesignorellireis7839 3 жыл бұрын
Teacher, when I expand the content column clicking in the two arrows Down, the Excel always create automatic folders in the "queries" left side with auto transform function. Question: It's possible to expand ".csv" "folder" with append without create queries folders? It's because I think the PQ It gets very dirty. In this vídeo you create a custom column. I that what need to be done? Thanks any way teacher. Cheers from Brazil.
@ExceedLearning
@ExceedLearning 3 жыл бұрын
Hi Felipe, yes, you can create a function that accepts the CSV path as input and work from thereon. There is no need to use a built-in combine function with pressing double down-pointing arrows. The function in Power Query used for that is Csv.Document().
@ndrtimdauti4173
@ndrtimdauti4173 Жыл бұрын
This is amazing! I tried it today and works! Wow! is there a way to ignore a folder that has no data? I have 10 clients (with different folder location) that send selling Infos monthly, but it might be that from time to time there are no sales for a particular month. In those cases I receive an error message that sais no data found on folder xyz. Is there a way to avoid this error and load the remaining 9 clients? Thank you!
@lopelle
@lopelle 2 жыл бұрын
Quite amazing solution for files and folders on a “C Drive”. And very clearly explained. But, how to apply the same or a similar solution when data are on SharePoint (Teams)?
@DanteCry23
@DanteCry23 Жыл бұрын
Did you find a solution?
@allabout1135
@allabout1135 3 жыл бұрын
Hi, First of all thanks for sharing the knowledge. Have question. I work with large data sets conected by power query. And problem is that, whem I make some modifications in power query steps query is refreshing each time what cost a lot of time with large data sets. Is it possible to disable query reffreshing?
@ExceedLearning
@ExceedLearning 3 жыл бұрын
Hi Aleksandrs, you can try disabling the background refresh option in PowerQuery, that way you can force to refresh only the selected query, not all the previews.
@numchuck180
@numchuck180 Жыл бұрын
I'm trying this with CSVs and it doesn't seem to work. I get a 'field ... of the record cannot be found' when I try to invoke the FXClean function @6:55
@syedaneesdurez7197
@syedaneesdurez7197 Жыл бұрын
Sir, Highly beneficial script. Pls share the script . Thanks in adv
@Mister_Bates
@Mister_Bates Жыл бұрын
Love the approach, but when I tried it with multiple files from different SharePoint (Web) URLs, I got a Formula.Firewall error. Is there any adjustment I could make to your approach so that I can process a list of SharePoint file URLs without bumping into Formula.Firewall?
@Mister_Bates
@Mister_Bates Жыл бұрын
So, after experimenting, it seems that to "avoid the formula.firewall", the functions should be embedded one inside the other, so that the query accesses the files to be processed, then processes each workbook using an in-line function, which in turn processes required sheets using a further in-line function inside the workbook in-line function. It avoids the firewall issue, but at the expense of limiting code re-use (since the in-line functions are inside a query and can't be used by other queries.
@DanteCry23
@DanteCry23 Жыл бұрын
@@Mister_Bates Did you find a better solution?
@DarrylBodkin
@DarrylBodkin 2 жыл бұрын
Excel.Objects keeps giving me table and sheet. Then I get an error when I invoke the custom function on the table.
@ExceedLearning
@ExceedLearning 2 жыл бұрын
Excel.Objects should return all the objects available in an excel file, including tables, sheets, defined names, and print areas. I suppose your error happens due to the custom function you invoke, so you should check it out
@DarrylBodkin
@DarrylBodkin 2 жыл бұрын
@@ExceedLearning correct I get the error when I invoke the custom function. The only thing that I didnt do like you for the custom function is to unpivot columns. Will try again.
@aparnac48
@aparnac48 Жыл бұрын
Hi, I tried this power query script, it works as shown with local data. But when I tried using SharePoint and OneDrive link as a path it's not working. It's giving me the following error- [DataFormat.Error] The supplied folder path must be a valid absolute path. Could you please help me how to connect this query to SharePoint folder?
@DanteCry23
@DanteCry23 Жыл бұрын
Did you find a solution?
@axelamoe
@axelamoe 2 жыл бұрын
Can you do this with PDF? its not working
Nested functions and Table.Buffer()
19:54
Exceed Learning
Рет қаралды 18 М.
Looks realistic #tiktok
00:22
Анастасия Тарасова
Рет қаралды 47 МЛН
OMG🤪 #tiktok #shorts #potapova_blog
00:50
Potapova_blog
Рет қаралды 18 МЛН
I Can't Believe We Did This...
00:38
Stokes Twins
Рет қаралды 81 МЛН
Always be more smart #shorts
00:32
Jin and Hattie
Рет қаралды 47 МЛН
How to SIMPLIFY DAX using Power Query
16:14
Access Analytic
Рет қаралды 13 М.
Efficiently extract Data Multiple Folders in Power BI
12:50
Low code No code
Рет қаралды 2,2 М.
List all the files in a folder and subfolders with Power Query
6:00
Excel Off The Grid
Рет қаралды 14 М.
Rename first N columns in Power Query
13:38
Exceed Learning
Рет қаралды 3,3 М.
How and why to Unpivot data with Power Query
16:40
Access Analytic
Рет қаралды 42 М.
List.Generate() Function and Looping in PowerQuery
23:56
Exceed Learning
Рет қаралды 24 М.
Looks realistic #tiktok
00:22
Анастасия Тарасова
Рет қаралды 47 МЛН