I have updated the video description to contain a link to @SQLBalls Github repo with the sample code we can currently distribute.
@DanielWeikert Жыл бұрын
Any reason to store data as files? You can't do anything with them. SQL Endpoint, Power Bi access,.. all require delta tables. Any thoughts? br
@Tales-from-the-Field Жыл бұрын
Paging @SQLBalls!
@SQLBalls Жыл бұрын
I tried to post earlier but it did not post, lets see if it does this time. I think it would depend on the scenario. In the past when I used Beutiful Soup to scrape data I was storing it in flat files in blob, then consuming them in a pipeline, then loading them to SQL, eventually I wrote a python pipeline to insert directly into SQL but it wasn't super efficient because it must use a cursor. Fabric really changes that. Where this was useful for me, was when asked to create a sample data set I opened up OneLake and copied it right over to my desktop and uploaded it to GitHub! I wanted to make sure I showed that both options were possible, but it would really depend on the scenario.
@GuillaumeBerthier Жыл бұрын
Great video again, would you be willing to share your Notebook so we can start with a baseline in our Fabric? Thanks
@Tales-from-the-Field Жыл бұрын
Hi @GuillaumeBerthier, paging @SQLBalls!
@SQLBalls Жыл бұрын
Hi @GuillaumeBerthier, Ok this could be a long post. Let me start with what I CAN do, I've uploaded an edited version of the notebook as well as a csv file with sample data. It is not legal to scrape data from a website where you do not have permission to, or if you intend to distribute the data or use it for profit. Unless the owner says it's ok. I need to make that very clear. I *cannot* give out the code to the scraper at this time, because while it is ok for me I cannot give that permission to the world at large. However, I will build a sample web page and next week I will make a video along the lines of "how to use Beautiful Soup to scrape data". Then I will repost the notebook using the code to scrape my website's data, which I can give people permission to do! I hope this helps!