Processing Large XML Wikipedia Dumps that won't fit in RAM in Python without Spark

Рет қаралды 14,203

Күн бұрын

The Python ElementTree object allows you to read any sized XML that you have time to process. Unlike a DOM the entire XML document does not need to be loaded. This video shows how the entire of Wikipedia can be processed without a large amount of RAM in Python.
My blog post for this video:
www.heatonresearch.com/2017/0...
The code for this video can be found here:
github.com/jeffheaton/present...

Пікірлер: 26

@opalkabert 4 жыл бұрын

I am not just liking this but want to thank you for your time to show this. It is awesome Jeff!

@biologyigcse 4 жыл бұрын

As a person who is just starting out in the the research domain and have to work with wiki dumps, this was a god send. THANKS a ton, you just saved me tons of time and mental stress. Did I say thanks yet. THANKS A TON. You sir, get a like, subscribe, notification enabling and I am sharing your channel on my twitter space.

@sadiko3000 4 жыл бұрын

I took a look at the content of your channel and it is very impressive. Please keep doing this!

@mariagraetsch3700 4 жыл бұрын

Thank you Jeff - your video provides a really structured example.

@DanielWeikert 4 жыл бұрын

Thanks a lot for your videos. Love to see more on how to deal with big data in python. Best regards

@BiancaAguglia 4 жыл бұрын

Thank you for another great video, Jeff. Not only is it useful but, as the zombie apocalypse **has** been on my mind lately, it is also very timely. 😁 As others have already commented, I also think it would be nice to see the same process in spark. Keep up the great work.

@woetotheconquered3451 2 жыл бұрын

You're amazing. Just what I needed

@mariumbegum7325 Жыл бұрын

Interesting video, keep it up!

@tonym5857 4 жыл бұрын

* stars video 👏👏👏. It would be nice to see the same process using big data tech like hdsf, spark, etc.

@paulowiz 3 жыл бұрын

I'm a beginner about that I will try this code after the file download =). Thanks for it

@lisanoorarida4009 4 жыл бұрын

Thank you so much. I am working on this right now. For the output, I need to generate a new XML file after filtering the wiki. I tried to use the modul but they said "ElementTree is not a streaming writer". What do you recommend?

@HeatonResearch 4 жыл бұрын

I have seen lxml used for that before, but have not done it myself.

@nonenogood Жыл бұрын

Hello Mr. Heaton. I wonder, can we get the 'text' data from the dataset into csv too?

@quackcharge 3 жыл бұрын

thanks so much!

@RollingcoleW Жыл бұрын

Helpful !

@saleem801 4 жыл бұрын

Has a spark implementation been made since?

@rohitreddy3609 3 жыл бұрын

Thank you for this amazing tutorial. It's very informative. Can you please explain how to create a dataset of topics from Wikipedia dump, say to retrieve 100 topics for eg.? My question is, how we can crawl Wikipedia to get documents and images? Thanks in advance.

@victoriar8179 4 жыл бұрын

thanks for the video! would be awesome to have this to process with spark

@HeatonResearch 4 жыл бұрын

Yes, that is coming. Once you start to add any NLP functions on that Wikipedia text the process can take weeks without Spark.

@tamastarisnyas1191 3 жыл бұрын

Hi there, thank you for the video, but there's an issue, namely when I use your code it won't fill the redirect column for some reason. Could you help me with this problem?

@HeatonResearch 3 жыл бұрын

Let me have a look at that!

@tamastarisnyas1191 3 жыл бұрын

@@HeatonResearch and another thing that i wanted to do is to grab the text of each article and connect it to the table as a separate column for each title. Could you give me some pointers or tips on how I can do this, please? Would help a lot. Been trying to do it, but it without success.