37C3 - Reconstructing game footage from a Game Boy's memory bus

No video

37C3 - Reconstructing game footage from a Game Boy's memory bus

Рет қаралды 4,422

media.ccc.de

Күн бұрын

media.ccc.de/v...
The GB Interceptor
How do you capture a video from an 1989's Game Boy without modding the original hardware? With an adapter cartridge that spies on the memory bus!
Let's talk about how to reconstruct the Game Boy's memory state, emulate its graphics unit and then encode the image into an MJPEG stream for anyone to use as a USB video class device. In realtime. On an rp2040 microcontroller.
The original goal of the open source project "GB Interceptor" was to capture gameplay for one specific game: Tetris. In order to live stream a Tetris tournaments from the contestant's personal Game Boys, the idea was to create an adapter that goes between the Game Boy and the game module to analyze the communication on the memory bus and reconstruct the game state.
It turns out that it is actually possible to reconstruct the entire memory state of almost any game and in fact create an rp2040-based adapter that acts as a USB video class device offering the on-screen game footage in realtime. Players can simply put this adapter into their Game Boy and use it like a webcam without additional drivers or knowledge.
An essential aspect of this concept is that the Game Boy basically runs all of its code directly from the ROM module, which makes it possible to directly follow the program counter of its 8bit CPU regardless of how the code branches. An image can then be recreated by emulating the graphics unit (PPU).
However, there are many edge cases like interrupts, data from registers that are not visible on the bus, the link cable, DMA operations, synchronization of CPU and PPU, game bugs and even bugs in the Game Boy hardware itself.
In this talk I will show how all this is done just on an rp2040 with spare cycles to encode everything as a 60fps MJPEG stream. I will shine a light on the edge cases - those that were solved and those that might just be unsolvable with this approach. And I will take you on a sightseeing tour through the 8bit hell that drives our iconic handheld from 1989.
Sebastian Staacks
events.ccc.de/...
#37c3 #HardwareMaking

Пікірлер: 11

@ThereOughtaBe 6 ай бұрын

Hey, this is Sebastian, hope you enjoyed my talk. I would like to add a few more resources, but I think KZfaq has removed my previous comment - probably because it was recognized as something that might divert traffic. So, instead of referring to an online version of the discrete cosine transform visualization from the talk and the exact 5bit encoding process, I can only elaborate on the latency question at the end of the talk: I have now done a screen-to-screen latency test where I got about 60ms. However, keep in mind that this will depend on several factors like the operating system, buffering of the video software as well as independent refreshes and input lag of the screen of the host system. Let me know if you have any more questions :)

@blinddarm8478 6 ай бұрын

Danke für den talk, war grandios. Inhaltlich und der Vortrag selbst!

@Asdayasman 6 ай бұрын

Sure I've got a question. Do you prefer left or right Twix?

@ThereOughtaBe 3 ай бұрын

@@Asdayasman I have been thinking about this for a few months now and tried a few things, but picking either always left me a bit disappointed. Then I tried something truly revolutionary: I took both Twix, cut off one end on each of them and put both of them together by slightly melting the chocolate at the end and applying a tiny amount of molten chocolate to the now open edge (not too much - you don't want to change the ratio of chocolate to caramel - that alters your taste experience). This results in a single long Twix. Now, here is the really important part: You take the united double-length Twix and put it into a salad spinner. Spin it generously (not too fast, though, or it might break). This does not change the Twix, but it changes you! As you have no idea anymore which half of the double-length Twix was originally the left and right Twix, you are free of any directionism. Any prejudice you might have had against either side is gone and you can enjoy the delicious bar without any autosuggestive flavor alteration.

@dameanvil 6 ай бұрын

00:17 🎮 Physicist Sebastian created the GB Interceptor, a USB video adapter for a 1989 Game Boy, allowing video capture without modifying the original hardware. 01:40 🕹 The GB Interceptor project enables recording and streaming from unmodified Game Boys, catering to Retro Gaming enthusiasts who wish to preserve the original gaming experience. 03:20 🎮 The Game Boy's unique memory bus allows direct observation of CPU instructions, making it possible to reconstruct game footage without loading code into RAM first. 09:23 ⚙ The GB Interceptor hardware is an open-source project, utilizing a Raspberry Pi Pico's rp2040 microcontroller with added bus transceivers for direct connection to the Game Boy's memory bus. 13:45 📹 The GB Interceptor functions as a USB video class device, providing plug-and-play compatibility with various systems for easy recording and streaming without the need for additional drivers. 15:32 🧠 The software tasks include recreating the Game Boy's video RAM (vram) from observed bus events, emulating the Graphics Processing Unit (PPU) for image rendering, and handling MJPEG encoding for video output. 18:56 💻 Sebastian explains the process of observing Game Boy CPU instructions, decoding opcodes, and mimicking the actions on the GB Interceptor to recreate the vram, essential for accurate video reproduction. 19:38 🎮 Game Boy memory bus decoding involves identifying op codes, data, and junk, relying on Game Boy's decisions to avoid wrong emulation decisions. 20:20 🔄 Conditional jumps in Game Boy assembly pose challenges due to varying cycle counts, requiring observation of addresses to determine jump outcomes. 22:35 🎰 The GB Interceptor struggles with invisible data, like the divider register, affecting games that use it for random number generation, such as Tetris. 24:56 🕹 Multiplayer Link Cable interactions pose challenges; if not controlling, data isn't visible to the GB Interceptor, impacting games like Tetris. 25:53 🔲 Gamepad input detection is indirect; GB Interceptor follows executed code triggered by button presses, with some exceptions requiring game-specific fixes. 26:48 ⚠ Interrupts, RAM code execution, and tricky DMA transfers are challenges, along with various edge cases, in capturing accurate Game Boy video. 30:31 🖼 Graphics emulation involves precise timing, distinguishing between V-blank and H-blank periods, with different levels of complexity based on when games make updates. 36:53 🔄 Game-specific fixes, hash database, and recognizing Loops aid in handling cases where standard methods fail in determining game state during video rendering. 38:55 🎥 The Game Boy runs at 60 FPS with a resolution of 160x144 pixels and 2 bits per pixel. 39:35 🖼 Uncompressed video faced challenges due to color formats, impacting the frame rate. 41:39 🔄 Frame blending at 29 FPS caused transparency effects to look odd in certain games. 42:06 📷 Explains the five steps to encode an MJPEG image and the challenges involved. 45:22 🔍 Upscaling the image by a factor of eight enabled a compression ratio of 5 bits per pixel. 48:54 🌈 Two modes for the GB Interceptor: 30 FPS with a color channel and 60 FPS grayscale mode. 51:02 🕹 Compatibility with Nintendo devices that support original Game Boy games (grayscale). 53:47 📊 The 60 FPS version provides a lossless capture result on the host machine. 55:52 🕹 Resource limitations prevent automatic correction of all game-specific glitches. 56:21 💻 Presentation slides were created using a text editor and presented in a web browser. 57:56 🧠 Chain DMA can be useful for tasks like lookup tables, but there's limited room for additional improvements in the GB Interceptor. 58:34 🎮 Game Boy Advance doesn't support the discussed technique; it's a solution more suitable for traditional systems. 58:46 ⏱ Video latency details are not readily available, but it might not be ideal for fast-action games preferred by retro enthusiasts. 59:42 🌐 The GB Interceptor's streaming technique is specific to Game Boy and grayscale images, with no known similar projects for other video formats.

@MeriaDuck 6 ай бұрын

This is Michael Steil level of deep dive! Loved it. The MJPEG encoding tricks were phenomenal. The idea to treat each pixel as 8x8 block making everything quite trivial and effectively lossless is a stroke of genius 🤯

@mantrayana6055 6 ай бұрын

Holy shit, I've rarely seen such good presentation slides and animations. I'd like to see more!

@asdmursu98 6 ай бұрын

wow that was amazing presentation! i mean the presentation itself were super informative and pleasant to follow. absolutely fascinating work !

@der_jonny 6 ай бұрын

Great talk. As a big fan of the classic gameboy i appreciate every minute you spend by analyzing this great console. inspiring work. thank you. this shows me that there are still some retro gamers out there.

@lassdasi 6 ай бұрын

What software did he use to create the live camera feed in the presentation? Edit: In 56:09 someone asked this question. @ThereOughtaBe did you make the JavaScript stuff by yourself or is there some public library for the animations and stuff?

@ThereOughtaBe 6 ай бұрын

So, yes, the camera stream uses the MediaStream Web API to show the two video streams (GB Interceptor and the camera through a cheap HDMI grabber) in HTML5 video elements. There actually is not that much JavaScript in there. The JPEG demo might already be one of the larger parts as most of the transitions and animations are just CSS. The code is mine, but the way it works is heavily inspired by impress.js - just a more basic version with elements like slide numbers and the chapter transitions that I use regularly. I basically assign the class "step" to any HTML element that is supposed to be a step in the presentation (could be anything, but it is mostly what you would call a slide). The script iterates over all step elements and assigns additional classes to them like "before", "after" and "active". Pressing forward and backward literally just increments or decrements an index which determines the "active" element. The rest is done by styling the the .before and .after versions with CSS and enabling transitions. With a bit of CSS experience it is not too time-consuming.