Imagine going through all this, then just telling the mapper to "rotate this one brush by 1°", and provide no context. He'd be flabbergasted

As both a programmer and a mapper, my guess is that it has to do with the Intel and AMD FPUs (floating point units). On old code, the two vendors can have a different epsilon (number so close to zero it rounds to zero). I'm 99% sure the brush was so close to parallel with the ray bouncing off it that dividing to get the difference in angles between them ended up being a divide by zero error on intel but not AMD, poisoning the color info with NaNs, throwing more and more of the map into darkness. If something like this happens again, you can confirm the problem by compiling the map with 1 or so light bounces and finding the dark patch in the world to get a clue where the problem lies. Also, to solve it faster in the future I'd recommend messing with the position of the lights, not the brushes, there's far fewer of them.

Piece of wood: _rotated by a millimeter_ Valve: "This little maneuver's gonna cost us 51 years"

>An error with map compilation >caused by one brush >rotated at this exact angle >in this exact place >as a func_detail >Localized entirely to Intel CPUs? >Yes. >May I see it?

Source engine likes to punish you, not because you did something wrong. But because it can.

I’d imagine valve is going through a different but similar idea of source fuckery with trying to get VSH to run in casual

She nan error on my Intel floating point unit til I *P A S T E S P E C I A L ! !*

Unironically now I want to join a Hardwood match and see the piece of wood for myself lmao. Like an attraction for tourists or something 😂.

And that, kids, is how you use the binary search algorithm.

Honestly great showcasing of how to find errors like this. Everyone that has a couple years of experience with Hammer had to do this at least once lol

how the fuck did a wooden wall being rotated at a specific angle cause the entire lighting to break lmao

Really puts into perspective how even just getting maps in involves more work than most people think

Oh my god, I remember this issue. Back in 2018 I kept posting this kind of "blacked out map" screenshots on your discord server. I remember you took my vmf and bruteforced it with cordon tool, just like seen here. I remember that you narrowed it down the a func_details that I made by skewing it. It was my lesson to never skew any brushes.

Compiling is such a fucked process, man. I don't know if this is true or not, but I've heard that an issue with Valve's compile is what led the Cashworks spawn bug to be possible.

My only possible consideration as to why this error happened, is that in the version of the compiler that is Intel-based, it was compiled to _slightly-different_ CPU instructions than the AMD version (think to the old 'small health pack gives 1 more health on Windows-hosted TF2 servers than linux-hosted servers because one uses a different multiply instruction' issue), and somehow the stray brush was causing the Intel-based light-bouncing solver which is specific to func_detail brushes (since they aren't world brushes and usually entities don't interact with lighting they're given special treatmenT) to FREAK THE HELL OUT and end up with an invalid number when trying to solve the light direction (which then proceeds to screw up the rest of the map because it bounces lights 100(!) times, which would lead to any light hitting a corrupted surface also becoming corrupted, because (any number) +-*/ NaN = NaN), but somehow the AMD solver doesn't because of it's different processing methods. That isn't so much Source-related spaghetti as it is just regular old code spaghetti, because programming just *adores* making such specific detailed bugs like this.

This is why I like to have all my world and detail geometry on-grid, even if its super tedious to make every vertex line up so no concave surfaces form. That being said, I've never run into any problems putting things off-grid, but even after 10 years of mapping source still scares me enough to not take chances.

I love how the solution to this problem was so simple but would have completely borked the update process because of how small it was. And like others have pointed out, this is a masterclass in using a binary search to solve an issue. Great video and presentation!

hammer is a piece of ancient technology that the space marines in 40k are terrified of uncovering

I'm so glad my dad has taught me how to troubleshoot problems, its the most useful skill i could possibly have because stuff like this happens so often

Just wanted you to know that I still come back to this almost a year later. Within the best debugging videos ever made

this looks a lot like a bug i had in a csgo map (free nuke) but it was caused by two displacements occupying the exact same space but with one flipped around

To be CPU specific, it's a math problem To only show up at specific angles, it's a float value being rounded, everything handles float differently; CPUs, OSs, programs, anything can change how rounding is done As for Intel vs AMD, it comes down to how many decimals they convert, I believe Intel goes 3 digits (maybe 2? This keeps changing over the years actually, so I wouldn't be able to tell for certain) further than AMD, I'm guessing when given really small float values, Intel will round way down where AMD has no other choice than round up For a more practical example, let's say the problematic angle was 0.000000001, Intel will keep the number as is since it can handle it, will give it to the program that will look at it and go "My math functions only handle 5 digits after 0, when parsing this reads [0.00000]", and give a bunch of NaN errors, meanwhile AMD will look at that value and say "Nope, too low, this is too ressource heavy, let's round it up to 0.00001" then give it to the program who will accept it without problems Someone probably commented on this already but at least I can say with confidence the Source spaghetti wasn't the culprit for once And if you're wondering why everyone and their mother is doing floating points differently, that's because everyone thinks their way of doing it is the better choice when in reality they all have issues, you saw what being too accurate can do, but you can probably guess why being less accurate would have its issues as well

I also have an Intel CPU on my PC, and I ran into this issue once while compiling a map. I also managed to fix it with similar methods, although I deleted the affected func_detail and remade it entirely. Great informational video, but one thing that I'd like to say; you should include a mention about the compile logs' GatherLight bounce stage "added RGB(nan, nan, nan)" in the description, so people who are experiencing the issue could find this video with a Google search and get a solution to it.

An angle of possibly less than one degree caused an entire map to break down... fucking hell source must be a nightmare.

I’ll never look at that wood the same ever again. Great work again Crash!

I guess it must have been a HARD wood to find

Good old Source Engine. Driving us insane since 2004.

Having had to go trough a similar process a few times, I've never seen such an obscure issue. Well trouble-shot!

I’m glad! Hardwood is my new favorite map, and it inspired me to get into mapping! I honestly find it the highlight of the summer update! (I still love selbyen)

I remembered to have similar issues about "wrong brushes" and "wrong map vertices" in normal goldsources maps that I made years ago. It made me stop working in Valve Hammer tool, because it's really annoying having unsolved problems into maps.

All this damn compiling makes live in-engine editing seems like a godsend

The method of searching for the problematic region is an excelent demonstration of binary search, bravo!

Peak Source Engine moment

That is definitely some kind of floating point precision bug, as "nan" is a value of a float, just an invalid float. I speculate that this might happen either because of some conversion somewhere (like from 64 bit precision to a 32) or simply a bug in intel's silicon, in which case... lol

Absolutely fascinating. Love videos like this. It's like the feeling of satisfaction you get from finally understanding a bug in your code without having to spend >3 hours debugging, lol.

I can really feel the nerdy love-hate relationship you have with TF2 and it's Source engine jank. Your passion for the game is infectious. TF2 wouldn't be the same without people like you dude.

Would LOVE to see more stuff like this- seeing solutions to weird obscure problems in hammer/source, it’s so interesting

tl;dr: Map was too hardwood. Love not only the explanation but going over the troubleshooting process. (Bug hunting is the only part of programming I can stand anymore.) I'm reminded of the parable of the printer technician. To paraphrase for those who don't know: Newspaper printer does an ope, techs can't figure it out, they call in a specialist. Specialist looks at the machine for 30 minutes, turns one screw three rotations, printer works again. Newspaper billed $5k and, when they balk at the $, specialist replies "$100 for the 30 minutes I was there, the rest for knowing what screw to turn and how much."

Troubleshooting is an underrated skill, good job bruv

Such an amazing explanation to an obscure problem. Marvelous work¡

It's nice seeing how things work

see, this is what happens when you have an engine from 2004 be held together by gorilla glue and duct tape. (as a mapper, i love yet hate source so much)

"wow, i love this map! i cant believe valve chose the map!" floating point numbers:

Crash: I did it, I save the world.

That's incredible man.. Troubleshooting such an obscure issue, sucessfully! Props mate, thanks for documenting it too

Such a weird problem with such a weird problem. Glad you found the solution!

I am really glad I tend to make maps more aligned to the grid, I would have never been able to figure this one out. Source is wild, man.

Wow almost like they should add Farmed and Brine to the Summer update maps (please 😢 (also Extinction (please!!))) Also Hardwood still crashes the game for me and my buddy on occasion - haven't been able to play on it for longer than 10 minutes total I'd say. Regardless of all of that, I love how you break down your thought process here. This kind of problem-solving is, I feel, the bread and butter to a lot of computer science applications. Cool to hear about it step by step!

I think LazyPurple put it best, “THAT’S BECAUSE IT MAKES NO FUCKING SENSEEEEEEE”

achieved with Sorse ✨

Thank you for servicing and finishing Hard Wood.

very cool insight to a mapping issue, this was a very good watch

I loved this video. Watching experts troubleshoot source engine jank is always lovely to see :)

Thank you for all your sanity breaking work Crash! Maybe take a mental break for a few days.

Good video as always. Ezcited to see what maps you su mit for this scream fortress, which I'm actually excited about for once

Interesting i remember having trouble with this all blacked out map error thing in the past.

would love to see a map tier list vid of yours about all the maps in tf2 (halloween maps etc included) would maybe seem interesting to hear your opinion on all maps in the game

I think it might be caused by a collision/input check outputting more than the native kernal of Intel can handle in the backend of reading he program through hammer, thus removing the oldest output in order to process the map. Oldest output being the lights.

I don't think even shounic could unfurl this knot of spaghetti. Great save dude!

i will laugh taunt at that fence brush

I encounter a bug like this nearly every single time I try to finalize a map. And it's ALWAYS a weird func_detail brush I have to remove/replace/modify in order to get it working again. It sometimes represents differently, sometimes the lighting breaks, sometimes the map won't compile, sometimes certain walls won't render or will render as an eldritch mess of polygons instead of a brush, but it's always solved using the exact same method you described above. Hell, sometimes the cordon trick doesn't even work because it's like, a T-junction error that only happens when a certain number of func_detail brushes intersect/touch non-func_detail brushes! At this point I usually start by seeing if the issue persists with only func_detail active since I have such a bad track record with those guys. I was actually hoping an established mapping figure like yourself would have learned some magic shortcut for this issue, but I guess I was debugging it correctly this entire time. Ah, the life of a source mapper. Pain.

Thank you for your work

I was hoping the conclusion would explain why it happens. I didn't know this engine was held together with duct tape and dreams. Also, congrats on describing "debugging". Welcome to the club!

Dude this rocks. Would love to see more Crash Case files

I really liked this video not only because of the whole story around it, but because it shows that sometimes "all" you have to do is go through 4 hours of simple testing and testing and testing to find the tiniest error and the simplest possible solution. By no means did you need to be a rocket scientist to do any of this. But ho boy did you need an admirable amount of patience, dedication and passion to narrow the issue half by half by half until you honed in on a fraction of a fraction of the map. Nice job dude. Thank you for doing so much for this community.

Crash, you're the MAN! Cheers pal!

This error happened on my map, koth_spacebase(check my steam workshop)! I was checking every brush just to find that one of my ramps was off-grid. Fixed it and bam! lighting fixed! The issue still haunts me. had no clue it was an Intel thing.

This video is actually fantastic.

Duplicate grouped brushes made into a function have caused me problems. My solution was to ungroup and then re-group anything duplicated just incase it became a function.

Mappers like you are greatly appreciated in TF2,just letting that one out.

the jank is real on this one

What an incredible journey, thank you for sharing it 🙏 My theory is that the Intel vs AMD difference is because Source is using some weird pipe for rounding things at angles, and Intel has different behaviors than AMD for that edge case. Maybe an overflow? @lowlevellearning this might be up your alley

Nothing like the satisfaction of finally fixing some super obscure issue that happens for absolutely unknown to you reasons.

Compilepal sometimes does not show errors that lead to some stupid errors or even making it impossible to compile the map

You know, I wonder if this is partially the result of having been made in Hammer++, which retains the full floating-point coordinates of off-grid vertices instead of rounding them. In fact, now I wonder what would happen if you loaded the unfixed map in OG Hammer, re-saved it, closed it, re-loaded it, and compiled it.

Great Video!

I was prepared for anything, but this exceeted everything I would've imagined

Source is probably this way because it's a modified version of a modified version of quake

UEAKCrash has revived!

holy shit an error so so specific caused so much damage godamm source

Please do more of these.

Once located, I would try working on the intersection with other brushes, but who knows if that would've helped.

Great work. Now, to figure out why half the dam of Phoenix, just about every single prop of Sulfur, and the vault door of Cashworks occasionally just fail to load at all.

The only thing I can think of is a division by zero error causing the NAN, but I'm not satisfied with that.

This gives me an idea to make a program to show the position of the brush causing a nan error when compiling. If a vmf of the troublesome brush could be made public, it could be used to detect those edge case bugs.

Great video

And this is why, kids, you always ask help with mapping issues because only Gaben knows how convoluted Source is.

Hardwood's lightning literally have a stroke over a single beam of wood being 3 degrees off.

Still bummed Shoreleave and Overgrown didn't make it. I also liked your fishery map too.

good vid im really sad there wasn't an actual explanation on why this fence at a specific angle was causing compiler errors on intel

I feel like the rounding error is probably the most likely one. As a rule, if you're gonna rotate anything in source, try and round its rotation to a full integer of degrees, since Source *really* doesn't double-check its math as much as it should, and it *REALLY* doesn't like decimals. Rescaling displacements, rotating func_details, all of it is gonna give you grief later on.

thank you crash for such bedtime story

back when i still used hammer, i always snapped vertices to the smallest grid when moving things off angle, not because of any game breaking issues, but because the objects would look more pleasing. Would've saved me in this case.

The Hardwood map looks really beautiful. I'm glad it was able to be fixed so it could be included.

Absolutely wild.

Superb video

Buff GoldSource Engine: "Modders can use me to make any kind of game they can possibly imagine!" Cheem Source Engine: "I can't render the lights because of the angle of this func detail."

Wow that a lot of work. It just a guess over lapping vertices points of the mesh vertex must have bug out since it baking. Not sure does merge all mesh?

13:15 that left ear clicking man.

when a simple angle on a func_detail is causing so much trouble...