*Video summary* This video uses as a motivating example a bug that fasterthanlime once ran into when the feature flag management software he was using (LaunchDarkly) read the staging-environment feature flag values in production. That caused all experimental code to be turned on in production, which led to many customers being affected by bugs in the experimental code. After talking about the difficulty of debugging this, fasterthanlime talks about how it’s important to prevent errors automatically and not rely on humans to configure things correctly. He says this particular issue is unusually hard to prevent automatically. He notes that the error could have been avoided if the LaunchDarkly API didn’t conflate the API key and the environment, but that LaunchDarkly would probably not be willing to change the API now. He eventually concludes that the easiest way to prevent that bug is to create a new feature flag called “environment” that just duplicates the name of the environment these feature flag values are for, then assert on application startup that the application environment matches the feature flag environment.

Team strict! I used to be on team "fast production" but found I kept screwing myself with weird bugs because I shadowed a variable or did some other Dumb Thing that Rust would never let me compile past. Now when I load old code of mine in Rust, I can add on to it fearlessly and things work. My >10k LOC Python projects? Yek. Rather rebuild it.

if you're storing the key in an encrypted file but one that a user can decrypt as part of a review, prepend the API key with 'prod-', 'stag-', 'deve-', etc. (as long as the length is the same) and strip off the first 5 characters before passing it to the launch darkly, and now you don't need a separate flag in launch darkly. Then, as part of the checklist for changes to that file, require the reviewer to verify the environment the key is prepended with matches the environment that key belongs to in launch darkly. It's still not automatic, but it's a start. You can then use that launch darkly flag with different values set for each environment (you can also use environment variables if you use those outside of launch darkly, too), and as soon as the launch darkly client initializes, check that key against the string you stripped off of the api key. And if they don't match, the program should die immediately. This way there's a review step that should catch the incorrect key, but even if that fails, there's the backstop of the start-up check. In order for that to fail, not only does the code author have to change the key, they have to change the prepended string to match the correct environment for the api key AND the reviewer has to miss that the prepended string in the file changed in addition to the key itself. At that point i think you're well into the grey are between accidentally setting the wrong key and either doing it on purpose or general atmosphere of negligence.

Love your talking head! Everyone should support you on Patreon.

Could create a network proxy inside your private pre-production network for SAAS products you use. If you can do this, then you'll hopefully see immediate failures if you accidentally try using them in production (since the hosts wouldn't resolve) outside your private network.

On the LaunchDarkly issue, an idea that came to me was to create a sort of canary flag - some piece of code that would fail loudly/warn someone if ever run. But then you would have to disable it for local dev/staging anyway, so I guess it's somewhat equivalent to your environment flag idea

"Number + array is legal" as an APLer I resent this remark XD

Amos in this video is the most anybody has ever looked like Daniel Armbruster, the lead singer of Rochester, New York rock band Joywave

Great video, thanks! But I can't help thinking its got two parts that might not be as related as they could be. I'm on team strict all the way, and I love the rust language, but I don't really see how that relates to the launchdarkley issue ... Unless, like ... you write a launchdarkley client api in rust, that makes it very easy to check feature flags once it is initialized, but that requires both the key and the name of the environment to initialize. So, ok, maybe the two parts of the video are related after all, but you left it pretty much to the viewer to find the relation. Which may be part of why your video posts are shorter than your text posts. :-)