I REGRET Not Telling Dave Farley THIS about Approval Testing

Рет қаралды 3,156

Күн бұрын

Dave Farley knows a lot of stuff and I really enjoyed our recent Engineering Room conversation! On Approval testing though, Dave still doesn't get it. This video is what I wish I had been able to say at the time to persuade him. I also show a demo of some actual approval testing.
▬ Links ▬▬▬▬▬▬▬▬▬▬
Dave and my Engineering Room conversation
➡️ • All Things AUTOMATED T...
Dave Farley's video on Approval testing
➡️ • Add APPROVAL TESTING T...
Dave Farley's KZfaq Channel
➡️ youtube.com/@ContinuousDelive...
CODE EXAMPLE MATERIAL
Description of the Vending Machine Kata
➡️ sammancoaching.org/kata_descr...
My demo code
➡️ github.com/emilybache/Vending...
in the 'demo_dave' branch
Verify - approval testing tool in the demo
➡️ github.com/VerifyTests/Verify/
Analysis tool
➡️ github.com/emilybache/Approva...
PATREON & SOCIAL MEDIA
If you like this content and want to support this and all my Katas on GitHub, please join my Patreon ➡️ / emilybache
LinkedIn ➡️ / emilybache
Mastodon ➡️ sw-development-is.social/web/...
X (Twitter) ➡️ / emilybache
▬ Video chapters ▬▬▬▬▬▬▬▬▬▬
00:00 - Introduction
01:17 - Emily Explains Approval Testing
04:00 - Code Example: Vending Machine Kata
15:06 - Conclusions
#softwareengineer #approvaltesting #davefarley #softwaredeveloper

Пікірлер: 45

@craigjbass 9 ай бұрын

This is great. It never occurred to me to approach approval tests this way.

@EmilyBache-tech-coach 9 ай бұрын

I'm encouraged you think so.

@marco.garofalo 9 ай бұрын

I found your videos very well explained and well thought, I appreciated your take on using approval testings as a TDD/BDD tool, that said it didn't seem to be very practical. The main problem, for me at least, is that approval testing forces you to jump in and out the tests to understand input and outputs/expectations, disrupting the TDD/BDD cycles and adding a bit of a slowdown, whereas a more traditional TDD/BDD approach would just have those co-located. Similarly, when I try to understand a new piece of code I go look into the test suite, which in a case where approval testing was used, would still give me a partial understanding/view. Beside that I really enjoyed the exercise, also because challenging our assumptions is what makes us learn, so thanks for sharing your experience and expertise with us.

@EmilyBache-tech-coach 9 ай бұрын

Thankyou for your thoughtful response, I'm happy you're willing to have your assumptions challenged! A couple of things occur to me - I don't find this 'jumping in and out of the tests' to be a real problem - I can understand a test failure usually just from the diff, often don't need to read the code. Secondly, the BDD cycle with a tool like Cucumber has a similar 'jump' between the feature file and the step definitions.

@TheFoxesTale 9 ай бұрын

What an amazing example, this opens up a whole new world of possibilities for my complex integrated system testing. Subscribed right away!

@EmilyBache-tech-coach 9 ай бұрын

Yes, sounds relevant for your situation. I hope it goes well!

@danielirvine 9 ай бұрын

I enjoyed watching your walkthrough of this technique - it was illuminating! I often mix up specific names of testing practices because my brain just doesn't work like that - I tend to just do whatever I think will help in that particular moment without necessarily thinking about what is that I'm doing! E.g. the difference between TDD and BDD was never a huge deal for me, but I get that for many people it's important to differentiate between the two. I've also grappled with the problem of self-referential tests, but I never stopped to think about a process that avoided that problem. Now I know that there's a way :) Thanks!

@EmilyBache-tech-coach 9 ай бұрын

Thankyou! I do find a demo does a lot to clear things up, especially when terms like TDD and BDD may not be terribly meaningful to people.

@skipodap1 9 ай бұрын

Great Video!

@EmilyBache-tech-coach 9 ай бұрын

Thanks!

@JhanaratoBhikkhu 11 күн бұрын

As an exercise, I used an approval test with the Leaderboard Kata in Python. I pretty much did what you did in this video, in my case implementing __str__ on the Leaderboard class. The main issue I had was the lack of integration with PyCharm. I had to manually approve files. Not a deal breaker though.

@EmilyBache-tech-coach 11 күн бұрын

That sounds like a good approach! I agree better tool support would help.

@jangohemmes352 7 ай бұрын

I was lucky enough to win the privilege to host a Patreon Q&A session with Dave, and this video came up. I told Dave about the video and tried to play the role from your side of the argument. That was tricky, as I myself am not quite convinced yet. The main thing in this video specifically is that I think the example is just too simply converted into regular asserts inside regular tests. The same kind of sketching and modelling process of expected output can still be done, and to my mind it's better than leaving the IDE to tinker in text files. What I hope to have gotten across to Dave though, is that you really do model expected output before running the test, and are not simply accepting whatever the code produced. (In his words: "testing the code that you wrote is the code that you wrote")

@EmilyBache-tech-coach 7 ай бұрын

That's really cool, thankyou for being willing to play the role from my side of the argument and talk to Dave about this topic. If he's started to understand that these tests are not 'self referential' then that's a big win! Your other argument here is that you could just use regular asserts and not leave the IDE and that this is a better workflow. I disagree, but perhaps that's a discussion for another day :-)

@larserikaa 9 ай бұрын

I've certainly suffered from approving too large XML, JSON or manual prints. No way you can approve them safely when too big. But your usage of printers are just super exemplary. And I thoroughly love the story abstraction. If Dave doesn't change his mind now, I don't know who can make him. 😁👌

@EmilyBache-tech-coach 9 ай бұрын

Happy to hear you recognize both the problem and the solutions I showed.

@banatibor83 9 ай бұрын

Hi Emily, I enjoyed the engineering room interview very much! I think you both are talking about the same thing under different names. I commented this on Dave's video but you both must have missed it :) Approval testing and Acceptance testing are both done in a BDD way, but done at different points of the software lifecycle. Approval test = safety net for refactoring Acceptance test = high level driving test for new functionality The tricky part is to realize when approval tests are used for adding new functionality they are called acceptance test. Hope this helps to smooth out the differences between you and Dave :)

@EmilyBache-tech-coach 9 ай бұрын

I'm glad you enjoyed listening to Dave and my conversation in the Engineering room. The terminology around this is a little bit involved, and it's good to point out the different points in the software lifecycle where different words make more or less sense. However I'm not sure I fully agree with your analysis here :-) I hope we'll continue a constructive dialog - with Dave and you and others.

@QmunkE 9 ай бұрын

Although you clearly can use approval testing with code like this as demonstrated, I think really the core of the argument is more whether it's worth doing in preference to (or in addition to) an xUnit style test. In cases where you already have good "traditional" unit tests in place, this kind of approach feels like it could add significant maintenance overhead to your tests and potentially couples unrelated tests to your implementation. On the flip side, I can also see that this allows you to spot cases where otherwise your test coverage might no longer be sufficient - adding a returns tray and verifying a coin doesn't get added to both the balance and the return is a useful test case you might not otherwise have!

@TheLucferreira 9 ай бұрын

I dont know if I got it wrong but I thought that by "legacy code" she meant "code that exists but that is not tested". In this case, often the code is also very difficult to test with "traditional" unit tests, and then approval testing can help a lot. In scenarios outside of that definition of legacy code, there is still parts that are difficult to test using xUnit style tests, like parts that involves what is displayed to the user, and I believe approval testing can also help in this case. Of course, other strategies do exist.

@EmilyBache-tech-coach 9 ай бұрын

If your code already has good xUnit style "traditional" unit tests then you are in a good position to begin with. Approval testing can help when you want to test bigger pieces, like a whole microservice or subsystem, and you would otherwise want to assert several things. Approval testing is also useful for code that doesn't have tests - legacy code - and that's where Dave and I do agree.

@jfmengels 9 ай бұрын

Thank you for the video! When/why would you choose to use approval testing instead of "more regular" tests in new code? For instance, these tests you have shown could have been equality assertions against strings, such as `assert.equal(print(actual), "VendingMachine Display ...")` (let's imagine that the language you're working with supports multi-line strings that makes them pretty easy to view, just like you view your files). Is it mostly to speed up the time to update the tests with new results, as tools don't tend to modify your test code? Having the results in a file makes it harder (and more error-prone) to find the the result of the test, which makes it less usable as documentation, no?

@EmilyBache-tech-coach 9 ай бұрын

Keeping the approved text in a separate file from the sourcecode has both advantages and disadvantages, as you point out. If the text is long, then it makes your test less readable to keep it mixed up in the sourcecode. This would also impact its value as documentation. I prefer to create tests where I can understand what the test does by looking at the approved file, without needing to look at the actual test code very much.

@UserDirk 9 ай бұрын

@@EmilyBache-tech-coachI'm still struggling to get a sense of when to apply the Approval Tests. You mention "I prefer to create tests where I can understand what the test does by looking at the approved file, without needing to look at the actual test code very much.". But this somehow does not resonate with me. Could you elaborate on this perhaps?

@EmilyBache-tech-coach 9 ай бұрын

@@UserDirk This is a great question and I don't think I can do it justice in a short comment here. I will make another video and try to address this better 🙂 Let me just say that I prefer Approval tests for larger-scale tests for bigger pieces of functionality, that need to check more than one thing. The approved file shows you all the important things that happened.

@UserDirk 9 ай бұрын

@@EmilyBache-tech-coach Thanks, looking forward to it!

@pierre-antoineguillaume98 9 ай бұрын

I like what you describe, in the end the printed results are less cluttered than their code counterparts. Good luck for your channel, I have shared it with my lead coworkers ! Thanks your your work, we use the kata of your repository in php !

@EmilyBache-tech-coach 9 ай бұрын

I'm happy to hear that! I hope your lead coworkers find it useful.

@PeterGfader 9 ай бұрын

13:15 Cool to see this helper tool in action. Maybe something that IntelliJ/VS will incorporate in the future. PS: I hope the teams behind these tools would focus more on "testing features" and less on "debugging features" :/

@EmilyBache-tech-coach 9 ай бұрын

Yes, I think there is a lot of scope for the toolmakers to better support approval testing, and developer testing in general. You can do quite a bit with plugins too - I think this analyze script could be a plugin in future, and that is quite achievable. Just need to find the time...

@PeterGfader 9 ай бұрын

Very cool. THANKS! @@EmilyBache-tech-coach

@haradan8887 9 ай бұрын

So this is just a different way of making assertions you can use for any automated tests? It seems like there's a potential disadvantage in needing to re-approve old tests. Say you add a new field to an object you're asserting on. If you assert on object equality, you can update your assertions implicitly by setting a default value for a new field. If you use approval testing, you need to either re-approve all your tests with the new field, or set up your printer to avoid the old tests printing that field at all, or avoid printing it if it has the default value. Do you think it's an extra effort of design to get that right? Do you default to one way or the other?

@EmilyBache-tech-coach 9 ай бұрын

Yes Approval testing is an alternative way to do assertions. Re-approving old tests is not a particularly big issue if you have many that have changed in the same way - with tool support it goes very quickly. You are exchanging the complexity of writing assertions with the complexity of writing a printer. I don't think the amount of code needed is very different, but the failures are easier to understand in approval testing.

@haradan8887 9 ай бұрын

@@EmilyBache-tech-coach Thank you so much. I'm so often frustrated by awkward output from an assertion failure. It sounds like this can solve a lot of problems.

@BementalSea 9 ай бұрын

An observation/question about your approach to TDD. At around minute 10:00, with all tests passing, you say "the next step is to write some production code" which you proceed to do. And that makes a test fail. So we have: tests green->add new production code->tests fail. Doesn't this violate Uncle Bob's first rule of TDD? "Write production code only to pass a failing test". It should be: tests red->add new production code, tests pass. You repeated this pattern several times. My TDD spidey sense was tingling! This raises a question for me: does approval testing itself make it difficult to do TDD? Or was this backward approach just an accident?

@JohnWilson-xl3rl 9 ай бұрын

This is a good observation and actually this process is much more similar to a TCR cycle than TDD but instead of auto reverting you are making an active choice to accept.

@EmilyBache-tech-coach 9 ай бұрын

I agree this is an interesting difference in approach to TDD. I think the essence of TDD is breaking work into small pieces and very frequently getting feedback on whether your implementation currently matches your expectations. So no, this green-red sequence wasn't an accident. Approval tests tell you what your code currently does compared with what you previously approved, and allows you to easily update your test's expectations. I'm interested in @johnwilson-xl3r's observation that it's more similar to TCR. I hadn't thought about it like that.

@craigjbass 9 ай бұрын

@@EmilyBache-tech-coach what you demonstrate feels disciplined enough to be equivalent to TDD but not exactly TDD by the letter, to me at least. I don’t think TCR always achieves equivalence with TDD, but this approach with approval testing seems to. I need to try it myself to fully form my opinions. With tcr I often dislike the lack of seeing the tests fail for the right reason before I start implementing, but when I use it for green to green refactors it’s amazingly valuable.

@JohnWilson-xl3rl 9 ай бұрын

@@craigjbass with TCR you can "see red" and check that you are asserting new behaviour (rather than copy pasta and existing test) by changing the test to be failing once you have successfully committed. This will revert the code but there is nothing to revert, ie no loss of progress but you can be sure the new test was the behaviour you just coded.

@JohnWilson-xl3rl 9 ай бұрын

Furthermore this is exactly what we are doing with Approval Testing, but as we are much more on the "outside" than TDD or TCR what we essentially have is this very elegant way of making incremental automated "acceptance" tests. You see this in the video as each approved approval becomes more like Emily's sketch (manual acceptance test). It really is a superb practice for the outside loop in double loop TDD. Thank you @EmilyBache-tech-coach for this video and the channel.

@ewinslow822 9 ай бұрын

Another word I hear for this is "snapshot testing" because instead of making targeted assertions on the expected outcome, you are just printing a snapshot of the state of the system at some point and relying on humans to be able to judge whether the snapshot "looks right." IMO an improvement would be to instead break this into two non-snapshot tests: scenario: when i insert a penny - the penny ends up in the return drawer - nothing else about the system changes

@EmilyBache-tech-coach 9 ай бұрын

Yes, some people use this language calling it a 'snapshot', however I don't really like to use that word. It implies a kind of casual impermanence. I'd rather talk about output that I explicitly chose to print and approve. Of course you could re-write this as two tests, as you suggest. I think the approval test version is easier to understand and less code overall though.

@tongobong1 6 ай бұрын

Dave Farley is a guru that doesn't quite understand software testing and to be honest many gurus don't understand it.

@EmilyBache-tech-coach 6 ай бұрын

I wouldn't go so far as to say Dave Farley doesn't understand software testing. I think he has judged Approval Testing without really understanding it though.