Writing and running full e2e integration tests using Go and for Go CLI applications. Lookingo into one of:
google/go-cmdtest: This package simplifies testing of command-line interfaces.
@lyse@lyse.isobeef.org or @movq@www.uninformativ.de Can I get your thoughts and opinions on any of these three options for testing CLI apps written in Go with the possibility of also measuring coverage – Which means using the test binary under test which I think all three solutions can do? (definitely the first one says it can).
Quickly tried out testcli and it’s a “no go” for me:
- Its README is out-of-date and has an old reference to a package that had its import path changed (easily fixed)
- Running the tests failed miserably as it could not find the
greetingbinary in the$PATH🤦♂️
da fuq?! I guess this doesn’t do what I thought – which is to build the test binary and use that to run CLI tests against so you can actually measure coverage 😔
Next!
prologic/greeting: A “Hello World” greeting command-line (CLI) application - greeting - Mills
This is eventually what I came up with so far… What do you think @lyse@lyse.isobeef.org ? 🤔
@lyse@lyse.isobeef.org Yeah I actually use this technique a lot in GoNix for basically all the Applets. I think this makes it easier to test. The cmdtest package is kind of cool though really, it basically implements the same kind of test runner as you may (or may not) have seen in the Mercurial test suite. The test files in testdata are essentially text files that look a bit like you’ve run something on the console and copied pasted them. This is brilliant for e2e cli integration testing 👌 And yes it manages to run the test binary so that coverage can also be measured which is fantastic 👌 – Of course this does not preclude you from writing unit tests for any other parts of your package/library that have a public facing API – But if your public facing API is just the CLI then this is a perfect fit 👌
So… This PR I cut this morning adds the basic structure for e2e cli integration tests as seen in my fancy “Hello World” greeting 😅 – What I like here is how straight forward it was to do 👌
@lyse@lyse.isobeef.org Oh btw… The reason a io.Writer is left out for “stderr” is that normally I tend to just set the logging output to os.Stderr, like log.SetOutput(os.Stderr) anyway and its not usually something I end up testing. Not sure if this is the best approach, but I’m only really interested in testing the “output”(s) and either error or non-error cases.
@prologic@twtxt.net have you ever tried expect
@tkanos@twtxt.net Yes a long time ago
@lyse@lyse.isobeef.org Yeah I agree errors should be tested too. That’s why Main() has the func Main(w io.Writer, args []string) error 😅 So you can actually assert on the error returned. For a CLI however, I’m not particularly a fan of logging errors to stderr too much (if at all). And re go-cmdtest FWIW a Terminal combines stdout and stderr too by default when displaying the output of a program 😅 – However I filed an issue against the cmdtest project and now I’m not so sure I want to continue using it, I may as well just figure out how to run the test binary under coverage and write the tests myself in Go.
@lyse@lyse.isobeef.org Hmmm good points but yeah it depends on the program 👌