Automated Testing

So obviously testing is an integral part of program development. Code doesn't work on the first time. Us coders screw up. Because despite all of the behind-the-scenes shortcuts and my obviously-beautiful writing style and everything else that's done to keep it all easy to understand and intuitive, sometimes it's just not intuitive enough. But I digress.

The problem is: how do I make sure that my program does what I think it should do? The answer, equally simple: try it. The obvious, naive way to do this is to take on the role of the user, and literally try it. And this is where automated testing comes in -- there will come some point where program complexity gets high enough that writing more code to test your main code is simpler than the alternative of doing it manually (for, you know, the same reason that some tasks get so finicky or repetitive that the time and effort spent writing the program is easier to put in than would be required to do it manually). I can imagine there being a trade-off, though; here's a list of things to possibly  consider:

• The obvious one: if the testing process is very short, you probably don't need automated testing. When the time required to write a thorough test is trivial compared to the time required to automate testing, I cannot see myself doing that automation without a good reason.

• If the testing process will have to be repeated many times, that's a point for automation. Imagine a project where functionality will be added incrementally: the older functions would have to be tested when they are first created, and then again whenever a change was made to ensure nothing had been broken. In this case, I can see automated testing being helpful over time.

• If a test requires a lot of iteration, or requires a lot of time to run, then I would probably choose to automate testing. But that's the main criteria I'd consider when deciding whether or not to automate anything, hm?

• If automated testing requires the usage of the same code as your actual program, then that testing is probably not doing much. Imagine the ersatz example program:
  
  x = [1, 2, 3]
  
  And the associated test snippet:
  
  def test_correct_value(self):    self.assertEqual(x, [1, 2, 3])
  
  Why are you even doing this? The clarity is already in your code; you can read it and feel confident you haven't misinterpreted it. In fact, you actually demonstrated that particular skill while writing the test case. Back to the premise, automated testing is useful when intuitive understanding is difficult. Here, it's not accomplishing anything.

• Finally, I can imagine some scenarios where your program is designed to solve some problem that you don't already know the answer to, to optimize something, for example. You would have to test the component functions, of course; for those I could decide how to test based on other criteria (though at this point I think the complexity has probably reached that threshold). But how to test your whole program, when you don't actually know what the output should be? Again, you can test your program for runtime errors and incorrectly-formatted output, but did it give you the right answer?

• At some point, you will have to just run your program and see what it gives you. It is possible to catch bugs -- I've done it -- that show answers that could technically be correct, but are suspicious-looking enough to catch your attention (for example, imagine this optimization program as a chess engine: what if it turns out that the program will only ever move knights?) Sometimes, stupid-answers will be your only chance to catch a bug in a complex program, and it's hard to imagine automated testing clever enough to catch all of them. Other times, at some level, you'll just have to trust that you've done your best, that you've evaluated every case you can, and hopefully you won't have missed anything.