Test case
From C
Contents |
How to write a good test case
What is a test case
A test case is a single small source code listing demonstrating your problem. Give us a good test case, and we'll see if we can help you solve/understand your problem.
What is a good test case?
A good test case meets the requirements from the checklist below. The quality of your test case matters: a good test case will get you helped quickly and effectively, a bad test case will get you ignored or, at best, lectured. The points below are not absolutes, but the more rules you break, the more you discourage people from helping you.
Checklist
- Use a suitable pasting mechanism.
Pasting more than three line of code in the channel is a mortal sin. Putting a raw source file on a web server can make it hard to talk about line numbers. The preferred mechanism is to use a paste site. Recommended paste sites for ##c are ideone.com and codepad.org. If for some reason you prefer to use another paste site, please make sure that it at least meets the following requirements:
- it should show line numbers;
- it should load fast;
- its default font should be readable (for example, the difference between '1' and 'l' and 'I' should be clearly visible);
- its color scheme should not induce a migraine;
- when we copy&paste the code displayed into our editor we should not get line numbers or other artifacts.
pastebin.* and nomorepasting.* sites are known not to meet these criteria. Don't use them.
- Make sure your pasted code is indented properly and has no other pasting artifacts.
When using paste sites or similar mechanisms, sometimes you end up with a paste in which indentation is lost or empty lines are stripped (or even added!). These things make the paste hard to read. Check to see if this is the case for your paste, and fix if necessary.
- Describe the problem - code alone is not enough!
A simple source listing accompanied by the phrase "it does not work" doesn't tell us what your problem actually is.
If your problem is a compiler error (or warning), copy-paste it into the source as a comment either at the end of the code or, if the snippet is very small, at the location that the error/warning refers to.
If the problem is a runtime error, ideally it should either be in the form of a program crash or a "logic error" (which means that the program produced output different from what you expected). In case of a program crash (segmentation fault, assertion failure, unexpected exception, etc), use a debugger to find out on which line the crash occurs and note it as above. In case of a logic error, explicitly mention (in a comment) what output you expected and what output the program actually produced.
- Make sure any line numbers you mention in the test case are correct
This should be an obvious requirement; incorrect line numbers are misleading.
- Use very high warning levels and debuggers.
People in ##c help others because they enjoy sharing their C skills, knowledge, experience, and resourcefulness. There is no skill involved or experience required for diagnosing problems simply by compiling the code with higher warning levels or running it through a debugger. This is grunt work that you should do yourself.
Of course, if you encounter a compiler warning or debugger message that you do not know how to interpret or fix, then by all means dedicate your test case to this and we'll be happy to help (unless it's a very debugger-software-specific thing, in which case it's more appropriate to find help channels specific to that debugger software).
- Make sure your test case is self-contained and actually reproduces the problem
A test case is said to reproduce the problem if, when we independently try to compile and run the test case locally, we get the same compiler error (or our compiler's version of it) or erroneous program output specified in the test case (see point 3). A test case that does not reproduce the problem is useless: if we get different errors or output, then the code apparently does not correspond to the actual code you're having problems with, and consequently analyzing it is pointless. The same is true if the test case works fine.
A test case consisting of randomly copy&paste'd bits of code that you thought were relevant can obviously not reproduce the problem. Worse, such test cases force your perception of the problem upon us, preventing us from taking a fresh look at it and making an unbiased analysis.
For us to be able to compile and run the test case locally, it must be self-contained, meaning that it must not have any external dependencies: it must not refer to functions/variables defined elsewhere, nor must it #include headers other than those of the C standard library. If you are making a test case to demonstrate a problem you encountered while developing a nontrivial program, this implies that you will have to isolate it (see the next point).
- Isolate the problem as much as possible
The larger the test case, the harder it is to analyze. This is especially true when the analyst is not familiar with the program that the test case was extracted from. Therefore, it is very important that the test case be as small as possible: every function/variable/class/template/etc that does not directly contribute to the problem should be cut. Isolation is generally by far the most time-consuming process in making a test case.
Fortunately, isolation is a skill that can be learned like any other, and that you will become proficient in over time. Furthermore, it is a skill that you will find tremendously useful; systematic isolation virtually guarantees that a problem can ultimately be analyzed. As you gradually become better at isolating problems, you'll find that increasingly often you will be able to isolate the problem to only a few lines in which the problem has become entirely obvious.
Experience shows that the vast majority of problems people have can be shown in less than 15 lines of code. As a rule of thumb, many of us will flat out ignore test cases larger than, say, 50 lines of code.
- Remember the scope of the channel
##c is a topical channel focusing on standard C — the language and its standard library. We are not interested in problems relating to the APIs of specific operating systems or libraries.
- Use English comments and identifiers
For comments, this should be an obvious requirement.
For identifiers, the choice of whether to use single-letter identifiers or more descriptive identifiers is entirely up to you. For many test cases, the former are actually quite okay. If however you decide to use descriptive identifiers, then we will assume that they were necessary to express the intent of the code, and will be put off if we cannot understand them because of the language in which they are written.
Credit
Thanks to http://eel.is/iso-c++/testcase/ for the original content this page was based on. Used with permission.
