aboutsummaryrefslogtreecommitdiffstats
path: root/doc
diff options
context:
space:
mode:
authorEli Zaretskii2017-10-28 14:25:07 +0300
committerEli Zaretskii2017-10-28 14:25:07 +0300
commit9102fb603ed74bd48bc8e16a09d8224dab7600b7 (patch)
treeab4c34886709236db36dfed34fdbee59a44fcd1c /doc
parent68182a471007e4cb72efe53c38af9cb3aae332d2 (diff)
downloademacs-9102fb603ed74bd48bc8e16a09d8224dab7600b7.tar.gz
emacs-9102fb603ed74bd48bc8e16a09d8224dab7600b7.zip
Add Index to ERT manual
* doc/misc/ert.texi: Add @syncodeindex directives. (Index): New node. (Top): Add Index to the top-level menus. Add index entries to all nodes.
Diffstat (limited to 'doc')
-rw-r--r--doc/misc/ert.texi113
1 files changed, 96 insertions, 17 deletions
diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index 4a2c29dcb9f..060807cede0 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -3,6 +3,10 @@
3@setfilename ../../info/ert.info 3@setfilename ../../info/ert.info
4@settitle Emacs Lisp Regression Testing 4@settitle Emacs Lisp Regression Testing
5@include docstyle.texi 5@include docstyle.texi
6@syncodeindex fn cp
7@syncodeindex vr cp
8@syncodeindex pg cp
9@syncodeindex ky cp
6@c %**end of header 10@c %**end of header
7 11
8@dircategory Emacs misc features 12@dircategory Emacs misc features
@@ -59,6 +63,7 @@ traditional software development methods.
59* How to Debug Tests:: What to do if a test fails. 63* How to Debug Tests:: What to do if a test fails.
60* Extending ERT:: ERT is extensible in several ways. 64* Extending ERT:: ERT is extensible in several ways.
61* Other Testing Concepts:: Features not in ERT. 65* Other Testing Concepts:: Features not in ERT.
66* Index:: Concept, Function and Variable Index
62* GNU Free Documentation License:: The license for this documentation. 67* GNU Free Documentation License:: The license for this documentation.
63 68
64@detailmenu 69@detailmenu
@@ -92,6 +97,10 @@ Other Testing Concepts
92* Mocks and Stubs:: Stubbing out code that is irrelevant to the test. 97* Mocks and Stubs:: Stubbing out code that is irrelevant to the test.
93* Fixtures and Test Suites:: How ERT differs from tools for other languages. 98* Fixtures and Test Suites:: How ERT differs from tools for other languages.
94 99
100Index
101
102* Index:: Concept, Function and Variable Index
103
95Appendix 104Appendix
96 105
97* GNU Free Documentation License:: The license for this documentation. 106* GNU Free Documentation License:: The license for this documentation.
@@ -102,6 +111,7 @@ Appendix
102 111
103@node Introduction 112@node Introduction
104@chapter Introduction 113@chapter Introduction
114@cindex introduction to ERT
105 115
106ERT allows you to define @emph{tests} in addition to functions, 116ERT allows you to define @emph{tests} in addition to functions,
107macros, variables, and the other usual Lisp constructs. Tests are 117macros, variables, and the other usual Lisp constructs. Tests are
@@ -169,6 +179,7 @@ Environment}.
169 179
170@node How to Run Tests 180@node How to Run Tests
171@chapter How to Run Tests 181@chapter How to Run Tests
182@cindex how to run ert tests
172 183
173You can run tests either in the Emacs you are working in, or on the 184You can run tests either in the Emacs you are working in, or on the
174command line in a separate Emacs process in batch mode (i.e., with no 185command line in a separate Emacs process in batch mode (i.e., with no
@@ -187,7 +198,10 @@ different Emacs versions.
187 198
188@node Running Tests Interactively 199@node Running Tests Interactively
189@section Running Tests Interactively 200@section Running Tests Interactively
201@cindex running tests interactively
202@cindex interactive testing
190 203
204@findex ert
191You can run the tests that are currently defined in your Emacs with 205You can run the tests that are currently defined in your Emacs with
192the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an 206the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an
193explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop 207explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop
@@ -232,6 +246,7 @@ F list-test
232 (different-atoms c d)))) 246 (different-atoms c d))))
233@end example 247@end example
234 248
249@cindex test results buffer
235At the top, there is a summary of the results: we ran all tests defined 250At the top, there is a summary of the results: we ran all tests defined
236in the current Emacs (@code{Selector: t}), 31 of them passed, and 2 251in the current Emacs (@code{Selector: t}), 31 of them passed, and 2
237failed unexpectedly. @xref{Expected Failures}, for an explanation of 252failed unexpectedly. @xref{Expected Failures}, for an explanation of
@@ -245,20 +260,29 @@ unexpected result. In the example above, there are two failures, both
245due to failed @code{should} forms. @xref{Understanding Explanations}, 260due to failed @code{should} forms. @xref{Understanding Explanations},
246for more details. 261for more details.
247 262
263@kindex TAB@r{, in ert results buffer}
264@kindex S-TAB@r{, in ert results buffer}
248In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between 265In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between
249buttons. Each name of a function or macro in this buffer is a button; 266buttons. Each name of a function or macro in this buffer is a button;
250moving point to it and typing @kbd{RET} jumps to its definition. 267moving point to it and typing @kbd{RET} jumps to its definition.
251 268
269@kindex r@r{, in ert results buffer}
270@kindex d@r{, in ert results buffer}
271@kindex .@r{, in ert results buffer}
272@kindex b@r{, in ert results buffer}
273@cindex backtrace of a failed test
252Pressing @kbd{r} re-runs the test near point on its own. Pressing 274Pressing @kbd{r} re-runs the test near point on its own. Pressing
253@kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the 275@kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the
254definition of the test near point (@kbd{RET} has the same effect if 276definition of the test near point (@kbd{RET} has the same effect if
255point is on the name of the test). On a failed test, @kbd{b} shows 277point is on the name of the test). On a failed test, @kbd{b} shows
256the backtrace of the failure. 278the backtrace of the failure.
257 279
280@kindex l@r{, in ert results buffer}
258@kbd{l} shows the list of @code{should} forms executed in the test. 281@kbd{l} shows the list of @code{should} forms executed in the test.
259If any messages were generated (with the Lisp function @code{message}) 282If any messages were generated (with the Lisp function @code{message})
260in a test or any of the code that it invoked, @kbd{m} will show them. 283in a test or any of the code that it invoked, @kbd{m} will show them.
261 284
285@kindex L@r{, in ert results buffer}
262By default, long expressions in the failure details are abbreviated 286By default, long expressions in the failure details are abbreviated
263using @code{print-length} and @code{print-level}. Pressing @kbd{L} 287using @code{print-length} and @code{print-level}. Pressing @kbd{L}
264while point is on a test failure will increase the limits to show more 288while point is on a test failure will increase the limits to show more
@@ -267,7 +291,11 @@ of the expression.
267 291
268@node Running Tests in Batch Mode 292@node Running Tests in Batch Mode
269@section Running Tests in Batch Mode 293@section Running Tests in Batch Mode
294@cindex running tests in batch mode
295@cindex batch-mode testing
270 296
297@findex ert-run-tests-batch
298@findex ert-run-tests-batch-and-exit
271ERT supports automated invocations from the command line or from 299ERT supports automated invocations from the command line or from
272scripts or makefiles. There are two functions for this purpose, 300scripts or makefiles. There are two functions for this purpose,
273@code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}. 301@code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}.
@@ -283,6 +311,7 @@ with a zero exit status if all tests passed, or nonzero if any tests
283failed or if anything else went wrong. It will also print progress 311failed or if anything else went wrong. It will also print progress
284messages and error diagnostics to standard output. 312messages and error diagnostics to standard output.
285 313
314@findex ert-summarize-tests-batch-and-exit
286You can also redirect the above output to a log file, say 315You can also redirect the above output to a log file, say
287@file{output.log}, and use the 316@file{output.log}, and use the
288@code{ert-summarize-tests-batch-and-exit} function to produce a neat 317@code{ert-summarize-tests-batch-and-exit} function to produce a neat
@@ -300,6 +329,8 @@ files that it requires are on your @code{load-path}.
300 329
301@node Test Selectors 330@node Test Selectors
302@section Test Selectors 331@section Test Selectors
332@cindex test selector
333@cindex selecting tests
303 334
304Functions like @code{ert} accept a @emph{test selector}, a Lisp 335Functions like @code{ert} accept a @emph{test selector}, a Lisp
305expression specifying a set of tests. Test selector syntax is similar 336expression specifying a set of tests. Test selector syntax is similar
@@ -314,17 +345,22 @@ to Common Lisp's type specifier syntax:
314@item A string is a regular expression that selects all tests with matching names. 345@item A string is a regular expression that selects all tests with matching names.
315@item A test (i.e., an object of @code{ert-test} data type) selects that test. 346@item A test (i.e., an object of @code{ert-test} data type) selects that test.
316@item A symbol selects the test that the symbol names. 347@item A symbol selects the test that the symbol names.
317@item @code{(member TESTS...)} selects the elements of TESTS, a list of 348@item @code{(member @var{tests}...)} selects the elements of
318tests or symbols naming tests. 349@var{tests}, a list of tests or symbols naming tests.
319@item @code{(eql TEST)} selects TEST, a test or a symbol naming a test. 350@item @code{(eql @var{test})} selects @var{test}, a test or a symbol
320@item @code{(and SELECTORS...)} selects the tests that match all SELECTORS. 351naming a test.
321@item @code{(or SELECTORS...)} selects the tests that match any SELECTOR. 352@item @code{(and @var{selectors}@dots{})} selects the tests that match
322@item @code{(not SELECTOR)} selects all tests that do not match SELECTOR. 353all @var{selectors}.
323@item @code{(tag TAG)} selects all tests that have TAG on their tags list. 354@item @code{(or @var{selectors}@dots{})} selects the tests that match
355any of the @var{selectors}.
356@item @code{(not @var{selector})} selects all tests that do not match
357@var{selector}.
358@item @code{(tag @var{tag})} selects all tests that have @var{tag} on
359their tags list.
324(Tags are optional labels you can apply to tests when you define them.) 360(Tags are optional labels you can apply to tests when you define them.)
325@item @code{(satisfies PREDICATE)} selects all tests that satisfy PREDICATE, 361@item @code{(satisfies @var{predicate})} selects all tests that
326a function that takes a test as argument and returns non-@code{nil} if 362satisfy @var{predicate}, a function that takes a test as argument and
327it is selected. 363returns non-@code{nil} if it is selected.
328@end itemize 364@end itemize
329 365
330Selectors that are frequently useful when selecting tests to run 366Selectors that are frequently useful when selecting tests to run
@@ -340,7 +376,9 @@ result in the last run, and tag-based selectors such as @code{(not
340 376
341@node How to Write Tests 377@node How to Write Tests
342@chapter How to Write Tests 378@chapter How to Write Tests
379@cindex how to write tests
343 380
381@findex ert-deftest
344ERT lets you define tests in the same way you define functions. You 382ERT lets you define tests in the same way you define functions. You
345can type @code{ert-deftest} forms in a buffer and evaluate them there 383can type @code{ert-deftest} forms in a buffer and evaluate them there
346with @code{eval-defun} or @code{compile-defun}, or you can save the 384with @code{eval-defun} or @code{compile-defun}, or you can save the
@@ -361,6 +399,7 @@ to find where a test was defined if the test was loaded from a file.
361@node The @code{should} Macro 399@node The @code{should} Macro
362@section The @code{should} Macro 400@section The @code{should} Macro
363 401
402@findex should@r{, ert macro}
364Test bodies can include arbitrary code; but to be useful, they need to 403Test bodies can include arbitrary code; but to be useful, they need to
365check whether the code being tested (or @emph{code under test}) 404check whether the code being tested (or @emph{code under test})
366does what it is supposed to do. The macro @code{should} is similar to 405does what it is supposed to do. The macro @code{should} is similar to
@@ -396,6 +435,8 @@ test failed, it helps to know that the function @code{+} returned 3
396here. ERT records the return value for any predicate called directly 435here. ERT records the return value for any predicate called directly
397within @code{should}. 436within @code{should}.
398 437
438@findex should-not@r{, ert macro}
439@findex should-error@r{, ert macro}
399In addition to @code{should}, ERT provides @code{should-not}, which 440In addition to @code{should}, ERT provides @code{should-not}, which
400checks that the predicate returns @code{nil}, and @code{should-error}, which 441checks that the predicate returns @code{nil}, and @code{should-error}, which
401checks that the form called within it signals an error. An example 442checks that the form called within it signals an error. An example
@@ -424,7 +465,10 @@ default.
424 465
425@node Expected Failures 466@node Expected Failures
426@section Expected Failures 467@section Expected Failures
468@cindex expected failures
469@cindex known bugs
427 470
471@vindex :expected-result
428Some bugs are complicated to fix, or not very important, and are left as 472Some bugs are complicated to fix, or not very important, and are left as
429@emph{known bugs}. If there is a test case that triggers the bug and 473@emph{known bugs}. If there is a test case that triggers the bug and
430fails, ERT will alert you of this failure every time you run all 474fails, ERT will alert you of this failure every time you run all
@@ -478,6 +522,9 @@ versions, specific architectures, etc.:
478@node Tests and Their Environment 522@node Tests and Their Environment
479@section Tests and Their Environment 523@section Tests and Their Environment
480 524
525@cindex skipping tests
526@cindex test preconditions
527@cindex preconditions of a test
481Sometimes, it doesn't make sense to run a test due to missing 528Sometimes, it doesn't make sense to run a test due to missing
482preconditions. A required Emacs feature might not be compiled in, the 529preconditions. A required Emacs feature might not be compiled in, the
483function to be tested could call an external binary which might not be 530function to be tested could call an external binary which might not be
@@ -491,6 +538,7 @@ available on the test machine, you name it. In this case, the macro
491 ...) 538 ...)
492@end lisp 539@end lisp
493 540
541@cindex tests and their environment
494The outcome of running a test should not depend on the current state 542The outcome of running a test should not depend on the current state
495of the environment, and each test should leave its environment in the 543of the environment, and each test should leave its environment in the
496same state it found it in. In particular, a test should not depend on 544same state it found it in. In particular, a test should not depend on
@@ -545,6 +593,8 @@ hook variables to @code{nil}. This avoids the above problems.
545 593
546@node Useful Techniques 594@node Useful Techniques
547@section Useful Techniques when Writing Tests 595@section Useful Techniques when Writing Tests
596@cindex useful techniques
597@cindex tips and tricks
548 598
549Testing simple functions that have no side effects and no dependencies 599Testing simple functions that have no side effects and no dependencies
550on their environment is easy. Such tests often look like this: 600on their environment is easy. Such tests often look like this:
@@ -582,6 +632,8 @@ Here's a more complicated test:
582 " signal(ert-test-failed (\"foo\"))"))))))) 632 " signal(ert-test-failed (\"foo\"))")))))))
583@end lisp 633@end lisp
584 634
635@findex make-ert-test
636@findex ert-equal-including-properties
585This test creates a test object using @code{make-ert-test} whose body 637This test creates a test object using @code{make-ert-test} whose body
586will immediately signal failure. It then runs that test and asserts 638will immediately signal failure. It then runs that test and asserts
587that it fails. Then, it creates a temporary buffer and invokes 639that it fails. Then, it creates a temporary buffer and invokes
@@ -639,6 +691,8 @@ a test failed.
639 691
640@node Understanding Explanations 692@node Understanding Explanations
641@section Understanding Explanations 693@section Understanding Explanations
694@cindex understanding explanations
695@cindex explanations, understanding
642 696
643Failed @code{should} forms are reported like this: 697Failed @code{should} forms are reported like this:
644 698
@@ -706,41 +760,55 @@ function registered. @xref{Defining Explanation Functions}.
706 760
707@node Interactive Debugging 761@node Interactive Debugging
708@section Interactive Debugging 762@section Interactive Debugging
763@cindex interactive debugging
764@cindex debugging failed tests
709 765
710Debugging failed tests essentially works the same way as debugging any 766Debugging failed tests essentially works the same way as debugging any
711other problems with Lisp code. Here are a few tricks specific to 767other problems with Lisp code. Here are a few tricks specific to
712tests: 768tests:
713 769
714@itemize 770@itemize
715@item Re-run the failed test a few times to see if it fails in the same way 771@cindex re-running a failed test
772@item
773Re-run the failed test a few times to see if it fails in the same way
716each time. It's good to find out whether the behavior is 774each time. It's good to find out whether the behavior is
717deterministic before spending any time looking for a cause. In the 775deterministic before spending any time looking for a cause. In the
718ERT results buffer, @kbd{r} re-runs the selected test. 776ERT results buffer, @kbd{r} re-runs the selected test.
719 777
720@item Use @kbd{.} to jump to the source code of the test to find out exactly 778@cindex jump to the test source code
779@item
780Use @kbd{.} to jump to the source code of the test to find out exactly
721what it does. Perhaps the test is broken rather than the code 781what it does. Perhaps the test is broken rather than the code
722under test. 782under test.
723 783
724@item If the test contains a series of @code{should} forms and you can't 784@item
785If the test contains a series of @code{should} forms and you can't
725tell which one failed, use @kbd{l}, which shows you the list of all 786tell which one failed, use @kbd{l}, which shows you the list of all
726@code{should} forms executed during the test before it failed. 787@code{should} forms executed during the test before it failed.
727 788
728@item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run 789@cindex show backtrace of failed test
790@item
791Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run
729the test with debugging enabled, this will enter the debugger and show 792the test with debugging enabled, this will enter the debugger and show
730the backtrace as well; but the top few frames shown there will not be 793the backtrace as well; but the top few frames shown there will not be
731relevant to you since they are ERT's own debugger hook. @kbd{b} 794relevant to you since they are ERT's own debugger hook. @kbd{b}
732strips them out, so it is more convenient. 795strips them out, so it is more convenient.
733 796
734@item If the test or the code under testing prints messages using 797@item
798If the test or the code under testing prints messages using
735@code{message}, use @kbd{m} to see what messages it printed before it 799@code{message}, use @kbd{m} to see what messages it printed before it
736failed. This can be useful to figure out how far it got. 800failed. This can be useful to figure out how far it got.
737 801
738@item You can instrument tests for debugging the same way you instrument 802@cindex instrumenting test for Edebug
803@item
804You can instrument tests for debugging the same way you instrument
739@code{defun}s for debugging: go to the source code of the test and 805@code{defun}s for debugging: go to the source code of the test and
740type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and 806type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and
741re-run the test with @kbd{r} or @kbd{d}. 807re-run the test with @kbd{r} or @kbd{d}.
742 808
743@item If you have been editing and rearranging tests, it is possible that 809@cindex discard obsolete test results
810@item
811If you have been editing and rearranging tests, it is possible that
744ERT remembers an old test that you have since renamed or removed: 812ERT remembers an old test that you have since renamed or removed:
745renamings or removals of definitions in the source code leave around a 813renamings or removals of definitions in the source code leave around a
746stray definition under the old name in the running process (this is a 814stray definition under the old name in the running process (this is a
@@ -751,6 +819,7 @@ forget about the obsolete test.
751 819
752@node Extending ERT 820@node Extending ERT
753@chapter Extending ERT 821@chapter Extending ERT
822@cindex extending ert
754 823
755There are several ways to add functionality to ERT. 824There are several ways to add functionality to ERT.
756 825
@@ -762,6 +831,7 @@ There are several ways to add functionality to ERT.
762 831
763@node Defining Explanation Functions 832@node Defining Explanation Functions
764@section Defining Explanation Functions 833@section Defining Explanation Functions
834@cindex defining explanation functions
765 835
766The explanation function for a predicate is a function that takes the 836The explanation function for a predicate is a function that takes the
767same arguments as the predicate and returns an @emph{explanation}. 837same arguments as the predicate and returns an @emph{explanation}.
@@ -772,6 +842,7 @@ comprehensible printed representation. If the return value of the
772predicate needs no explanation for a given list of arguments, the 842predicate needs no explanation for a given list of arguments, the
773explanation function should return @code{nil}. 843explanation function should return @code{nil}.
774 844
845@vindex ert-explainer@r{, property}
775To associate an explanation function with a predicate, add the 846To associate an explanation function with a predicate, add the
776property @code{ert-explainer} to the symbol that names the predicate. 847property @code{ert-explainer} to the symbol that names the predicate.
777The value of the property should be the symbol that names the 848The value of the property should be the symbol that names the
@@ -780,6 +851,7 @@ explanation function.
780 851
781@node Low-Level Functions for Working with Tests 852@node Low-Level Functions for Working with Tests
782@section Low-Level Functions for Working with Tests 853@section Low-Level Functions for Working with Tests
854@cindex low-level functions
783 855
784Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch} 856Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch}
785are implemented on top of the lower-level test handling code in the 857are implemented on top of the lower-level test handling code in the
@@ -807,6 +879,7 @@ For information on mocks, stubs, fixtures, or test suites, see below.
807 879
808@node Mocks and Stubs 880@node Mocks and Stubs
809@section Other Tools for Emacs Lisp 881@section Other Tools for Emacs Lisp
882@cindex mocks and stubs
810 883
811Stubbing out functions or using so-called @emph{mocks} can make it 884Stubbing out functions or using so-called @emph{mocks} can make it
812easier to write tests. See 885easier to write tests. See
@@ -820,6 +893,7 @@ offers mocks for Emacs Lisp and can be used in conjunction with ERT.
820 893
821@node Fixtures and Test Suites 894@node Fixtures and Test Suites
822@section Fixtures and Test Suites 895@section Fixtures and Test Suites
896@cindex fixtures
823 897
824In many ways, ERT is similar to frameworks for other languages like 898In many ways, ERT is similar to frameworks for other languages like
825SUnit or JUnit. However, two features commonly found in such 899SUnit or JUnit. However, two features commonly found in such
@@ -877,6 +951,11 @@ e.g., to run quick tests during interactive development and slow tests less
877often. This can be achieved with the @code{:tag} argument to 951often. This can be achieved with the @code{:tag} argument to
878@code{ert-deftest} and @code{tag} test selectors. 952@code{ert-deftest} and @code{tag} test selectors.
879 953
954@node Index
955@unnumbered Index
956
957@printindex cp
958
880@node GNU Free Documentation License 959@node GNU Free Documentation License
881@appendix GNU Free Documentation License 960@appendix GNU Free Documentation License
882@include doclicense.texi 961@include doclicense.texi