Add backtrace-mode and use it in the debugger, ERT and Edebug

* doc/lispref/debugging.texi (Using Debugger): Remove explanation of
backtrace buffer.  Refer to new node.
(Backtraces): New node.
(Debugger Commands): Refer to new node.  Remove 'v'.
* doc/lispref/edebug.texi (Edebug Misc): Refer to new node.
* doc/misc/ert.texi (Running Tests Interactively): Refer to new node.

* lisp/emacs-lisp-backtrace.el: New file.
* test/lisp/emacs-lisp/backtrace-tests.el: New file.

* lisp/emacs-lisp/debug.el: (debugger-buffer-state): New cl-defstruct.
(debugger--restore-buffer-state): New function.
(debug): Use a debugger-buffer-state object to save and restore buffer
state.  Fix bug#15749 by leaving an unused buffer in debugger-mode,
empty, instead of in fundamental-mode, and then when reusing a buffer,
not calling debugger-mode if the buffer is already in debugger-mode.
(debugger-insert-backtrace): Remove.
(debugger-setup-buffer): Use backtrace-mode.
(debugger--insert-header): New function.
(debugger-continue, debugger-return-value): Change check for flags to
use backtrace-frames.
(debugger-frame-number): Determine backtrace frame number from
backtrace-frames.
(debugger--locals-visible-p, debugger--insert-locals)
(debugger--show-locals, debugger--hide-locals)
(debugger-toggle-locals): Remove.
(debugger-mode-map): Make a child of backtrace-mode-map.  Move
navigation commands to backtrace-mode-map.  Bind 'q' to debugger-quit
instead of top-level.  Make Help Follow menu item call
backtrace-help-follow-symbol.
(debugger-mode): Derive from backtrace-mode.
(debug-help-follow): Remove.  Move body of this function to
'backtrace-help-follow-symbol' in backtrace.el.
(debugger-quit): New function.

* lisp/emacs-lisp/edebug.el (edebug-unwrap-results): Remove warning
in docstring about circular results.
(edebug-unwrap): Use pcase.
(edebug-unwrap1): New function to unwrap circular objects.
(edebug-unwrap*): Use it.
(edebug--frame): New cl-defstruct.
(edebug-backtrace): Call the buffer *Edebug Backtrace* and use
backtrace-mode.  Get the frames from edebug--backtrace-frames.
(edebug--backtrace-frames, edebug--unwrap-and-add-info)
(edebug--symbol-not-prefixed-p): New functions.

* lisp/emacs-lisp/lisp-mode.el
(lisp-el-font-lock-keywords-for-backtraces)
(lisp-el-font-lock-keywords-for-backtraces-1)
(lisp-el-font-lock-keywords-for-backtraces-2): New constants.

* lisp/emacs-lisp/ert.el (ert--print-backtrace): Remove.
(ert--run-test-debugger): Use backtrace-get-frames.
(ert-run-tests-batch): Use backtrace-to-string.
(ert-results-pop-to-backtrace-for-test-at-point): Use backtrace-mode.
(ert--insert-backtrace-header): New function.

* tests/lisp/emacs-lisp/ert-tests.el (ert-test--which-file):
Use backtrace-frame slot accessor.

3 files changed, 79 insertions, 26 deletions

diff --git a/doc/lispref/debugging.texi b/doc/lispref/debugging.texi
index 1b1f87465db..b5a73a255af 100644
--- a/doc/lispref/debugging.texi
+++ b/doc/lispref/debugging.texi
@@ -81,7 +81,8 @@ debugger recursively.  @xref{Recursive Editing}.
 * Function Debugging::    Entering it when a certain function is called.
 * Variable Debugging::    Entering it when a variable is modified.
 * Explicit Debug::        Entering it at a certain point in the program.
-* Using Debugger::        What the debugger does; what you see while in it.
+* Using Debugger::        What the debugger does.
+* Backtraces::            What you see while in the debugger.
 * Debugger Commands::     Commands used while in the debugger.
 * Invoking the Debugger:: How to call the function @code{debug}.
 * Internals of Debugger:: Subroutines of the debugger, and global variables.
@@ -392,32 +393,79 @@ this is not what you want, you can either set
 @code{eval-expression-debug-on-error} to @code{nil}, or set
 @code{debug-on-error} to @code{nil} in @code{debugger-mode-hook}.
 
+  The debugger itself must be run byte-compiled, since it makes
+assumptions about the state of the Lisp interpreter.  These
+assumptions are false if the debugger is running interpreted.
+
+@node Backtraces
+@subsection Backtraces
+@cindex backtrace buffer
+
+Debugger mode is derived from Backtrace mode, which is also used to
+show backtraces by Edebug and ERT.  (@pxref{Edebug} and @ref{Top,the
+ERT manual,, ert, ERT: Emacs Lisp Regression Testing})
+
+@cindex stack frame
+The backtrace buffer shows you the functions that are executing and
+their argument values.  When a backtrace buffer is created, it shows
+each stack frame on one, possibly very long, line.  (A stack frame is
+the place where the Lisp interpreter records information about a
+particular invocation of a function.)  The most recently called
+function will be at the top.
+
 @cindex current stack frame
-  The backtrace buffer shows you the functions that are executing and
-their argument values.  It also allows you to specify a stack frame by
-moving point to the line describing that frame.  (A stack frame is the
-place where the Lisp interpreter records information about a particular
-invocation of a function.)  The frame whose line point is on is
-considered the @dfn{current frame}.  Some of the debugger commands
-operate on the current frame.  If a line starts with a star, that means
-that exiting that frame will call the debugger again.  This is useful
-for examining the return value of a function.
-
-  If a function name is underlined, that means the debugger knows
-where its source code is located.  You can click with the mouse on
-that name, or move to it and type @key{RET}, to visit the source code.
+In a backtrace you can specify a stack frame by moving point to a line
+describing that frame.  The frame whose line point is on is considered
+the @dfn{current frame}.
+
+If a function name is underlined, that means Emacs knows where its
+source code is located.  You can click with the mouse on that name, or
+move to it and type @key{RET}, to visit the source code.  You can also
+type @key{RET} while point is on any name of a function or variable
+which is not underlined, to see help information for that symbol in a
+help buffer, if any exists. The @code{xref-find-definitions} command,
+bound to @key{M-.}, can also be used on any identifier in a backtrace
+(@pxref{Looking Up Identifiers,,,emacs,Emacs manual}).
+
+In backtraces, the tails of long lists and the ends of long strings,
+vectors or structures, as well as objects which are deeply nested,
+will be printed as underlined ``...''.  You can click with the mouse
+on a ``...'', or type @key{RET} while point is on it, to show the part
+of the object that was hidden.  To control how much abbreviation is
+done, customize @code{backtrace-line-length}.
+
+Here is a list of commands for navigating and viewing backtraces:
 
-  The debugger itself must be run byte-compiled, since it makes
-assumptions about how many stack frames are used for the debugger
-itself.  These assumptions are false if the debugger is running
-interpreted.
+@table @kbd
+@item v
+Toggle the display of local variables of the current stack frame.
+
+@item p
+Move to the beginning of the frame, or to the beginning
+of the previous frame.
+
+@item n
+Move to the beginning of the next frame.
+
+@item +
+Add line breaks and indentation to the top-level Lisp form at point to
+make it more readable.
+
+@item =
+Collapse the top-level Lisp form at point back to a single line.
+
+@item #
+Toggle @code{print-circle} for the frame at point.
+
+@end table
 
 @node Debugger Commands
 @subsection Debugger Commands
 @cindex debugger command list
 
   The debugger buffer (in Debugger mode) provides special commands in
-addition to the usual Emacs commands.  The most important use of
+addition to the usual Emacs commands and to the Backtrace mode commands
+described in the previous section.  The most important use of
 debugger commands is for stepping through code, so that you can see
 how control flows.  The debugger can step through the control
 structures of an interpreted function, but cannot do so in a
@@ -427,6 +475,11 @@ the same function.  (To do this, visit the source for the function and
 type @kbd{C-M-x} on its definition.)  You cannot use the Lisp debugger
 to step through a primitive function.
 
+Some of the debugger commands operate on the current frame.  If a
+frame starts with a star, that means that exiting that frame will call the
+debugger again.  This is useful for examining the return value of a
+function.
+
 @c FIXME: Add @findex for the following commands?  --xfq
   Here is a list of Debugger mode commands:
 
@@ -502,8 +555,6 @@ Display a list of functions that will invoke the debugger when called.
 This is a list of functions that are set to break on entry by means of
 @code{debug-on-entry}.
 
-@item v
-Toggle the display of local variables of the current stack frame.
 @end table
 
 @node Invoking the Debugger
diff --git a/doc/lispref/edebug.texi b/doc/lispref/edebug.texi
index b9cc1d5afc2..0e0a2e8a643 100644
--- a/doc/lispref/edebug.texi
+++ b/doc/lispref/edebug.texi
@@ -442,8 +442,8 @@ Redisplay the most recently known expression result in the echo area
 Display a backtrace, excluding Edebug's own functions for clarity
 (@code{edebug-backtrace}).
 
-You cannot use debugger commands in the backtrace buffer in Edebug as
-you would in the standard debugger.
+@xref{Debugging,, Backtraces, elisp}, for the commands which work
+in a backtrace buffer.
 
 The backtrace buffer is killed automatically when you continue
 execution.
diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index 82e0e27ed1c..e2aeeb1353a 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -273,9 +273,11 @@ moving point to it and typing @kbd{@key{RET}} jumps to its definition.
 @cindex backtrace of a failed test
 Pressing @kbd{r} re-runs the test near point on its own.  Pressing
 @kbd{d} re-runs it with the debugger enabled.  @kbd{.} jumps to the
-definition of the test near point (@kbd{@key{RET}} has the same effect if
-point is on the name of the test).  On a failed test, @kbd{b} shows
-the backtrace of the failure.
+definition of the test near point (@kbd{@key{RET}} has the same effect
+if point is on the name of the test).  On a failed test, @kbd{b} shows
+the backtrace of the failure.  @xref{Debugging,, Backtraces, elisp,
+the Emacs Lisp Reference Manual}, for more information about
+backtraces.
 
 @kindex l@r{, in ert results buffer}
 @kbd{l} shows the list of @code{should} forms executed in the test.