Amend parse-partial-sexp correctly to handle two character comment delimiters

Do this by adding a new field to the parser state: the syntax of the last
character scanned, should that be the first char of a (potential) two char
construct, nil otherwise.
This should make the parser state complete.
Also document element 9 of the parser state.  Also refactor the code a bit.

* src/syntax.c (struct lisp_parse_state): Add a new field.
(SYNTAX_FLAGS_COMSTARTEND_FIRST): New function.
(internalize_parse_state): New function, extracted from scan_sexps_forward.
(back_comment): Call internalize_parse_state.
(forw_comment): Return the syntax of the last character scanned to the caller
when that character might be the first of a two character construct.
(Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
(scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
information via the address parameter `state'.  Remove the code which converts
from external to internal form of `state'.  Access buffer contents only from
`from' onwards.  Reformulate code at the top of the main loop correctly to
recognize comment openers when starting in the middle of one.  Call
forw_comment with extra argument (for return of syntax value of possible first
char of a two char construct).
(Fparse_partial_sexp): Document elements 9, 10 of the parser state in the
doc string.  Clarify the doc string in general.  Call
internalize_parse_state.  Take account of the new elements when consing up the
output parser state.

* doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
element 10.  Minor wording corrections (remove reference to "trivial
cases").
(Low Level Parsing): Minor corrections.

* etc/NEWS: Note new element 10, and documentation of element 9 of parser
state.

1 files changed, 12 insertions, 0 deletions

diff --git a/etc/NEWS b/etc/NEWS
index d963dee2c63..ea321539426 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -175,6 +175,18 @@ a new window when opening man pages when there's already one, use
         (inhibit-same-window . nil)
         (mode . Man-mode))))
 
++++
+** `parse-partial-sexp' state has a new element.  Element 10 is
+non-nil when the last character scanned might be the first character
+of a two character construct, i.e. a comment delimiter or escaped
+character.  Its value is the syntax of that last character.
+
++++
+** `parse-partial-sexp''s state, element 9, has now been confirmed as
+permanent and documented, and may be used by Lisp programs.  Its value
+is a list of currently open parenthesis positions, starting with the
+outermost parenthesis.
+
 
 * Changes in Emacs 25.2 on Non-Free Operating Systems