Fix C++ fontification problems 500 bytes after typing a space, and other bugs

Also implement the "asymmetric space" rule for fontifying otherwise
ambiguous
declarations/expressions.

* lisp/progmodes/cc-engine.el (c-before-change-check-<>-operators): Don't set
c-new-BEG or c-new-END when there is no need.
(c-forward-decl-or-cast-1): Add "CASE 17.5" to implement the "asymmetric
space" rule.

* lisp/progmodes/cc-fonts.el (c-get-fontification-context): New function,
extracted from c-font-lock-declarations.  Add to this function processing to
make `context' 'decl for lines contained within parens when these are also
declarations.
(c-font-lock-declarations): Call the newly extracted function above in place
of inline code.

* lisp/progmodes/cc-mode.el (c-fl-decl-start): Set point before calling
c-literal-start.

* lisp/progmodes/cc-vars.el (c-asymmetry-fontification-flag): New user option.

* doc/misc/cc-mode.texi (Misc Font Locking): New node documenting the new
"asymmetric fontification" rule, including the variable
c-asymmetric-fontification-flag.

5 files changed, 225 insertions, 96 deletions

diff --git a/doc/misc/cc-mode.texi b/doc/misc/cc-mode.texi
index a29873b03b3..91e20fa7247 100644
--- a/doc/misc/cc-mode.texi
+++ b/doc/misc/cc-mode.texi
@@ -274,6 +274,7 @@ Font Locking
 * Font Locking Preliminaries::
 * Faces::
 * Doc Comments::
+* Misc Font Locking::
 * AWK Mode Font Locking::
 
 Configuration Basics
@@ -1821,6 +1822,7 @@ sections apply to the other languages.
 * Font Locking Preliminaries::
 * Faces::
 * Doc Comments::
+* Misc Font Locking::
 * AWK Mode Font Locking::
 @end menu
 
@@ -2023,7 +2025,7 @@ since those aren't syntactic errors in themselves.
 
 
 @comment !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-@node    Doc Comments, AWK Mode Font Locking, Faces, Font Locking
+@node    Doc Comments, Misc Font Locking, Faces, Font Locking
 @comment node-name, next, previous, up
 @section Documentation Comments
 @cindex documentation comments
@@ -2099,9 +2101,63 @@ initialization and the result is prepended.  For an example, see
 If you add support for another doc comment style, please consider
 contributing it: send a note to @email{bug-cc-mode@@gnu.org}.
 
+@comment !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+@node    Misc Font Locking, AWK Mode Font Locking, Doc Comments, Font Locking
+@comment  node-name,  next,  previous,  up
+@section Miscellaneous Font Locking
+@comment !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
+In some languages, particularly in C++, there are constructs which are
+syntactically ambiguous---they could be either declarations or
+expressions, and @ccmode{} cannot tell for sure which.  Often such a
+construct is one of the operators @samp{*} or @samp{&} surrounded by
+two identifiers.
+
+Experience shows that very often when such a construct is a
+declaration it will be written with the operator touching exactly one
+of the identifiers, like:
+
+@example
+foo *bar
+@end example
+or
+@example
+foo& bar
+@end example
+
+.  Whether such code is fontified depends on the setting of
+@code{c-asymmetry-fontification-flag}.
+
+@defvar c-asymmetry-fontification-flag
+@vindex asymmetry-fontification-flag (c-)
+When @code{c-asymmetry-fontification-flag} is non-nil (which it is by
+default), code like the above, with white space either before or after
+the operator, but not both, is fontified as a declaration.  When the
+variable is nil, such a construct gets the default face.
+@end defvar
+
+When the construct is an expression there will often be white space
+both before and after the operator or there will be no white space
+around it at all, like:
+
+@example
+foo * bar
+@end example
+or
+@example
+foo&bar
+@end example
+.
+
+Such code is not fontified as a declaration.  (Typically, the
+identifiers don't get a non-default face.)
+
+For clarity's sake, we emphasize that the ``asymmetry'' rule in this
+section only applies when CC Mode cannot disambiguate a construct in
+any other way.
 
 @comment !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
-@node    AWK Mode Font Locking,  , Doc Comments, Font Locking
+@node    AWK Mode Font Locking,  , Misc Font Locking, Font Locking
 @comment  node-name,  next,  previous,  up
 @section AWK Mode Font Locking
 @comment !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
diff --git a/lisp/progmodes/cc-engine.el b/lisp/progmodes/cc-engine.el
index bdc77dc5028..de15d1d82fc 100644
--- a/lisp/progmodes/cc-engine.el
+++ b/lisp/progmodes/cc-engine.el
@@ -6243,9 +6243,9 @@ comment at the start of cc-engine.el for more info."
                     (eq (char-before) ?<))
         (c-backward-token-2)
         (when (eq (char-after) ?<)
-          (c-clear-<-pair-props-if-match-after beg)))
+          (c-clear-<-pair-props-if-match-after beg)
+          (setq new-beg (point))))
       (c-forward-syntactic-ws)
-      (setq new-beg (point))
 
       ;; ...Then the ones with < before end and > after end.
       (goto-char (if end-lit-limits (cdr end-lit-limits) end))
@@ -6254,9 +6254,9 @@ comment at the start of cc-engine.el for more info."
                   (eq (char-before) ?>))
         (c-end-of-current-token)
         (when (eq (char-before) ?>)
-          (c-clear->-pair-props-if-match-before end (1- (point)))))
+          (c-clear->-pair-props-if-match-before end (1- (point)))
+          (setq new-end (point))))
       (c-backward-syntactic-ws)
-      (setq new-end (point))
 
       ;; Extend the fontification region, if needed.
       (and new-beg
@@ -8863,7 +8863,29 @@ comment at the start of cc-engine.el for more info."
                  ;; it as a declaration if "a" has been used as a type
                  ;; somewhere else (if it's a known type we won't get here).
                  (setq maybe-expression t)
-                 (throw 'at-decl-or-cast t)))
+                 (throw 'at-decl-or-cast t))
+
+               ;; CASE 17.5
+               (when (and c-asymmetry-fontification-flag
+                          got-prefix-before-parens
+                          at-type
+                          (or (not got-suffix)
+                              at-decl-start))
+                 (let ((space-before-id
+                        (save-excursion
+                          (goto-char name-start)
+                          (or (bolp) (memq (char-before) '(?\  ?\t)))))
+                       (space-after-type
+                        (save-excursion
+                          (goto-char type-start)
+                          (and (c-forward-type)
+                               (progn (c-backward-syntactic-ws) t)
+                               (or (eolp)
+                                   (memq (char-after) '(?\  ?\t)))))))
+                   (when (not (eq (not space-before-id)
+                                  (not space-after-type)))
+                     (setq maybe-expression t)
+                     (throw 'at-decl-or-cast t)))))
 
            ;; CASE 18
            (when (and (not (memq context '(nil top)))
diff --git a/lisp/progmodes/cc-fonts.el b/lisp/progmodes/cc-fonts.el
index f623b9f3332..923f077b411 100644
--- a/lisp/progmodes/cc-fonts.el
+++ b/lisp/progmodes/cc-fonts.el
@@ -1117,6 +1117,124 @@ casts and declarations are fontified.  Used on level 2 and higher."
           (setq pos (point))))))     ; acts to make the `while' form continue.
   nil)
 
+(defun c-get-fontification-context (match-pos not-front-decl &optional toplev)
+  ;; Return a cons (CONTEXT . RESTRICTED-<>-ARGLISTS) for MATCH-POS.
+  ;; NOT-FRONT-DECL is non-nil when a declaration later in the buffer than
+  ;; MATCH-POS has already been parsed.  TOPLEV is non-nil when MATCH-POS is
+  ;; known to be at "top level", i.e. outside any braces, or directly inside a
+  ;; namespace, class, etc.
+  ;;
+  ;; CONTEXT is the fontification context of MATCH-POS, and is one of the
+  ;; following:
+  ;; 'decl     In a comma-separated declaration context (typically
+  ;;           inside a function declaration arglist).
+  ;; '<>       In an angle bracket arglist.
+  ;; 'arglist  Some other type of arglist.
+  ;; 'top      Some other context and point is at the top-level (either
+  ;;           outside any braces or directly inside a class or namespace,
+  ;;           etc.)
+  ;; nil       Some other context or unknown context.  Includes
+  ;;           within the parens of an if, for, ... construct.
+  ;; 'not-decl Definitely not in a declaration.
+  ;;
+  ;; RESTRICTED-<>-ARGLISTS is non-nil when a scan of template/generic
+  ;; arguments lists (i.e. lists enclosed by <...>) is more strict about what
+  ;; characters it allows within the list.
+  (let ((type (and (> match-pos (point-min))
+                   (c-get-char-property (1- match-pos) 'c-type))))
+    (cond ((not (memq (char-before match-pos) '(?\( ?, ?\[ ?< ?{)))
+           (cons (and toplev 'top) nil))
+          ;; A control flow expression or a decltype
+          ((and (eq (char-before match-pos) ?\()
+                (save-excursion
+                  (goto-char match-pos)
+                  (backward-char)
+                  (c-backward-token-2)
+                  (or (looking-at c-block-stmt-2-key)
+                      (looking-at c-block-stmt-1-2-key)
+                      (looking-at c-typeof-key))))
+           (cons nil t))
+          ;; Near BOB.
+          ((<= match-pos (point-min))
+           (cons 'arglist t))
+          ;; Got a cached hit in a declaration arglist.
+          ((eq type 'c-decl-arg-start)
+           (cons 'decl nil))
+          ;; We're inside (probably) a brace list.
+          ((eq type 'c-not-decl)
+           (cons 'not-decl nil))
+          ;; Inside a C++11 lambda function arglist.
+          ((and (c-major-mode-is 'c++-mode)
+                (eq (char-before match-pos) ?\()
+                (save-excursion
+                  (goto-char match-pos)
+                  (c-backward-token-2)
+                  (and
+                   (c-safe (goto-char (scan-sexps (point) -1)))
+                   (c-looking-at-c++-lambda-capture-list))))
+           (c-put-char-property (1- match-pos) 'c-type
+                                'c-decl-arg-start)
+           (cons 'decl nil))
+          ;; We're inside a brace list.
+          ((and (eq (char-before match-pos) ?{)
+                (save-excursion
+                  (goto-char (1- match-pos))
+                  (consp
+                   (c-looking-at-or-maybe-in-bracelist))))
+           (c-put-char-property (1- match-pos) 'c-type
+                                'c-not-decl)
+           (cons 'not-decl nil))
+          ;; We're inside an "ordinary" open brace.
+          ((eq (char-before match-pos) ?{)
+           (cons (and toplev 'top) nil))
+          ;; Inside an angle bracket arglist.
+          ((or (eq type 'c-<>-arg-sep)
+               (eq (char-before match-pos) ?<))
+           (cons '<> nil))
+          ;; Got a cached hit in some other type of arglist.
+          (type
+           (cons 'arglist t))
+          (not-front-decl
+           ;; The point is within the range of a previously
+           ;; encountered type decl expression, so the arglist
+           ;; is probably one that contains declarations.
+           ;; However, if `c-recognize-paren-inits' is set it
+           ;; might also be an initializer arglist.
+           ;;
+           ;; The result of this check is cached with a char
+           ;; property on the match token, so that we can look
+           ;; it up again when refontifying single lines in a
+           ;; multiline declaration.
+           (c-put-char-property (1- match-pos)
+                                'c-type 'c-decl-arg-start)
+           (cons 'decl nil))
+          ;; Got an open paren preceded by an arith operator.
+          ((and (eq (char-before match-pos) ?\()
+                (save-excursion
+                  (and (zerop (c-backward-token-2 2))
+                       (looking-at c-arithmetic-op-regexp))))
+           (cons nil nil))
+          ;; At start of a declaration inside a declaration paren.
+          ((save-excursion
+             (and (memq (char-before match-pos) '(?\( ?\,))
+                  (c-go-up-list-backward match-pos)
+                  (eq (char-after) ?\()
+                  (let ((type (c-get-char-property (point) 'c-type)))
+                    (or (memq type '(c-decl-arg-start c-decl-type-start))
+                        (and
+                         (progn (c-backward-syntactic-ws) t)
+                         (c-back-over-compound-identifier)
+                         (progn
+                           (c-backward-syntactic-ws)
+                           (or (bobp)
+                               (progn
+                                 (setq type (c-get-char-property (1- (point))
+                                                                 'c-type))
+                                 (memq type '(c-decl-arg-start
+                                              c-decl-type-start))))))))))
+           (cons 'decl nil))
+          (t (cons 'arglist t)))))
+
 (defun c-font-lock-declarations (limit)
   ;; Fontify all the declarations, casts and labels from the point to LIMIT.
   ;; Assumes that strings and comments have been fontified already.
@@ -1231,95 +1349,15 @@ casts and declarations are fontified.  Used on level 2 and higher."
             ;; "<" for the sake of C++-style template arglists.
             ;; Ignore "(" when it's part of a control flow construct
             ;; (e.g. "for (").
-            (let ((type (and (> match-pos (point-min))
-                             (c-get-char-property (1- match-pos) 'c-type))))
-              (cond ((not (memq (char-before match-pos) '(?\( ?, ?\[ ?< ?{)))
-                     (setq context (and toplev 'top)
-                           c-restricted-<>-arglists nil))
-                    ;; A control flow expression or a decltype
-                    ((and (eq (char-before match-pos) ?\()
-                          (save-excursion
-                            (goto-char match-pos)
-                            (backward-char)
-                            (c-backward-token-2)
-                            (or (looking-at c-block-stmt-2-key)
-                                (looking-at c-block-stmt-1-2-key)
-                                (looking-at c-typeof-key))))
-                     (setq context nil
-                           c-restricted-<>-arglists t))
-                    ;; Near BOB.
-                    ((<= match-pos (point-min))
-                     (setq context 'arglist
-                           c-restricted-<>-arglists t))
-                    ;; Got a cached hit in a declaration arglist.
-                    ((eq type 'c-decl-arg-start)
-                     (setq context 'decl
-                           c-restricted-<>-arglists nil))
-                    ;; We're inside (probably) a brace list.
-                    ((eq type 'c-not-decl)
-                     (setq context 'not-decl
-                           c-restricted-<>-arglists nil))
-                    ;; Inside a C++11 lambda function arglist.
-                    ((and (c-major-mode-is 'c++-mode)
-                          (eq (char-before match-pos) ?\()
-                          (save-excursion
-                            (goto-char match-pos)
-                            (c-backward-token-2)
-                            (and
-                             (c-safe (goto-char (scan-sexps (point) -1)))
-                             (c-looking-at-c++-lambda-capture-list))))
-                     (setq context 'decl
-                           c-restricted-<>-arglists nil)
-                     (c-put-char-property (1- match-pos) 'c-type
-                                          'c-decl-arg-start))
-                    ;; We're inside a brace list.
-                    ((and (eq (char-before match-pos) ?{)
-                          (save-excursion
-                            (goto-char (1- match-pos))
-                            (consp
-                             (c-looking-at-or-maybe-in-bracelist))))
-                     (setq context 'not-decl
-                           c-restricted-<>-arglists nil)
-                     (c-put-char-property (1- match-pos) 'c-type
-                                          'c-not-decl))
-                    ;; We're inside an "ordinary" open brace.
-                    ((eq (char-before match-pos) ?{)
-                     (setq context (and toplev 'top)
-                           c-restricted-<>-arglists nil))
-                    ;; Inside an angle bracket arglist.
-                    ((or (eq type 'c-<>-arg-sep)
-                         (eq (char-before match-pos) ?<))
-                     (setq context '<>
-                           c-restricted-<>-arglists nil))
-                    ;; Got a cached hit in some other type of arglist.
-                    (type
-                     (setq context 'arglist
-                           c-restricted-<>-arglists t))
-                    ((if inside-macro
-                         (< match-pos max-type-decl-end-before-token)
-                       (< match-pos max-type-decl-end))
-                     ;; The point is within the range of a previously
-                     ;; encountered type decl expression, so the arglist
-                     ;; is probably one that contains declarations.
-                     ;; However, if `c-recognize-paren-inits' is set it
-                     ;; might also be an initializer arglist.
-                     (setq context 'decl
-                           c-restricted-<>-arglists nil)
-                     ;; The result of this check is cached with a char
-                     ;; property on the match token, so that we can look
-                     ;; it up again when refontifying single lines in a
-                     ;; multiline declaration.
-                     (c-put-char-property (1- match-pos)
-                                          'c-type 'c-decl-arg-start))
-                    ;; Got an open paren preceded by an arith operator.
-                    ((and (eq (char-before match-pos) ?\()
-                          (save-excursion
-                            (and (zerop (c-backward-token-2 2))
-                                 (looking-at c-arithmetic-op-regexp))))
-                     (setq context nil
-                           c-restricted-<>-arglists nil))
-                    (t (setq context 'arglist
-                             c-restricted-<>-arglists t))))
+            (let ((got-context
+                   (c-get-fontification-context
+                    match-pos
+                    (< match-pos (if inside-macro
+                                     max-type-decl-end-before-token
+                                   max-type-decl-end))
+                    toplev)))
+              (setq context (car got-context)
+                    c-restricted-<>-arglists (cdr got-context)))
 
             ;; Check we haven't missed a preceding "typedef".
             (when (not (looking-at c-typedef-key))
diff --git a/lisp/progmodes/cc-mode.el b/lisp/progmodes/cc-mode.el
index 8326e6a6f29..20c63d4dbe2 100644
--- a/lisp/progmodes/cc-mode.el
+++ b/lisp/progmodes/cc-mode.el
@@ -1363,6 +1363,7 @@ Note that the style variables are always made local to the buffer."
   ;; This function is called indirectly from font locking stuff - either from
   ;; c-after-change (to prepare for after-change font-locking) or from font
   ;; lock context (etc.) fontification.
+  (goto-char pos)
   (let ((lit-start (c-literal-start))
         (new-pos pos)
         capture-opener
diff --git a/lisp/progmodes/cc-vars.el b/lisp/progmodes/cc-vars.el
index 1114b21381d..ccd4fd29940 100644
--- a/lisp/progmodes/cc-vars.el
+++ b/lisp/progmodes/cc-vars.el
@@ -1634,6 +1634,18 @@ names)."))
   :type 'c-extra-types-widget
   :group 'c)
 
+(defcustom c-asymmetry-fontification-flag t
+  "Whether to fontify certain ambiguous constructs by white space asymmetry.
+
+In the fontification engine, it is sometimes impossible to determine
+whether a construct is a declaration or an expression.  This happens
+particularly in C++, due to ambiguities in the language.  When such a
+construct is like \"foo * bar\" or \"foo &bar\", and this variable is non-nil
+(the default), the construct will be fontified as a declaration if there is
+white space either before or after the operator, but not both."
+  :type 'boolean
+  :group 'c)
+
 (defvar c-noise-macro-with-parens-name-re "\\<\\>")
 (defvar c-noise-macro-name-re "\\<\\>")