Expand host names in Tramp's ad-hoc multi-hop file names

* doc/misc/tramp.texi (Quick Start Guide): Improve wording.
(Change file name syntax): Say, that `tramp-file-name-regexp' is
not constant.
(Ad-hoc multi-hops): Explain host name expansion.

* etc/NEWS: Mention that host names in Tramp ad-hoc multi-hop file
names must match the previous hop for methods like "su" or "sudo".
Fix typos.

* lisp/net/tramp.el (tramp-find-method, tramp-find-user):
Adapt docstring.
(tramp-find-host): Mark default value.
(tramp-dissect-file-name): Expand host name for hops.
(tramp-dissect-hop-name, tramp-make-tramp-hop-name): New defuns.
(tramp-clear-passwd): Simplify.

* test/lisp/net/tramp-tests.el (tramp-test02-file-name-dissect)
(tramp-test02-file-name-dissect-simplified)
(tramp-test02-file-name-dissect-separate)
(tramp-test26-file-name-completion): Extend tests.

4 files changed, 188 insertions, 61 deletions

diff --git a/doc/misc/tramp.texi b/doc/misc/tramp.texi
index 7c5ebf334ae..128501c3908 100644
--- a/doc/misc/tramp.texi
+++ b/doc/misc/tramp.texi
@@ -464,7 +464,7 @@ this case it is written as @code{host#port}.
 @cindex @option{plink} method
 
 If your local host runs an SSH client, and the remote host runs an SSH
-server, the most simple remote file name is
+server, the simplest remote file name is
 @file{@trampfn{ssh,user@@host,/path/to/file}}.  The remote file name
 @file{@trampfn{ssh,,}} opens a remote connection to yourself on the
 local host, and is taken often for testing @value{tramp}.
@@ -2459,9 +2459,10 @@ and @code{user@@} parts are optional.
 
 @defvar tramp-file-name-regexp
 This variable keeps a regexp which matches the selected remote file
-name syntax.  However, it is not recommended to use this variable in
-external packages, a call of @code{file-remote-p} is much more
-appropriate.
+name syntax.  Its value changes after every call of
+@code{tramp-change-syntax}.  However, it is not recommended to use
+this variable in external packages, a call of @code{file-remote-p} is
+much more appropriate.
 @ifinfo
 @pxref{Magic File Names, , , elisp}
 @end ifinfo
@@ -2585,9 +2586,9 @@ directory contents.
 @cindex multi-hop, ad-hoc
 @cindex proxy hosts, ad-hoc
 
-@value{tramp} file name syntax can accommodate ad hoc specification of
+@value{tramp} file name syntax can accommodate ad-hoc specification of
 multiple proxies without using @code{tramp-default-proxies-alist}
-configuration setup(@pxref{Multi-hops}).
+configuration setup (@pxref{Multi-hops}).
 
 Each proxy is specified using the same syntax as the remote host
 specification minus the file name part.  Each hop is separated by a
@@ -2600,8 +2601,6 @@ proxy @samp{bird@@bastion} to a remote file on @samp{you@@remotehost}:
 @kbd{C-x C-f @value{prefix}ssh@value{postfixhop}bird@@bastion|ssh@value{postfixhop}you@@remotehost@value{postfix}/path @key{RET}}
 @end example
 
-Proxies can take patterns @code{%h} or @code{%u}.
-
 @value{tramp} adds the ad-hoc definitions on the fly to
 @code{tramp-default-proxies-alist} and is available for re-use
 during that Emacs session.  Subsequent @value{tramp} connections to
@@ -2618,6 +2617,17 @@ For ad-hoc definitions to be saved automatically in
 @end lisp
 @end defopt
 
+Ad-hoc proxies can take patterns @code{%h} or @code{%u} like in
+@code{tramp-default-proxies-alist}.  The following file name expands
+to user @code{root} on host @code{remotehost}, starting with an
+@option{ssh} session on host @code{remotehost}:
+@samp{@value{prefix}ssh@value{postfixhop}%h|su@value{postfixhop}remotehost@value{postfix}}.
+
+On the other hand, if a trailing hop does not specifiy a host name,
+the host name of the previous hop is reused. Therefore, the following
+file name is equivalent to the previous example:
+@samp{@value{prefix}ssh@value{postfixhop}remotehost|su@value{postfixhop}@value{postfix}}.
+
 
 @node Remote processes
 @section Integration with other Emacs packages
diff --git a/etc/NEWS b/etc/NEWS
index 09f0362fed7..be6668ed42e 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -172,7 +172,7 @@ the data.
 
 +++
 ** The Network Security Manager now allows more fine-grained control
-of what checks to run via the `network-security-protocol-checks'
+of what checks to run via the 'network-security-protocol-checks'
 variable.
 
 +++
@@ -356,7 +356,7 @@ shown in the currently selected window.
 ** Comint
 
 +++
-*** 'send-invisible' is now an obsolete alias for `comint-send-invisible'.
+*** 'send-invisible' is now an obsolete alias for 'comint-send-invisible'.
 Also, 'shell-strip-ctrl-m' is declared obsolete.
 
 +++
@@ -391,7 +391,7 @@ facilities to aid more casual SQL developers layout queries and
 complex expressions.
 
 *** 'sql-use-indent-support' (default t) enables SQL indention support.
-The `sql-indent' package from ELPA must be installed to get the
+The 'sql-indent' package from ELPA must be installed to get the
 indentation support in 'sql-mode' and 'sql-interactive-mode'.
 
 *** 'sql-mode-hook' and 'sql-interactive-mode-hook' changed.
@@ -420,7 +420,7 @@ This enables more efficient backends.  See the docstring of
 
 ** Package
 
-*** New function `package-get-version` lets packages query their own version.
+*** New function 'package-get-version' lets packages query their own version.
 Example use in auctex.el: (defconst auctex-version (package-get-version))
 
 *** New 'package-quickstart' feature.
@@ -747,6 +747,10 @@ are obsoleted in GVFS.
 *** The user option 'tramp-ignored-file-name-regexp' allows to disable
 Tramp for some look-alike remote file names.
 
++++
+*** For some connection methods, like "su" or "sudo", the host name in
+ad-hoc multi-hop file names must match the previous hop.
+
 ** Register
 ---
 *** The return value of method 'register-val-describe' includes the
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index e629ce17315..2e6cdf999a5 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -1267,7 +1267,7 @@ entry does not exist, return nil."
 (defun tramp-find-method (method user host)
   "Return the right method string to use.
 This is METHOD, if non-nil. Otherwise, do a lookup in
-`tramp-default-method-alist'."
+`tramp-default-method-alist' and `tramp-default-method'."
   (when (and method
              (or (string-equal method "")
                  (string-equal method tramp-default-method-marker)))
@@ -1292,7 +1292,7 @@ This is METHOD, if non-nil. Otherwise, do a lookup in
 (defun tramp-find-user (method user host)
   "Return the right user string to use.
 This is USER, if non-nil. Otherwise, do a lookup in
-`tramp-default-user-alist'."
+`tramp-default-user-alist' and `tramp-default-user'."
   (let ((result
          (or user
              (let ((choices tramp-default-user-alist)
@@ -1312,18 +1312,24 @@ This is USER, if non-nil. Otherwise, do a lookup in
 
 (defun tramp-find-host (method user host)
   "Return the right host string to use.
-This is HOST, if non-nil. Otherwise, it is `tramp-default-host'."
-  (or (and (> (length host) 0) host)
-      (let ((choices tramp-default-host-alist)
-            lhost item)
-        (while choices
-          (setq item (pop choices))
-          (when (and (string-match (or (nth 0 item) "") (or method ""))
-                     (string-match (or (nth 1 item) "") (or user "")))
-            (setq lhost (nth 2 item))
-            (setq choices nil)))
-        lhost)
-      tramp-default-host))
+This is HOST, if non-nil. Otherwise, do a lookup in
+`tramp-default-host-alist' and `tramp-default-host'."
+  (let ((result
+         (or (and (> (length host) 0) host)
+             (let ((choices tramp-default-host-alist)
+                   lhost item)
+               (while choices
+                 (setq item (pop choices))
+                 (when (and (string-match (or (nth 0 item) "") (or method ""))
+                            (string-match (or (nth 1 item) "") (or user "")))
+                   (setq lhost (nth 2 item))
+                   (setq choices nil)))
+               lhost)
+             tramp-default-host)))
+    ;; We must mark, whether a default value has been used.
+    (if (or (> (length host) 0) (null result))
+        result
+      (propertize result 'tramp-default t))))
 
 (defun tramp-dissect-file-name (name &optional nodefault)
   "Return a `tramp-file-name' structure of NAME, a remote file name.
@@ -1343,7 +1349,7 @@ default values are used."
             (host      (match-string (nth 3 tramp-file-name-structure) name))
             (localname (match-string (nth 4 tramp-file-name-structure) name))
             (hop       (match-string (nth 5 tramp-file-name-structure) name))
-            domain port)
+            domain port v)
         (when user
           (when (string-match tramp-user-with-domain-regexp user)
             (setq domain (match-string 2 user)
@@ -1359,14 +1365,34 @@ default values are used."
             (setq host (replace-match "" nil t host))))
 
         (unless nodefault
-          (setq method (tramp-find-method method user host)
-                user (tramp-find-user method user host)
-                host (tramp-find-host method user host)))
+          (when hop
+            (setq v (tramp-dissect-hop-name hop)
+                  hop (and hop (tramp-make-tramp-hop-name v))))
+          (let ((tramp-default-host
+                 (or (and v (not (string-match "%h" (tramp-file-name-host v)))
+                          (tramp-file-name-host v))
+                     tramp-default-host)))
+            (setq method (tramp-find-method method user host)
+                  user (tramp-find-user method user host)
+                  host (tramp-find-host method user host)
+                  hop
+                  (and hop
+                       (format-spec hop (format-spec-make ?h host ?u user))))))
 
         (make-tramp-file-name
          :method method :user user :domain domain :host host :port port
          :localname localname :hop hop)))))
 
+(defun tramp-dissect-hop-name (name &optional nodefault)
+  "Return a `tramp-file-name' structure of `hop' part of NAME.
+See `tramp-dissect-file-name' for details."
+  (tramp-dissect-file-name
+   (concat
+    tramp-prefix-format
+    (replace-regexp-in-string
+     (concat tramp-postfix-hop-regexp "$") tramp-postfix-host-format name))
+   nodefault))
+
 (defun tramp-buffer-name (vec)
   "A name for the connection buffer VEC."
   (let ((method (tramp-file-name-method vec))
@@ -1433,6 +1459,14 @@ the form (METHOD USER DOMAIN HOST PORT LOCALNAME &optional HOP)."
             tramp-postfix-host-format
             localname)))
 
+(defun tramp-make-tramp-hop-name (vec)
+  "Construct a Tramp hop name from VEC."
+  (replace-regexp-in-string
+   tramp-prefix-regexp ""
+   (replace-regexp-in-string
+    (concat tramp-postfix-host-regexp "$") tramp-postfix-hop-format
+    (tramp-make-tramp-file-name vec 'noloc))))
+
 (defun tramp-completion-make-tramp-file-name (method user host localname)
   "Construct a Tramp file name from METHOD, USER, HOST and LOCALNAME.
 It must not be a complete Tramp file name, but as long as there are
@@ -2313,7 +2347,7 @@ Falls back to normal file name handler if no Tramp file name handler exists."
                        (tramp-message
                         v 1 "Interrupt received in operation %s"
                         (cons operation args)))
-                     ;; Propagate the quit signal.
+                     ;; Propagate the signal.
                      (signal (car err) (cdr err)))
 
                     ;; When we are in completion mode, some failed
@@ -4508,13 +4542,7 @@ Invokes `password-read' if available, `read-passwd' else."
         (hop (tramp-file-name-hop vec)))
     (when hop
       ;; Clear also the passwords of the hops.
-      (tramp-clear-passwd
-       (tramp-dissect-file-name
-        (concat
-         tramp-prefix-format
-         (replace-regexp-in-string
-          (concat tramp-postfix-hop-regexp "$")
-          tramp-postfix-host-format hop)))))
+      (tramp-clear-passwd (tramp-dissect-hop-name hop)))
     (auth-source-forget
      `(:max 1 ,(and user-domain :user) ,user-domain
        :host ,host-port :port ,method))
diff --git a/test/lisp/net/tramp-tests.el b/test/lisp/net/tramp-tests.el
index 6a08cbb5ab2..ceda70947c8 100644
--- a/test/lisp/net/tramp-tests.el
+++ b/test/lisp/net/tramp-tests.el
@@ -763,8 +763,8 @@ handled properly.  BODY shall not contain a timeout."
         "|-:user2@host2"
         "|-:user3@host3:/path/to/file"))
       (format "/%s:%s@%s|%s:%s@%s|%s:%s@%s:"
-              "-" "user1" "host1"
-              "-" "user2" "host2"
+              "method1" "user1" "host1"
+              "method2" "user2" "host2"
               "method3" "user3" "host3")))
 
     ;; Expand `tramp-default-user-alist'.
@@ -778,9 +778,9 @@ handled properly.  BODY shall not contain a timeout."
         "/method1:host1"
         "|method2:host2"
         "|method3:host3:/path/to/file"))
-      (format "/%s:%s|%s:%s|%s:%s@%s:"
-              "method1" "host1"
-              "method2" "host2"
+      (format "/%s:%s@%s|%s:%s@%s|%s:%s@%s:"
+              "method1" "user1" "host1"
+              "method2" "user2" "host2"
               "method3" "user3" "host3")))
 
     ;; Expand `tramp-default-host-alist'.
@@ -794,9 +794,36 @@ handled properly.  BODY shall not contain a timeout."
         "/method1:user1@"
         "|method2:user2@"
         "|method3:user3@:/path/to/file"))
-      (format "/%s:%s@|%s:%s@|%s:%s@%s:"
-              "method1" "user1"
-              "method2" "user2"
+      (format "/%s:%s@%s|%s:%s@%s|%s:%s@%s:"
+              "method1" "user1" "host1"
+              "method2" "user2" "host2"
+              "method3" "user3" "host3")))
+
+    ;; Ad-hoc user name and host name expansion.
+    (setq tramp-default-method-alist nil
+          tramp-default-user-alist nil
+          tramp-default-host-alist nil)
+    (should
+     (string-equal
+      (file-remote-p
+       (concat
+        "/method1:user1@host1"
+        "|method2:user2@"
+        "|method3:user3@:/path/to/file"))
+      (format "/%s:%s@%s|%s:%s@%s|%s:%s@%s:"
+              "method1" "user1" "host1"
+              "method2" "user2" "host1"
+              "method3" "user3" "host1")))
+    (should
+     (string-equal
+      (file-remote-p
+       (concat
+        "/method1:%u@%h"
+        "|method2:%u@%h"
+        "|method3:user3@host3:/path/to/file"))
+      (format "/%s:%s@%s|%s:%s@%s|%s:%s@%s:"
+              "method1" "user3" "host3"
+              "method2" "user3" "host3"
               "method3" "user3" "host3")))))
 
 (ert-deftest tramp-test02-file-name-dissect-simplified ()
@@ -1067,9 +1094,9 @@ handled properly.  BODY shall not contain a timeout."
               "/host1"
               "|host2"
               "|host3:/path/to/file"))
-            (format "/%s|%s|%s@%s:"
-                    "host1"
-                    "host2"
+            (format "/%s@%s|%s@%s|%s@%s:"
+                    "user1" "host1"
+                    "user2" "host2"
                     "user3" "host3")))
 
           ;; Expand `tramp-default-host-alist'.
@@ -1083,9 +1110,35 @@ handled properly.  BODY shall not contain a timeout."
               "/user1@"
               "|user2@"
               "|user3@:/path/to/file"))
-            (format "/%s@|%s@|%s@%s:"
-                    "user1"
-                    "user2"
+            (format "/%s@%s|%s@%s|%s@%s:"
+                    "user1" "host1"
+                    "user2" "host2"
+                    "user3" "host3")))
+
+          ;; Ad-hoc user name and host name expansion.
+          (setq tramp-default-user-alist nil
+                tramp-default-host-alist nil)
+          (should
+           (string-equal
+            (file-remote-p
+             (concat
+              "/user1@host1"
+              "|user2@"
+              "|user3@:/path/to/file"))
+            (format "/%s@%s|%s@%s|%s@%s:"
+                    "user1" "host1"
+                    "user2" "host1"
+                    "user3" "host1")))
+          (should
+           (string-equal
+            (file-remote-p
+             (concat
+              "/%u@%h"
+              "|%u@%h"
+              "|user3@host3:/path/to/file"))
+            (format "/%s@%s|%s@%s|%s@%s:"
+                    "user3" "host3"
+                    "user3" "host3"
                     "user3" "host3"))))
 
       ;; Exit.
@@ -1670,9 +1723,9 @@ handled properly.  BODY shall not contain a timeout."
               "/[/user1@host1"
               "|/user2@host2"
               "|/user3@host3]/path/to/file"))
-            (format "/[/%s@%s|/%s@%s|%s/%s@%s]"
-                    "user1" "host1"
-                    "user2" "host2"
+            (format "/[%s/%s@%s|%s/%s@%s|%s/%s@%s]"
+                    "method1" "user1" "host1"
+                    "method2" "user2" "host2"
                     "method3" "user3" "host3")))
 
           ;; Expand `tramp-default-user-alist'.
@@ -1686,9 +1739,9 @@ handled properly.  BODY shall not contain a timeout."
               "/[method1/host1"
               "|method2/host2"
               "|method3/host3]/path/to/file"))
-            (format "/[%s/%s|%s/%s|%s/%s@%s]"
-                    "method1" "host1"
-                    "method2" "host2"
+            (format "/[%s/%s@%s|%s/%s@%s|%s/%s@%s]"
+                    "method1" "user1" "host1"
+                    "method2" "user2" "host2"
                     "method3" "user3" "host3")))
 
           ;; Expand `tramp-default-host-alist'.
@@ -1702,9 +1755,36 @@ handled properly.  BODY shall not contain a timeout."
               "/[method1/user1@"
               "|method2/user2@"
               "|method3/user3@]/path/to/file"))
-            (format "/[%s/%s@|%s/%s@|%s/%s@%s]"
-                    "method1" "user1"
-                    "method2" "user2"
+            (format "/[%s/%s@%s|%s/%s@%s|%s/%s@%s]"
+                    "method1" "user1" "host1"
+                    "method2" "user2" "host2"
+                    "method3" "user3" "host3")))
+
+          ;; Ad-hoc user name and host name expansion.
+          (setq tramp-default-method-alist nil
+                tramp-default-user-alist nil
+                tramp-default-host-alist nil)
+          (should
+           (string-equal
+            (file-remote-p
+             (concat
+              "/[method1/user1@host1"
+              "|method2/user2@"
+              "|method3/user3@]/path/to/file"))
+            (format "/[%s/%s@%s|%s/%s@%s|%s/%s@%s]"
+                    "method1" "user1" "host1"
+                    "method2" "user2" "host1"
+                    "method3" "user3" "host1")))
+          (should
+           (string-equal
+            (file-remote-p
+             (concat
+              "/[method1/%u@%h"
+              "|method2/%u@%h"
+              "|method3/user3@host3]/path/to/file"))
+            (format "/[%s/%s@%s|%s/%s@%s|%s/%s@%s]"
+                    "method1" "user3" "host3"
+                    "method2" "user3" "host3"
                     "method3" "user3" "host3"))))
 
       ;; Exit.
@@ -3491,6 +3571,7 @@ This tests also `make-symbolic-link', `file-truename' and `add-name-to-file'."
   (when (not (memq system-type '(cygwin windows-nt)))
     (let ((method (file-remote-p tramp-test-temporary-file-directory 'method))
           (host (file-remote-p tramp-test-temporary-file-directory 'host))
+          (vec (tramp-dissect-file-name tramp-test-temporary-file-directory))
           (orig-syntax tramp-syntax))
       (when (and (stringp host) (string-match tramp-host-with-port-regexp host))
         (setq host (match-string 1 host)))
@@ -3501,6 +3582,10 @@ This tests also `make-symbolic-link', `file-truename' and `add-name-to-file'."
                (if (tramp--test-expensive-test)
                    (tramp-syntax-values) `(,orig-syntax)))
             (tramp-change-syntax syntax)
+            ;; This has cleaned up all connection data, which are used
+            ;; for completion.  We must refill the cache.
+            (tramp-set-connection-property vec "property" nil)
+
             (let ;; This is needed for the `simplified' syntax.
                 ((method-marker
                   (if (zerop (length tramp-method-regexp))