Make emacs prefer an existing ~/.emacs.d to an existing XDG location

* doc/emacs/custom.texi (Find Init): Update description of how Emacs
finds its init file directory and the interaction with
$XDG_CONFIG_HOME
(Early Init File): Correct XDG location of early-init.el

* etc/NEWS: Update description to make it clear the ~/.emacs.d is
preferred, even if the XDG location exists.

* lisp/startup.el: Prefer ~/.emacs.d even if the XDG location exists.

* lib-src/emacsclient.c (open_config): Prefer home directory the XDG
location.

4 files changed, 76 insertions, 65 deletions

diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index 1958a86b92d..2e04a6b22d3 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -2655,26 +2655,37 @@ library.  @xref{Hooks}.
 
   Emacs normally finds your init file in a location under your home
 directory.  @xref{Init File}.  By default this location is
-@file{~/.config/emacs/init.el} where @file{~/} stands for your home directory.
+@file{~/.emacs.d/init.el} where @file{~/} stands for your home directory.
 This default can be overridden as described below.
 
-  If @env{XDG_CONFIG_HOME} is set in your environment, its
-value replaces @file{~/.config} in the name of the default
-init file.
-
-  If the default init file's parent directory does not exist but the
-directory @file{~/.emacs.d} does exist, Emacs looks for your init file
+Emacs looks for your init file
 using the filenames @file{~/.emacs.el}, @file{~/.emacs}, or
 @file{~/.emacs.d/init.el}; you can choose to use any one of these
 names.  (Note that only the locations directly in your home directory
-have a leading dot in the location's basename.)  Although this is
-backward-compatible with older Emacs versions, modern POSIX platforms
-prefer putting your initialization files under @file{~/.config} so
-that troubleshooting a problem that might be due to a bad init file,
-or archiving a collection of init files, can be done by renaming that
-directory.  To help older Emacs versions find configuration files in
-their current default locations, you can execute the following
-Emacs Lisp code:
+have a leading dot in the location's basename.)
+
+Emacs can also look in an XDG-compatible location for @file{init.el},
+the default is the directory @file{~/.config/emacs}.  This can be
+overriden by setting @env{XDG_CONFIG_HOME} in your environment, its
+value replaces @file{~/.config} in the name of the default XDG init
+file.  However @file{~/.emacs.d} and @file{~/.emacs} are always
+preferred if they exist, which means that you must delete or rename
+them in order to use the XDG location.
+
+Note also that if neither the XDG location nor @file{~/.emacs.d}
+exist, then Emacs will create @file{~/.emacs.d} (and therefore use it
+during subsequent invocations).
+
+Emacs will set @var{user-emacs-directory} to the directory it decides
+to use.
+
+Although this is backward-compatible with older Emacs versions, modern
+POSIX platforms prefer putting your initialization files under
+@file{~/.config} so that troubleshooting a problem that might be due
+to a bad init file, or archiving a collection of init files, can be
+done by renaming that directory.  To help older Emacs versions find
+configuration files in their current default locations, you can
+execute the following Emacs Lisp code:
 
 @example
 (make-symbolic-link ".config/emacs" "~/.emacs.d")
@@ -2694,7 +2705,7 @@ otherwise, it looks up the home directory corresponding to that user
 name in the system's data base of users.
 
   For brevity the rest of the Emacs documentation generally uses just
-the current default location @file{~/.config/emacs/init.el} for the
+the current default location @file{~/.emacs.d/init.el} for the
 init file.
 @c  LocalWords:  backtab
 
@@ -2740,7 +2751,7 @@ Type @kbd{C-q}, followed by the key you want to bind, to insert @var{char}.
 @xref{Init File}.  However, it is sometimes desirable
 to have customizations that take effect during Emacs startup earlier than the
 normal init file is processed.  Such customizations can be put in the early
-init file, @file{~/.config/emacs.d/early-init.el} or @file{~/.emacs.d/early-init.el}.  This file is loaded before the
+init file, @file{~/.config/emacs/early-init.el} or @file{~/.emacs.d/early-init.el}.  This file is loaded before the
 package system and GUI is initialized, so in it you can customize variables
 that affect frame appearance as well as the package initialization process,
 such as @code{package-enable-at-startup}, @code{package-load-list}, and
diff --git a/etc/NEWS b/etc/NEWS
index c1e00395c84..38fff3f7568 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -161,25 +161,24 @@ Emacs 28.
 * Startup Changes in Emacs 27.1
 
 +++
-** Emacs now uses the XDG convention for init files.
-For example, it looks for init.el in "~/.config/emacs/init.el", and
-similarly for other init files.
-
-The 'XDG_CONFIG_HOME' environment variable (which defaults to "~/.config")
-specifies the parent directory of these and other configuration files,
-and will override their traditional locations (the home directory,
-"~/.emacs.d", etc.).
-
-Emacs will still look for init files in their traditional locations if
-"$XDG_CONFIG_HOME/emacs" does not exist, but "~/.emacs.d" or
-"~/.emacs" does exist, so invoking Emacs with XDG_CONFIG_HOME='/nowhere'
-might be useful if your new-location init files are scrambled, or if
-you want to force Emacs to ignore files under 'XDG_CONFIG_HOME' for
-some other reason.
-
-If neither "$XDG_CONFIG_HOME/emacs" nor "~/.emacs.d" exist, Emacs will
-now default to "$XDG_CONFIG_HOME/emacs", and will create that directory
-and set 'user-emacs-directory' to point to it.
+** Emacs can now use the XDG convention for init files.
+The 'XDG_CONFIG_HOME' environment variable (which defaults to
+"~/.config") specifies the XDG configuration parent directory.  Emacs
+checks for "init.el" and other configuration files inside the "emacs"
+subdirectory of 'XDG_CONFIG_HOME', i.e. "$XDG_CONFIG_HOME/emacs/init.el"
+
+However, Emacs will still initially look for init files in their
+traditional locations if "~/.emacs.d" or "~/.emacs" exist, even if
+"$XDG_CONFIG_HOME/emacs" also exists.  This means that you must delete
+or rename any existing "~/.emacs.d" and "~/.emacs" to enable use of
+the XDG directory.
+
+If "~/.emacs.d" does not exist, and Emacs has decided to use it
+(i.e. "$XDG_CONFIG_HOME/emacs" does not exist), Emacs will create it.
+Emacs will never create "$XDG_CONFIG_HOME/emacs".
+
+Whichever directory Emacs decides to use, it will set
+'user-emacs-directory' to point to it.
 
 +++
 ** Emacs can now be configured using an early init file.
diff --git a/lib-src/emacsclient.c b/lib-src/emacsclient.c
index 645ff04c6d4..204064f1871 100644
--- a/lib-src/emacsclient.c
+++ b/lib-src/emacsclient.c
@@ -924,21 +924,22 @@ open_config (char const *home, char const *xdg, char const *config_file)
   char *configname = xmalloc (max (xdgsubdirsize, homesubdirsizemax)
                               + strlen (config_file));
   FILE *config;
-  if (xdg || home)
+
+  if (home)
     {
-      strcpy ((xdg
-               ? stpcpy (stpcpy (configname, xdg), "/emacs/server/")
-               : stpcpy (stpcpy (configname, home), "/.config/emacs/server/")),
-              config_file);
+      strcpy (stpcpy (stpcpy (configname, home), "/.emacs.d/server/"),
+              config_file);
       config = fopen (configname, "rb");
     }
   else
     config = NULL;
 
-  if (! config && home)
+  if (! config && (xdg || home))
     {
-      strcpy (stpcpy (stpcpy (configname, home), "/.emacs.d/server/"),
-              config_file);
+      strcpy ((xdg
+               ? stpcpy (stpcpy (configname, xdg), "/emacs/server/")
+               : stpcpy (stpcpy (configname, home), "/.config/emacs/server/")),
+              config_file);
       config = fopen (configname, "rb");
     }
 
diff --git a/lisp/startup.el b/lisp/startup.el
index 2a85c004da1..1f545c66922 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -497,28 +497,28 @@ DIRS are relative."
 (defvar startup--xdg-config-home-emacs)
 
 ;; Return the name of the init file directory for Emacs, assuming
-;; XDG-DIR is the XDG location and USER-NAME is the user name.
-;; If USER-NAME is nil or "", use the current user.
-;; Prefer the XDG location unless it does does not exist and the
-;; .emacs.d location does exist.
+;; XDG-DIR is the XDG location and USER-NAME is the user name.  If
+;; USER-NAME is nil or "", use the current user.  Prefer the XDG
+;; location only if the .emacs.d location does not exist.
 (defun startup--xdg-or-homedot (xdg-dir user-name)
-  (if (file-exists-p xdg-dir)
-      xdg-dir
-    (let ((emacs-d-dir (concat "~" user-name
-                               (if (eq system-type 'ms-dos)
-                                   "/_emacs.d/"
-                                 "/.emacs.d/"))))
-      (if (or (file-exists-p emacs-d-dir)
-              (if (eq system-type 'windows-nt)
-                  (if (file-directory-p (concat "~" user-name))
-                      (directory-files (concat "~" user-name) nil
-                                       "\\`[._]emacs\\(\\.elc?\\)?\\'"))
-                (file-exists-p (concat "~" init-file-user
-                                       (if (eq system-type 'ms-dos)
-                                           "/_emacs"
-                                         "/.emacs")))))
-          emacs-d-dir
-        xdg-dir))))
+  (let ((emacs-d-dir (concat "~" user-name
+                             (if (eq system-type 'ms-dos)
+                                 "/_emacs.d/"
+                               "/.emacs.d/"))))
+    (cond
+     ((or (file-exists-p emacs-d-dir)
+          (if (eq system-type 'windows-nt)
+              (if (file-directory-p (concat "~" user-name))
+                  (directory-files (concat "~" user-name) nil
+                                   "\\`[._]emacs\\(\\.elc?\\)?\\'"))
+            (file-exists-p (concat "~" init-file-user
+                                   (if (eq system-type 'ms-dos)
+                                       "/_emacs"
+                                     "/.emacs")))))
+      emacs-d-dir)
+     ((file-exists-p xdg-dir)
+      xdg-dir)
+     (t emacs-d-dir))))
 
 (defun normal-top-level ()
   "Emacs calls this function when it first starts up.