Add early init file, stop package-initialize insertion

Discussion on emacs-devel leading up to this change (approximately 150
messages):

- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00433.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00023.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-09/msg00599.html
- https://lists.gnu.org/archive/html/emacs-devel/2017-10/msg00332.html

* lisp/startup.el (early-init-file): New variable.
(load-user-init-file): New function.
(command-line): Load the early init file using `load-user-init-file'.
Move the check for an invalid username to just before that, and move
the initialization of the package system to just after.  Load the
regular init file using `load-user-init-file'.
* lisp/emacs-lisp/package.el (package--ensure-init-file): Remove
definition, usage, and documentation.
(package--init-file-ensured): Remove definition and usage.

* src/lread.c (Vuser_init_file): Note change in semantics due to its
usage while loading the early init file.

* doc/emacs/custom.texi: Document early init file.
* doc/emacs/package.texi: Document changes to when package-initialize
is called.  Change terminology for package 'loading'.
* doc/lispref/os.texi: Document early init file.  Update startup
summary.
* doc/lispref/package.texi: Document changes to when
package-initialize is called, and advise against calling it in the
init file.  Change terminology for package 'loading'.
* doc/misc/org.texi: Don't recommend to call package-initialize in the
init file.

6 files changed, 109 insertions, 65 deletions

diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index 2726690f09f..b985d12cde4 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -2167,6 +2167,7 @@ Manual}.
 * Terminal Init::       Each terminal type can have an init file.
 * Find Init::           How Emacs finds the init file.
 * Init Non-ASCII::      Using non-@acronym{ASCII} characters in an init file.
+* Early Init File::     Another init file, which is read early on.
 @end menu
 
 @node Init Syntax
@@ -2567,3 +2568,20 @@ instance:
 
 @noindent
 Type @kbd{C-q}, followed by the key you want to bind, to insert @var{char}.
+
+@node Early Init File
+@subsection The Early Init File
+@cindex early init file
+
+  Most customizations for Emacs can be put in the normal init file,
+@file{.emacs} or @file{~/.emacs.d/init.el}.  However, it is sometimes
+desirable to have customizations that take effect during Emacs startup
+earlier than the normal init file is processed.  Such customizations
+can be put in the early init file, @file{~/.emacs.d/early-init.el}.
+This file is loaded before the package system is initialized, so in it
+you can customize variables that affect the initialization process,
+such as @code{package-enable-at-startup} and @code{package-load-list}.
+@xref{Package Installation}.
+
+  For more information on the early init file, @pxref{Early Init
+File,,, elisp, The Emacs Lisp Reference Manual}.
diff --git a/doc/emacs/emacs.texi b/doc/emacs/emacs.texi
index 163b6f23d84..65affeb59fe 100644
--- a/doc/emacs/emacs.texi
+++ b/doc/emacs/emacs.texi
@@ -1163,6 +1163,7 @@ The Emacs Initialization File
 * Terminal Init::       Each terminal type can have an init file.
 * Find Init::           How Emacs finds the init file.
 * Init Non-ASCII::      Using non-@acronym{ASCII} characters in an init file.
+* Early Init File::     Another init file, which is read early on.
 
 Dealing with Emacs Trouble
 
diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index bc6afb7966a..6c7493790d3 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -241,57 +241,55 @@ lower-priority archives will not be shown in the menu, if the same
 package is available from a higher-priority archive.  (This is
 controlled by the value of @code{package-menu-hide-low-priority}.)
 
-  Once a package is downloaded and installed, it is @dfn{loaded} into
-the current Emacs session.  Loading a package is not quite the same as
-loading a Lisp library (@pxref{Lisp Libraries}); loading a package
-adds its directory to @code{load-path} and loads its autoloads.  The
-effect of a package's autoloads varies from package to package.  Most
-packages just make some new commands available, while others have more
+  Once a package is downloaded and installed, it is made available to
+the current Emacs session.  Making a package available adds its
+directory to @code{load-path} and loads its autoloads.  The effect of
+a package's autoloads varies from package to package.  Most packages
+just make some new commands available, while others have more
 wide-ranging effects on the Emacs session.  For such information,
 consult the package's help buffer.
 
-  By default, Emacs also automatically loads all installed packages in
-subsequent Emacs sessions.  This happens at startup, after processing
-the init file (@pxref{Init File}).  As an exception, Emacs does not
-load packages at startup if invoked with the @samp{-q} or
-@samp{--no-init-file} options (@pxref{Initial Options}).
+  After a package is installed, it is automatically made available by
+Emacs in all subsequent sessions.  This happens at startup, before
+processing the init file but after processing the early init file
+(@pxref{Early Init File,,, elisp, The Emacs Lisp Reference Manual}).
+As an exception, Emacs does not make packages available at startup if
+invoked with the @samp{-q} or @samp{--no-init-file} options
+(@pxref{Initial Options}).
 
 @vindex package-enable-at-startup
-  To disable automatic package loading, change the variable
-@code{package-enable-at-startup} to @code{nil}.
+  To keep Emacs from automatically making packages available at
+startup, change the variable @code{package-enable-at-startup} to
+@code{nil}.  You must do this in the early init file (@pxref{Early
+Init File,,, elisp, The Emacs Lisp Reference Manual}), as the variable
+is read before loading the regular init file.  Currently this variable
+cannot be set via Customize.
 
 @findex package-initialize
-  The reason automatic package loading occurs after loading the init
-file is that user options only receive their customized values after
-loading the init file, including user options which affect the
-packaging system.  In some circumstances, you may want to load
-packages explicitly in your init file (usually because some other code
-in your init file depends on a package).  In that case, your init file
-should call the function @code{package-initialize}.  It is up to you
-to ensure that relevant user options, such as @code{package-load-list}
-(see below), are set up prior to the @code{package-initialize} call.
-This will automatically set @code{package-enable-at-startup} to @code{nil}, to
-avoid loading the packages again after processing the init file.
-Alternatively, you may choose to completely inhibit package loading at
-startup, and invoke the command @kbd{M-x package-initialize} to load
-your packages manually.
+  If you have set @code{package-enable-at-startup} to @code{nil}, you
+can still make packages available either during or after startup.  To
+make installed packages available during startup, call the function
+@code{package-initialize} in your init file.  To make installed
+packages available after startup, invoke the command @kbd{M-x
+package-initialize}.
 
 @vindex package-load-list
-  For finer control over package loading, you can use the variable
-@code{package-load-list}.  Its value should be a list.  A list element
-of the form @code{(@var{name} @var{version})} tells Emacs to load
-version @var{version} of the package named @var{name}.  Here,
-@var{version} should be a version string (corresponding to a specific
-version of the package), or @code{t} (which means to load any
-installed version), or @code{nil} (which means no version; this
-disables the package, preventing it from being loaded).  A list
-element can also be the symbol @code{all}, which means to load the
-latest installed version of any package not named by the other list
-elements.  The default value is just @code{'(all)}.
-
-  For example, if you set @code{package-load-list} to @code{'((muse
-"3.20") all)}, then Emacs only loads version 3.20 of the @samp{muse}
-package, plus any installed version of packages other than
+  For finer control over which packages are made available at startup,
+you can use the variable @code{package-load-list}.  Its value should
+be a list.  A list element of the form @w{@code{(@var{name}
+@var{version})}} tells Emacs to make available version @var{version} of
+the package named @var{name}.  Here, @var{version} should be a version
+string (corresponding to a specific version of the package), or
+@code{t} (which means to make available any installed version), or
+@code{nil} (which means no version; this disables the package,
+preventing it from being made available).  A list element can also be
+the symbol @code{all}, which means to make available the latest
+installed version of any package not named by the other list elements.
+The default value is just @code{'(all)}.
+
+  For example, if you set @code{package-load-list} to @w{@code{'((muse
+"3.20") all)}}, then Emacs only makes available version 3.20 of the
+@samp{muse} package, plus any installed version of packages other than
 @samp{muse}.  Any other version of @samp{muse} that happens to be
 installed will be ignored.  The @samp{muse} package will be listed in
 the package menu with the @samp{held} status.
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 42be60449de..be2d148a1e2 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -95,6 +95,22 @@ if requested by environment variables such as @env{LANG}.
 @item
 It does some basic parsing of the command-line arguments.
 
+@item
+It loads your early init file (@pxref{Early Init File,,, emacs, The
+GNU Emacs Manual}).  This is not done if the options @samp{-q},
+@samp{-Q}, or @samp{--batch} were specified.  If the @samp{-u} option
+was specified, Emacs looks for the init file in that user's home
+directory instead.
+
+@item
+It calls the function @code{package-initialize} to activate any
+optional Emacs Lisp package that has been installed.  @xref{Packaging
+Basics}.  However, Emacs doesn't initialize packages when
+@code{package-enable-at-startup} is @code{nil} or when it's started
+with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
+initialize packages in the latter case, @code{package-initialize}
+should be called explicitly (e.g., via the @samp{--funcall} option).
+
 @vindex initial-window-system@r{, and startup}
 @vindex window-system-initialization-alist
 @item
@@ -154,15 +170,6 @@ It loads your abbrevs from the file specified by
 (@pxref{Abbrev Files, abbrev-file-name}).  This is not done if the
 option @samp{--batch} was specified.
 
-@item
-It calls the function @code{package-initialize} to activate any
-optional Emacs Lisp package that has been installed.  @xref{Packaging
-Basics}.  However, Emacs doesn't initialize packages when
-@code{package-enable-at-startup} is @code{nil} or when it's started
-with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
-initialize packages in the latter case, @code{package-initialize}
-should be called explicitly (e.g., via the @samp{--funcall} option).
-
 @vindex after-init-time
 @item
 It sets the variable @code{after-init-time} to the value of
@@ -361,6 +368,7 @@ Equivalent to @samp{-q --no-site-file --no-splash}.
 @cindex init file
 @cindex @file{.emacs}
 @cindex @file{init.el}
+@cindex @file{early-init.el}
 
   When you start Emacs, it normally attempts to load your @dfn{init
 file}.  This is either a file named @file{.emacs} or @file{.emacs.el}
@@ -384,6 +392,19 @@ file; this way, even if you have su'd, Emacs still loads your own init
 file.  If those environment variables are absent, though, Emacs uses
 your user-id to find your home directory.
 
+@cindex early init file
+  Emacs also attempts to load a second init file, called the
+@dfn{early init file}, if it exists.  This is a file named
+@file{early-init.el} in your @file{~/.emacs.d} directory.  The
+difference between the early init file and the regular init file is
+that the early init file is loaded much earlier during the startup
+process, so you can use it to customize some things that are
+initialized before loading the regular init file.  For example, you
+can customize the process of initializing the package system, by
+setting variables such as @var{package-load-list} or
+@var{package-enable-at-startup}.  @xref{Package Installation,,,
+emacs,The GNU Emacs Manual}.
+
 @cindex default init file
   An Emacs installation may have a @dfn{default init file}, which is a
 Lisp library named @file{default.el}.  Emacs finds this file through
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 21dfe1c2717..877aaf89a35 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -105,24 +105,32 @@ adds the package's content directory to @code{load-path}, and
 evaluates the autoload definitions in @file{@var{name}-autoloads.el}.
 
   Whenever Emacs starts up, it automatically calls the function
-@code{package-initialize} to load installed packages.  This is done
-after loading the init file and abbrev file (if any) and before
-running @code{after-init-hook} (@pxref{Startup Summary}).  Automatic
-package loading is disabled if the user option
-@code{package-enable-at-startup} is @code{nil}.
+@code{package-initialize} to make installed packages available to the
+current session.  This is done after loading the early init file, but
+before loading the regular init file (@pxref{Startup Summary}).
+Packages are not automatically made available if the user option
+@code{package-enable-at-startup} is set to @code{nil} in the early
+init file.
 
 @deffn Command package-initialize &optional no-activate
 This function initializes Emacs' internal record of which packages are
-installed, and loads them.  The user option @code{package-load-list}
-specifies which packages to load; by default, all installed packages
-are loaded.  If called during startup, this function also sets
+installed, and makes the packages available to the current session.
+The user option @code{package-load-list} specifies which packages to
+make available; by default, all installed packages are made available.
+If called during startup, this function also sets
 @code{package-enable-at-startup} to @code{nil}, to avoid accidentally
-loading the packages twice.  @xref{Package Installation,,, emacs, The
-GNU Emacs Manual}.
+evaluating package autoloads more than once.  @xref{Package
+Installation,,, emacs, The GNU Emacs Manual}.
 
 The optional argument @var{no-activate}, if non-@code{nil}, causes
 Emacs to update its record of installed packages without actually
-loading them; it is for internal use only.
+making them available; it is for internal use only.
+
+In most cases, you should not need to call @code{package-initialize},
+as this is done automatically during startup.  Simply make sure to put
+any code that should run before @code{package-initialize} in the early
+init file, and any code that should run after it in the primary init
+file (@xref{Init File,,, emacs, The GNU Emacs Manual}).
 @end deffn
 
 @node Simple Packages
diff --git a/doc/misc/org.texi b/doc/misc/org.texi
index aa3b029ab7c..68aa01ca18c 100644
--- a/doc/misc/org.texi
+++ b/doc/misc/org.texi
@@ -890,9 +890,7 @@ Elisp libraries.  You can install Org with @kbd{M-x package-install RET org}.
 been visited, i.e., where no Org built-in function have been loaded.
 Otherwise autoload Org functions will mess up the installation.
 
-Then, to make sure your Org configuration is taken into account, initialize
-the package system with @code{(package-initialize)} in your Emacs init file
-before setting any Org option.  If you want to use Org's package repository,
+If you want to use Org's package repository,
 check out the @uref{http://orgmode.org/elpa.html, Org ELPA page}.
 
 @subsubheading Downloading Org as an archive