* lisp/emacs-lisp/package.el: New quickstart feature

(package--quickstart-pkgs): New var.
(package-activate-1): Obey and fill it.
(package-activate-all): New function.
(package-initialize): Call it.
Set package-initialized before activating the packages.
(package-installed-p): Make it work before package.el is initialized in
the case where min-version is not specified.
(package-install, package-delete): Refresh the quickstart if applicable.
(package-quickstart, package-quickstart-file): New vars.
(package--quickstart-maybe-refresh, package-quickstart-refresh):
New functions.

* lisp/startup.el (command-line): Use package-activate-all rather than
package-initialize.

* doc/lispref/package.texi (Packaging Basics):
* doc/emacs/package.texi (Package Installation):
* doc/lispref/os.texi (Startup Summary): Refer to package-activate-all.

8 files changed, 192 insertions, 48 deletions

diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index a69888cdbd6..c3cfaabb8d3 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -2602,17 +2602,16 @@ Type @kbd{C-q}, followed by the key you want to bind, to insert @var{char}.
 @cindex early init file
 
   Most customizations for Emacs can be put in the normal init file,
-@file{.emacs} or @file{~/.emacs.d/init.el}.  However, it is sometimes
-desirable to have customizations that take effect during Emacs startup
-earlier than the normal init file is processed.  Such customizations
-can be put in the early init file, @file{~/.emacs.d/early-init.el}.
-This file is loaded before the package system is initialized, so in it
-you can customize variables that affect the package initialization
-process, such as @code{package-enable-at-startup},
-@code{package-load-list}, and @code{package-user-dir}.  Note that
-variables like @code{package-archives} which only affect the
-installation of new packages, and not the process of making
-already-installed packages available, may be customized in the regular
+@file{.emacs} or @file{~/.emacs.d/init.el}.  However, it is sometimes desirable
+to have customizations that take effect during Emacs startup earlier than the
+normal init file is processed.  Such customizations can be put in the early
+init file, @file{~/.emacs.d/early-init.el}.  This file is loaded before the
+package system and GUI is initialized, so in it you can customize variables
+that affect frame appearance as well as the package initialization process,
+such as @code{package-enable-at-startup}, @code{package-load-list}, and
+@code{package-user-dir}.  Note that variables like @code{package-archives}
+which only affect the installation of new packages, and not the process of
+making already-installed packages available, may be customized in the regular
 init file.  @xref{Package Installation}.
 
   For more information on the early init file, @pxref{Init File,,,
diff --git a/doc/emacs/package.texi b/doc/emacs/package.texi
index be749348729..43f5a8497d9 100644
--- a/doc/emacs/package.texi
+++ b/doc/emacs/package.texi
@@ -263,13 +263,13 @@ startup, change the variable @code{package-enable-at-startup} to
 is read before loading the regular init file.  Currently this variable
 cannot be set via Customize.
 
-@findex package-initialize
+@findex package-activate-all
   If you have set @code{package-enable-at-startup} to @code{nil}, you
 can still make packages available either during or after startup.  To
 make installed packages available during startup, call the function
-@code{package-initialize} in your init file.  To make installed
-packages available after startup, invoke the command @kbd{M-x
-package-initialize}.
+@code{package-activate-all} in your init file.  To make installed
+packages available after startup, invoke the command @kbd{M-:
+(package-activate-all) RET}.
 
 @vindex package-load-list
   For finer control over which packages are made available at startup,
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 0e30ad519a8..77ecb667f4f 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -103,12 +103,12 @@ was specified, Emacs looks for the init file in that user's home
 directory instead.
 
 @item
-It calls the function @code{package-initialize} to activate any
+It calls the function @code{package-activate-all} to activate any
 optional Emacs Lisp package that has been installed.  @xref{Packaging
-Basics}.  However, Emacs doesn't initialize packages when
+Basics}.  However, Emacs doesn't activate the packages when
 @code{package-enable-at-startup} is @code{nil} or when it's started
 with one of the options @samp{-q}, @samp{-Q}, or @samp{--batch}.  To
-initialize packages in the latter case, @code{package-initialize}
+activate the packages in the latter case, @code{package-activate-all}
 should be called explicitly (e.g., via the @samp{--funcall} option).
 
 @vindex initial-window-system@r{, and startup}
diff --git a/doc/lispref/package.texi b/doc/lispref/package.texi
index 7e7a8cd9bc8..37c1ee6697d 100644
--- a/doc/lispref/package.texi
+++ b/doc/lispref/package.texi
@@ -105,16 +105,15 @@ adds the package's content directory to @code{load-path}, and
 evaluates the autoload definitions in @file{@var{name}-autoloads.el}.
 
   Whenever Emacs starts up, it automatically calls the function
-@code{package-initialize} to make installed packages available to the
+@code{package-activate-all} to make installed packages available to the
 current session.  This is done after loading the early init file, but
 before loading the regular init file (@pxref{Startup Summary}).
 Packages are not automatically made available if the user option
 @code{package-enable-at-startup} is set to @code{nil} in the early
 init file.
 
-@deffn Command package-initialize &optional no-activate
-This function initializes Emacs' internal record of which packages are
-installed, and makes the packages available to the current session.
+@defun package-activate-all
+This function makes the packages available to the current session.
 The user option @code{package-load-list} specifies which packages to
 make available; by default, all installed packages are made available.
 If called during startup, this function also sets
@@ -122,15 +121,20 @@ If called during startup, this function also sets
 evaluating package autoloads more than once.  @xref{Package
 Installation,,, emacs, The GNU Emacs Manual}.
 
-The optional argument @var{no-activate}, if non-@code{nil}, causes
-Emacs to update its record of installed packages without actually
-making them available; it is for internal use only.
-
-In most cases, you should not need to call @code{package-initialize},
+In most cases, you should not need to call @code{package-activate-all},
 as this is done automatically during startup.  Simply make sure to put
-any code that should run before @code{package-initialize} in the early
+any code that should run before @code{package-activate-all} in the early
 init file, and any code that should run after it in the primary init
 file (@pxref{Init File,,, emacs, The GNU Emacs Manual}).
+@end defun
+
+@deffn Command package-initialize &optional no-activate
+This function initializes Emacs' internal record of which packages are
+installed, and then calls @code{package-activate-all}.
+
+The optional argument @var{no-activate}, if non-@code{nil}, causes
+Emacs to update its record of installed packages without actually
+making them available.
 @end deffn
 
 @node Simple Packages
diff --git a/etc/NEWS b/etc/NEWS
index 2f43125cefa..04774c13e55 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -59,14 +59,20 @@ package system is initialized given that initialization now happens
 before loading the regular init file (see below).
 
 +++
-** Emacs now calls 'package-initialize' before loading the init file.
+** Installed packages are now activated *before* loading the init file.
 This is part of a change intended to eliminate the behavior of
 package.el inserting a call to 'package-initialize' into the init
 file, which was previously done when Emacs was started.  As a result
 of this change, it is no longer necessary to call 'package-initialize'
-in your init file.  However, if your init file changes the values of
-'package-load-list' or 'package-user-dir', then that code needs to be
-moved to the early init file (see above).
+in your init file.
+
+However, if your init file changes the values of 'package-load-list' or
+'package-user-dir', or sets 'package-enable-at-startup' to nil then it won't
+work right without some adjustment:
+- you can move that code to the early init file (see above), so those settings
+  apply before Emacs tries to activate the packages.
+- you can use the new 'package-quickstart` so activation of packages does not
+  need to pay attention to 'package-load-list' or 'package-user-dir' any more.
 
 
 * Changes in Emacs 27.1
@@ -149,6 +155,17 @@ for abbrevs that have them.
 It now treats the optional 2nd argument to mean that the URL should be
 shown in the currently selected window.
 
+** Package
+*** New 'package-quickstart' feature
+When 'package-quickstart' is non-nil, package.el precomputes a big autoloads
+file so that activation of packages can be done much faster, which can speed up
+your startup significantly.
+It also causes variables like package-user-dir and package-load-list to be
+consulted when 'package-quickstart-refresh' is run rather than at startup so
+you don't need to set them in your early init file.
+
+*** New function 'package-activate-all'.
+
 ** Ecomplete
 *** The ecomplete sorting has changed to a decay-based algorithm.
 This can be controlled by the new `ecomplete-sort-predicate' variable.
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 67533679b99..b9fdf732ef4 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -681,6 +681,9 @@ PKG-DESC is a `package-desc' object."
 (defvar Info-directory-list)
 (declare-function info-initialize "info" ())
 
+(defvar package--quickstart-pkgs t
+  "If set to a list, we're computing the set of pkgs to activate.")
+
 (defun package--load-files-for-activation (pkg-desc reload)
   "Load files for activating a package given by PKG-DESC.
 Load the autoloads file, and ensure `load-path' is setup.  If
@@ -723,7 +726,10 @@ correspond to previously loaded files (those returned by
             (message "Unable to activate package `%s'.\nRequired package `%s-%s' is unavailable"
                      name (car req) (package-version-join (cadr req)))
             (throw 'exit nil))))
-      (package--load-files-for-activation pkg-desc reload)
+      (if (listp package--quickstart-pkgs)
+          ;; We're only collecting the set of packages to activate!
+          (push pkg-desc package--quickstart-pkgs)
+        (package--load-files-for-activation pkg-desc reload))
       ;; Add info node.
       (when (file-exists-p (expand-file-name "dir" pkg-dir))
         ;; FIXME: not the friendliest, but simple.
@@ -1463,18 +1469,34 @@ that code in the early init-file."
   (setq package-enable-at-startup nil)
   (package-load-all-descriptors)
   (package-read-all-archive-contents)
+  (setq package--initialized t)
   (unless no-activate
+    (package-activate-all))
+  ;; This uses `package--mapc' so it must be called after
+  ;; `package--initialized' is t.
+  (package--build-compatibility-table))
+
+(defvar package-quickstart-file)
+
+;;;###autoload
+(defun package-activate-all ()
+  "Activate all installed packages.
+The variable `package-load-list' controls which packages to load."
+  (setq package-enable-at-startup nil)
+  (if (file-readable-p package-quickstart-file)
+      ;; Skip load-source-file-function which would slow us down by a factor
+      ;; 2 (this assumes we were careful to save this file so it doesn't need
+      ;; any decoding).
+      (let ((load-source-file-function nil))
+        (load package-quickstart-file))
+    (unless package--initialized
+      (package-initialize t))
     (dolist (elt package-alist)
       (condition-case err
           (package-activate (car elt))
         ;; Don't let failure of activation of a package arbitrarily stop
         ;; activation of further packages.
-        (error (message "%s" (error-message-string err))))))
-  (setq package--initialized t)
-  ;; This uses `package--mapc' so it must be called after
-  ;; `package--initialized' is t.
-  (package--build-compatibility-table))
-
+        (error (message "%s" (error-message-string err)))))))
 
 ;;;; Populating `package-archive-contents' from archives
 ;; This subsection populates the variables listed above from the
@@ -1856,18 +1878,26 @@ If PACKAGE is a symbol, it is the package name and MIN-VERSION
 should be a version list.
 
 If PACKAGE is a `package-desc' object, MIN-VERSION is ignored."
-  (unless package--initialized (error "package.el is not yet initialized!"))
-  (if (package-desc-p package)
-      (let ((dir (package-desc-dir package)))
+  (cond
+   ((package-desc-p package)
+    (let ((dir (package-desc-dir package)))
         (and (stringp dir)
-             (file-exists-p dir)))
+             (file-exists-p dir))))
+   ((and (not package--initialized)
+         (null min-version)
+         package-activated-list)
+    ;; We used the quickstart: make it possible to use package-installed-p
+    ;; even before package is fully initialized.
+    (memq package package-activated-list))
+   ((not package--initialized) (error "package.el is not yet initialized!"))
+   (t
     (or
      (let ((pkg-descs (cdr (assq package package-alist))))
        (and pkg-descs
             (version-list-<= min-version
                              (package-desc-version (car pkg-descs)))))
      ;; Also check built-in packages.
-     (package-built-in-p package min-version))))
+     (package-built-in-p package min-version)))))
 
 (defun package-download-transaction (packages)
   "Download and install all the packages in PACKAGES.
@@ -1918,7 +1948,9 @@ to install it but still mark it as selected."
                      (package-compute-transaction (list pkg)
                                                   (package-desc-reqs pkg)))
                  (package-compute-transaction () (list (list pkg))))))
-        (package-download-transaction transaction)
+        (progn
+          (package-download-transaction transaction)
+          (package--quickstart-maybe-refresh))
       (message "`%s' is already installed" name))))
 
 (defun package-strip-rcs-id (str)
@@ -2090,7 +2122,9 @@ If NOSAVE is non-nil, the package is not removed from
              (delete pkg-desc pkgs)
              (unless (cdr pkgs)
                (setq package-alist (delq pkgs package-alist))))
-           (message "Package `%s' deleted." (package-desc-full-name pkg-desc))))))
+           (package--quickstart-maybe-refresh)
+           (message "Package `%s' deleted."
+                    (package-desc-full-name pkg-desc))))))
 
 ;;;###autoload
 (defun package-reinstall (pkg)
@@ -3415,6 +3449,95 @@ The list is displayed in a buffer named `*Packages*'."
   (interactive)
   (list-packages t))
 
+;;;; Quickstart: precompute activation actions for faster start up.
+
+;; Activating packages via `package-initialize' is costly: for N installed
+;; packages, it needs to read all N <pkg>-pkg.el files first to decide
+;; which packages to activate, and then again N <pkg>-autoloads.el files.
+;; To speed this up, we precompute a mega-autoloads file which is the
+;; concatenation of all those <pkg>-autoloads.el, so we can activate
+;; all packages by loading this one file (and hence without initializing
+;; package.el).
+
+;; Other than speeding things up, this also offers a bootstrap feature:
+;; it lets us activate packages according to package-load-list and
+;; package-user-dir even before those vars are set.
+
+(defcustom package-quickstart nil
+  "Precompute activation actions to speed up startup.
+This requires the use of `package-quickstart-refresh' every time the
+activations need to be changed, such as when `package-load-list' is modified."
+  :type 'boolean)
+
+(defcustom package-quickstart-file
+  (locate-user-emacs-file "package-quickstart.el")
+  "Location of the file used to speed up activation of packages at startup."
+  :type 'file)
+
+(defun package--quickstart-maybe-refresh ()
+  (if package-quickstart
+      ;; FIXME: Delay refresh in case we're installing/deleting
+      ;; several packages!
+      (package-quickstart-refresh)
+    (delete-file package-quickstart-file)))
+
+(defun package-quickstart-refresh ()
+  "(Re)Generate the `package-quickstart-file'."
+  (interactive)
+  (package-initialize 'no-activate)
+  (require 'info)
+  (let ((package--quickstart-pkgs ())
+        ;; Pretend we haven't activated anything yet!
+        (package-activated-list ())
+        ;; Make sure we can load this file without load-source-file-function.
+        (coding-system-for-write 'emacs-internal)
+        (Info-directory-list '("")))
+    (dolist (elt package-alist)
+      (condition-case err
+          (package-activate (car elt))
+        ;; Don't let failure of activation of a package arbitrarily stop
+        ;; activation of further packages.
+        (error (message "%s" (error-message-string err)))))
+    (setq package--quickstart-pkgs (nreverse package--quickstart-pkgs))
+    (with-temp-file package-quickstart-file
+      (emacs-lisp-mode)                 ;For `syntax-ppss'.
+      (insert ";;; Quickstart file to activate all packages at startup  -*- lexical-binding:t -*-\n")
+      (insert ";; ¡¡ This file is autogenerated by `package-quickstart-refresh', DO NOT EDIT !!\n\n")
+      (dolist (pkg package--quickstart-pkgs)
+        (let* ((file
+                ;; Prefer uncompiled files (and don't accept .so files).
+                (let ((load-suffixes '(".el" ".elc")))
+                  (locate-library (package--autoloads-file-name pkg))))
+               (pfile (prin1-to-string file)))
+          (insert "(let ((load-file-name " pfile "))\n")
+          (insert-file-contents file)
+          ;; Fixup the special #$ reader form and throw away comments.
+          (while (re-search-forward "#\\$\\|^;\\(.*\n\\)" nil 'move)
+            (unless (nth 8 (syntax-ppss))
+              (replace-match (if (match-end 1) "" pfile) t t)))
+          (unless (bolp) (insert "\n"))
+          (insert ")\n")))
+      (pp `(setq package-activated-list
+                 (append ',(mapcar #'package-desc-name package--quickstart-pkgs)
+                         package-activated-list))
+          (current-buffer))
+      (let ((info-dirs (butlast Info-directory-list)))
+        (when info-dirs
+          (pp `(progn (require 'info)
+                      (info-initialize)
+                      (setq Info-directory-list
+                            (append ',info-dirs Info-directory-list)))
+              (current-buffer))))
+      ;; Use `\s' instead of a space character, so this code chunk is not
+      ;; mistaken for an actual file-local section of package.el.
+      (insert "
+;; Local\sVariables:
+;; version-control: never
+;; no-byte-compile: t
+;; no-update-autoloads: t
+;; End:
+"))))
+
 (provide 'package)
 
 ;;; package.el ends here
diff --git a/lisp/files.el b/lisp/files.el
index bad2639fa62..1491f6049c2 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -3627,7 +3627,8 @@ local variables, but directory-local variables may still be applied."
                                  (push (cons (if (eq var 'eval)
                                                  'eval
                                                (indirect-variable var))
-                                             val) result))))))
+                                             val)
+                                       result))))))
                     (forward-line 1))))))))
       ;; Now we've read all the local variables.
       ;; If HANDLE-MODE is t, return whether the mode was specified.
diff --git a/lisp/startup.el b/lisp/startup.el
index 2669342edae..1faeabf23b8 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1185,7 +1185,7 @@ please check its value")
                                 (package--description-file subdir)
                                 subdir))))
                    (throw 'package-dir-found t)))))))
-       (package-initialize))
+       (package-activate-all))
 
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.