emacs/test/lisp, branch scratch/package-security Emacs is the extensible, customizable, self-documenting real-time display editor. Support expiration of metadata by package archives 2020-11-21T23:38:35+00:00 Stefan Kangas 2020-09-07T05:31:56+00:00 bcde5f86c5a7f3a84115807520631a4f12fb6f70 Expiring package metadata is done by checking the timestamp in package archive file. This is intended to limit the effectiveness of a replay attack. The onus is on the package archives to implement a secure and reasonable policy. (Debian uses 7 days before metadata expires.) Together with package checksums, this adds sufficient protection against metadata replay attacks. (Bug#19479) * lisp/emacs-lisp/package.el (package-check-timestamp): New defcustom. (bad-timestamp): New error. (package--parse-header-from-buffer) (package--parse-valid-until-from-buffer) (package--parse-last-updated-from-buffer) (package--archive-verify-timestamp) (package--archive-verify-not-expired) (package--compare-archive-timestamps) (package--check-archive-timestamp): New defuns. (package--download-one-archive): Check timestamp of the 'archive-contents' file using above functions. It is only checked if it exists, which makes this change backwards compatible. * lisp/calendar/iso8601.el (iso8601-parse): Add autoload cookie. * test/lisp/emacs-lisp/package-tests.el (package-test-parse-valid-until-from-buffer) (package-test-parse-last-updated-from-buffer) (package-test-archive-verify-timestamp) (package-test-check-archive-timestamp) (package-test-check-archive-timestamp/not-expired) (package-test-check-archive-timestamp/expired): New tests. * test/lisp/emacs-lisp/package-resources/archives/older/archive-contents: * test/lisp/emacs-lisp/package-resources/archives/newer/archive-contents: New files. * doc/lispref/package.texi (Package Archives, Archive Web Server): Document how to increase the security of a package archive using checksums, signing and timestamps. 
Expiring package metadata is done by checking the timestamp in package
archive file.  This is intended to limit the effectiveness of a replay
attack.  The onus is on the package archives to implement a secure and
reasonable policy.  (Debian uses 7 days before metadata expires.)

Together with package checksums, this adds sufficient protection
against metadata replay attacks.  (Bug#19479)

* lisp/emacs-lisp/package.el (package-check-timestamp): New defcustom.
(bad-timestamp): New error.
(package--parse-header-from-buffer)
(package--parse-valid-until-from-buffer)
(package--parse-last-updated-from-buffer)
(package--archive-verify-timestamp)
(package--archive-verify-not-expired)
(package--compare-archive-timestamps)
(package--check-archive-timestamp): New defuns.
(package--download-one-archive): Check timestamp of the
'archive-contents' file using above functions.  It is only checked if
it exists, which makes this change backwards compatible.

* lisp/calendar/iso8601.el (iso8601-parse): Add autoload cookie.

* test/lisp/emacs-lisp/package-tests.el
(package-test-parse-valid-until-from-buffer)
(package-test-parse-last-updated-from-buffer)
(package-test-archive-verify-timestamp)
(package-test-check-archive-timestamp)
(package-test-check-archive-timestamp/not-expired)
(package-test-check-archive-timestamp/expired): New tests.

* test/lisp/emacs-lisp/package-resources/archives/older/archive-contents:
* test/lisp/emacs-lisp/package-resources/archives/newer/archive-contents:
New files.

* doc/lispref/package.texi (Package Archives, Archive Web Server):
Document how to increase the security of a package archive using
checksums, signing and timestamps.
Support package checksum verification 2020-11-21T23:38:35+00:00 Stefan Kangas 2019-10-04T08:36:14+00:00 733e674af4f66ba7e9f0614b931c44484acce2b9 Package checksum verification is the first step towards protecting users of package.el against replay attacks. Signing individual packages still allows a hostile actor to distribute an out-of-date package containing a known security defect. To avoid that, we need to distribute signed package metadata (the ELPA "archive-contents" file) and checksums for the individual packages together. (Bug#19479) A subsequent patch will add support for last-update and expiration timestamps in "archive-contents", without which the protection against replay attacks will be largely ineffective. Taken together, this feature will make signing individual packages obsolete. We will instead rely on signing the metadata, package checksums and timestamps. Note that individual package signatures should still be distributed for a long time still to support old versions of Emacs. For more on replay attacks, see: https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html * lisp/emacs-lisp/package.el (package-verify-checksums): New defcustom. (package-desc, package--ac-desc) (package--add-to-archive-contents, package-install-from-archive): New fields 'size' and 'checksums'. (package-desc-filename): New function. (package-error): New error type. (bad-signature): Inherit from error type 'package-error'. (bad-checksum, bad-size): New error types. (package-insecure-hash-algorithms): New constant. (package--verify-package-checksum) (package--verify-package-size): New function to verify that the checksum and size of a package corresponds to the checksum and size data in the "archive-contents" file on the package archive. (package--show-verify-checksum-error): New function to show details of an error on checksum verification. * lisp/emacs-lisp/package-x.el (package-upload-buffer-internal): Update to use above new fields 'size' and 'checksums'. * test/lisp/emacs-lisp/package-tests.el (package-test-refresh-contents) (package-test-install-single-from-archive) (package-test-list-filter-by-archive) (package-test-list-filter-by-status): Update tests. (with-install-using-checksum): New macro. (package-test-install-wrong-size-single) (package-test-install-wrong-size-tar): New tests for size checking. (package-test-install-with-checksum/single-valid) (package-test-install-with-checksum/single-invalid) (package-test-install-with-checksum/tar-valid) (package-test-install-with-checksum/tar-invalid): New tests for installing packages with checksums. (package-test-verification-text) (package-tests-valid-md5-checksum) (package-tests-valid-sha256-checksum) (package-tests-valid-sha512-checksum): New variables. (package-tests--run-verify-checksums-test): New macro. (package-test-verify-package-checksums-nil/ignore-invalid) (package-test-verify-package-checksums-allow-missing) (package-test-verify-package-checksums-allow-missing/missing) (package-test-verify-package-checksums-allow-missing/ignore-unsupported) (package-test-verify-package-checksums-t) (package-test-verify-package-checksums-t/invalid-fails) (package-test-verify-package-checksums-t/missing-fails) (package-test-verify-package-checksums-all) (package-test-verify-package-checksums-all/invalid-fails) (package-test-verify-package-checksums-all/missing-fails) (package-test-verify-package-checksums-all/no-supported-hash-fails) (package-test-verify-package-checksums-all/ignore-unsupported) (package-test-verify-package-size): New tests for the checksum support. * test/lisp/emacs-lisp/package-resources/archive-contents: * test/lisp/emacs-lisp/package-resources/checksum-invalid-1.0.el: * test/lisp/emacs-lisp/package-resources/checksum-valid-123.el: * test/lisp/emacs-lisp/package-resources/checksum-valid-tar-0.99.tar: * test/lisp/emacs-lisp/package-resources/checksum-valid-tar-0.99.tar: New test data files. * doc/emacs/package.texi (Package Installation): Document package checksum checking. * etc/NEWS: Announce it. 
Package checksum verification is the first step towards protecting
users of package.el against replay attacks.  Signing individual
packages still allows a hostile actor to distribute an out-of-date
package containing a known security defect.  To avoid that, we need to
distribute signed package metadata (the ELPA "archive-contents" file)
and checksums for the individual packages together.  (Bug#19479)

A subsequent patch will add support for last-update and expiration
timestamps in "archive-contents", without which the protection against
replay attacks will be largely ineffective.

Taken together, this feature will make signing individual packages
obsolete.  We will instead rely on signing the metadata, package
checksums and timestamps.  Note that individual package signatures
should still be distributed for a long time still to support old
versions of Emacs.

For more on replay attacks, see:
https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html

* lisp/emacs-lisp/package.el (package-verify-checksums): New
defcustom.
(package-desc, package--ac-desc)
(package--add-to-archive-contents, package-install-from-archive): New
fields 'size' and 'checksums'.
(package-desc-filename): New function.

(package-error): New error type.
(bad-signature): Inherit from error type 'package-error'.
(bad-checksum, bad-size): New error types.
(package-insecure-hash-algorithms): New constant.
(package--verify-package-checksum)
(package--verify-package-size): New function to verify that the
checksum and size of a package corresponds to the checksum and size
data in the "archive-contents" file on the package archive.
(package--show-verify-checksum-error): New function to show
details of an error on checksum verification.

* lisp/emacs-lisp/package-x.el (package-upload-buffer-internal):
Update to use above new fields 'size' and 'checksums'.

* test/lisp/emacs-lisp/package-tests.el (package-test-refresh-contents)
(package-test-install-single-from-archive)
(package-test-list-filter-by-archive)
(package-test-list-filter-by-status): Update tests.
(with-install-using-checksum): New macro.
(package-test-install-wrong-size-single)
(package-test-install-wrong-size-tar): New tests for size checking.
(package-test-install-with-checksum/single-valid)
(package-test-install-with-checksum/single-invalid)
(package-test-install-with-checksum/tar-valid)
(package-test-install-with-checksum/tar-invalid): New tests for
installing packages with checksums.
(package-test-verification-text)
(package-tests-valid-md5-checksum)
(package-tests-valid-sha256-checksum)
(package-tests-valid-sha512-checksum): New variables.
(package-tests--run-verify-checksums-test): New macro.
(package-test-verify-package-checksums-nil/ignore-invalid)
(package-test-verify-package-checksums-allow-missing)
(package-test-verify-package-checksums-allow-missing/missing)
(package-test-verify-package-checksums-allow-missing/ignore-unsupported)
(package-test-verify-package-checksums-t)
(package-test-verify-package-checksums-t/invalid-fails)
(package-test-verify-package-checksums-t/missing-fails)
(package-test-verify-package-checksums-all)
(package-test-verify-package-checksums-all/invalid-fails)
(package-test-verify-package-checksums-all/missing-fails)
(package-test-verify-package-checksums-all/no-supported-hash-fails)
(package-test-verify-package-checksums-all/ignore-unsupported)
(package-test-verify-package-size): New tests for the checksum
support.

* test/lisp/emacs-lisp/package-resources/archive-contents:
* test/lisp/emacs-lisp/package-resources/checksum-invalid-1.0.el:
* test/lisp/emacs-lisp/package-resources/checksum-valid-123.el:
* test/lisp/emacs-lisp/package-resources/checksum-valid-tar-0.99.tar:
* test/lisp/emacs-lisp/package-resources/checksum-valid-tar-0.99.tar:
New test data files.

* doc/emacs/package.texi (Package Installation): Document package
checksum checking.
* etc/NEWS: Announce it.
Use cl-letf instead of unwind-protect in a test 2020-11-20T22:28:03+00:00 Mauro Aranda 2020-11-20T22:28:03+00:00 abd15e088e99b1c6334a427879fead0d557b7447 * test/lisp/cus-edit-tests.el (cus-edit-tests-customize-saved/show-obsolete): Good use case for cl-letf, so use it. Suggested by Stefan Monnier <monnier@iro.umontreal.ca> in: https://lists.gnu.org/archive/html/emacs-devel/2020-11/msg00914.html 
* test/lisp/cus-edit-tests.el (cus-edit-tests-customize-saved/show-obsolete):
Good use case for cl-letf, so use it.
Suggested by Stefan Monnier <monnier@iro.umontreal.ca> in:
https://lists.gnu.org/archive/html/emacs-devel/2020-11/msg00914.html
Take care of a FIXME in cus-edit-tests.el 2020-11-20T21:37:30+00:00 Mauro Aranda 2020-11-20T21:37:30+00:00 6924320aebce9ba885877da45e43d8d573da8bf6 * test/lisp/cus-edit-tests.el (cus-edit-tests-customize-saved/show-obsolete): Add a fake saved-value property, to be able check that the obsolete option is present in the Customize buffer. Expect the test to pass now. 
* test/lisp/cus-edit-tests.el (cus-edit-tests-customize-saved/show-obsolete):
Add a fake saved-value property, to be able check that the obsolete
option is present in the Customize buffer.  Expect the test to pass
now.
Hide obsolete options in most customize commands 2020-11-20T13:33:25+00:00 Stefan Kangas 2020-11-20T13:33:25+00:00 b4b1bd6e03d739871dd3ba51e8de0be2d272b766 * lisp/cus-edit.el (custom--filter-obsolete-variables): New defun. * lisp/cus-edit.el (customize-changed-options) (customize-apropos, custom-group-value-create): Hide obsolete user options. (Bug#44598) * test/lisp/cus-edit-tests.el: New file. 
* lisp/cus-edit.el (custom--filter-obsolete-variables): New defun.
* lisp/cus-edit.el (customize-changed-options)
(customize-apropos, custom-group-value-create): Hide obsolete user
options.  (Bug#44598)
* test/lisp/cus-edit-tests.el: New file.
Remove some compat code for old versions and XEmacs 2020-11-19T19:43:51+00:00 Stefan Kangas 2020-11-19T05:42:46+00:00 75555b5b6bc517911404fc769b02f583e40f6c35 * lisp/ibuf-ext.el (ibuffer-old-saved-filters-warning) (ibuffer-maybe-save-stuff): Assume customize-save-variable is bound; it is autoloaded. * lisp/cedet/semantic/symref/grep.el (semantic-symref-perform-search): * lisp/password-cache.el (password-cache-remove): * lisp/cedet/semantic/bovine/el.el (semantic-dependency-tag-file): Remove Emacs 21 compat code. * lisp/cedet/semantic/sort.el (semantic-string-lessp-ci): Remove Emacs 20 compat code. * test/lisp/cedet/semantic-utest.el (semantic-utest-temp-directory): * lisp/mail/supercite.el (sc-ask): Remove XEmacs compat code. * lisp/progmodes/idlw-shell.el (idlwave-shell-mode): * lisp/progmodes/idlwave.el (idlwave-mode): Remove commented out compat code. 
* lisp/ibuf-ext.el (ibuffer-old-saved-filters-warning)
(ibuffer-maybe-save-stuff): Assume customize-save-variable is
bound; it is autoloaded.
* lisp/cedet/semantic/symref/grep.el (semantic-symref-perform-search):
* lisp/password-cache.el (password-cache-remove):
* lisp/cedet/semantic/bovine/el.el (semantic-dependency-tag-file):
Remove Emacs 21 compat code.
* lisp/cedet/semantic/sort.el (semantic-string-lessp-ci):
Remove Emacs 20 compat code.
* test/lisp/cedet/semantic-utest.el (semantic-utest-temp-directory):
* lisp/mail/supercite.el (sc-ask): Remove XEmacs compat code.
* lisp/progmodes/idlw-shell.el (idlwave-shell-mode):
* lisp/progmodes/idlwave.el (idlwave-mode): Remove commented out
compat code.
Test that substitute-command-keys preserves text properties 2020-11-17T01:21:17+00:00 Stefan Kangas 2020-11-17T01:17:14+00:00 27655f9f38ab8037b599ce20341662d767fdc58c * test/lisp/help-tests.el (help-substitute-command-keys/preserves-text-properties): New test. (Bug#17052) 
* test/lisp/help-tests.el
(help-substitute-command-keys/preserves-text-properties): New test.
(Bug#17052)
perl-mode and cperl-mode: Recognize regex after "return" 2020-11-16T23:20:26+00:00 Harald Jörg 2020-11-16T23:20:26+00:00 f5f9e100972598b1bb9cea4c0445777db2e1131e * lisp/progmodes/cperl-mode.el (cperl-find-pods-heres): Add "return" to the keywords which start a regex. * lisp/progmodes/perl-mode.el (defconst): Add "return" to 'perl--syntax-exp-intro-keywords' (Bug#26850). * test/lisp/progmodes/cperl-mode-tests.el (cperl-test-bug-28650): New test (bug#26850). 
* lisp/progmodes/cperl-mode.el (cperl-find-pods-heres): Add
"return" to the keywords which start a regex.

* lisp/progmodes/perl-mode.el (defconst): Add "return" to
'perl--syntax-exp-intro-keywords' (Bug#26850).

* test/lisp/progmodes/cperl-mode-tests.el (cperl-test-bug-28650):
New test (bug#26850).
Don't quote lambdas in several places 2020-11-14T16:04:23+00:00 Stefan Kangas 2020-11-14T16:04:23+00:00 61dca6e92ac972b832e889fbeab9b6131fc896fa * lisp/align.el (align-highlight-rule): * lisp/bookmark.el (bookmark-maybe-sort-alist): * lisp/emacs-lisp/advice.el (ad-read-advice-name) (ad-retrieve-args-form, ad-make-hook-form, defadvice) (ad-with-originals): * lisp/foldout.el (foldout-inhibit-key-bindings): * lisp/gnus/gnus-bookmark.el (gnus-bookmark-maybe-sort-alist): * lisp/mail/rfc822.el (rfc822-addresses-1): * lisp/net/eudcb-ldap.el (eudc-ldap-cleanup-record-simple): * lisp/net/net-utils.el (network-connection-to-service): * lisp/net/socks.el (socks-build-auth-list): * lisp/org/ox-odt.el (org-odt--image-size): * lisp/pcomplete.el (pcomplete-command-completion-function) (pcomplete-default-completion-function, pcomplete-opt): * lisp/progmodes/cperl-mode.el (cperl-highlight-charclass) (cperl-tags-hier-init, cperl-tags-treeify) (cperl-next-interpolated-REx, cperl-time-fontification): * lisp/shadowfile.el (shadow-copy-files, shadow-shadows-of-1) (shadow-save-buffers-kill-emacs): * lisp/strokes.el (strokes-renormalize-to-grid): * lisp/tempo.el (tempo-insert, tempo-forward-mark) (tempo-backward-mark): * lisp/textmodes/artist.el (artist-submit-bug-report): * lisp/textmodes/ispell.el (ispell-complete-word): * lisp/url/url-auth.el (url-get-authentication): * lisp/url/url-cache.el (url-cache-create-filename-human-readable): * lisp/vcursor.el (vcursor-find-window): * test/lisp/textmodes/reftex-tests.el (reftex-parse-bibtex-entry-test): Don't quote lambdas. 
* lisp/align.el (align-highlight-rule):
* lisp/bookmark.el (bookmark-maybe-sort-alist):
* lisp/emacs-lisp/advice.el (ad-read-advice-name)
(ad-retrieve-args-form, ad-make-hook-form, defadvice)
(ad-with-originals):
* lisp/foldout.el (foldout-inhibit-key-bindings):
* lisp/gnus/gnus-bookmark.el (gnus-bookmark-maybe-sort-alist):
* lisp/mail/rfc822.el (rfc822-addresses-1):
* lisp/net/eudcb-ldap.el (eudc-ldap-cleanup-record-simple):
* lisp/net/net-utils.el (network-connection-to-service):
* lisp/net/socks.el (socks-build-auth-list):
* lisp/org/ox-odt.el (org-odt--image-size):
* lisp/pcomplete.el (pcomplete-command-completion-function)
(pcomplete-default-completion-function, pcomplete-opt):
* lisp/progmodes/cperl-mode.el (cperl-highlight-charclass)
(cperl-tags-hier-init, cperl-tags-treeify)
(cperl-next-interpolated-REx, cperl-time-fontification):
* lisp/shadowfile.el (shadow-copy-files, shadow-shadows-of-1)
(shadow-save-buffers-kill-emacs):
* lisp/strokes.el (strokes-renormalize-to-grid):
* lisp/tempo.el (tempo-insert, tempo-forward-mark)
(tempo-backward-mark):
* lisp/textmodes/artist.el (artist-submit-bug-report):
* lisp/textmodes/ispell.el (ispell-complete-word):
* lisp/url/url-auth.el (url-get-authentication):
* lisp/url/url-cache.el (url-cache-create-filename-human-readable):
* lisp/vcursor.el (vcursor-find-window):
* test/lisp/textmodes/reftex-tests.el
(reftex-parse-bibtex-entry-test): Don't quote lambdas.
Add an option to preserve ANSI sequences 2020-11-14T15:24:38+00:00 Pablo Barbáchano 2020-11-14T15:24:26+00:00 8700319109c06932cd66615c85fc21e386b40df6 * lisp/ansi-color.el Add an option to preserve the ANSI sequences * test/lisp/ansi-color-tests.el: Add tests (bug#44589). 
* lisp/ansi-color.el Add an option to preserve the ANSI sequences
* test/lisp/ansi-color-tests.el: Add tests (bug#44589).