Merge from lexical-binding branch.

* doc/lispref/eval.texi (Eval): Discourage the use of `eval'.
Document its new `lexical' argument.

* doc/lispref/variables.texi (Defining Variables): Mention the new meaning of `defvar'.
(Lexical Binding): New sub-section.

* lisp/Makefile.in (BIG_STACK_DEPTH, BIG_STACK_OPTS, BYTE_COMPILE_FLAGS):
New variables.
(compile-onefile, .el.elc, compile-calc, recompile): Use them.
(COMPILE_FIRST): Add macroexp and cconv.
* lisp/makefile.w32-in: Mirror changes in Makefile.in.

* lisp/vc/cvs-status.el:
* lisp/vc/diff-mode.el:
* lisp/vc/log-edit.el:
* lisp/vc/log-view.el:
* lisp/vc/smerge-mode.el:
* lisp/textmodes/bibtex-style.el:
* textmodes/css.el:
* lisp/startup.el:
* lisp/uniquify.el:
* lisp/minibuffer.el: 
* lisp/newcomment.el: 
* lisp/reveal.el: 
* lisp/server.el: 
* lisp/mpc.el: 
* lisp/emacs-lisp/smie.el: 
* lisp/doc-view.el: 
* lisp/dired.el: 
* lisp/abbrev.el: Use lexical binding.

* lisp/custom.el (custom-initialize-default, custom-declare-variable):
Use `defvar'.

* lisp/files.el (lexical-binding): Declare safe.

* lisp/help-fns.el (help-split-fundoc): Return nil if there's nothing else
than the arglist.
(help-add-fundoc-usage): Don't add `Not documented'.
(help-function-arglist): Handle closures, subroutines, and new
byte-code-functions.
(help-make-usage): Remove leading underscores.
(describe-function-1): Handle closures.
(describe-variable): Use special-variable-p for completion.

* lisp/simple.el (with-wrapper-hook, apply-partially): Move to subr.el.

* lisp/subr.el (apply-partially): Use new closures rather than CL.
(--dolist-tail--, --dotimes-limit--): Don't declare dynamic.
(dolist, dotimes): Use slightly different expansion for lexical code.
(functionp): Move to C.
(letrec): New macro.
(with-wrapper-hook): Use it and apply-partially instead of CL.
(eval-after-load): Preserve lexical-binding.
(save-window-excursion, with-output-to-temp-buffer): Turn them
into macros.

* lisp/emacs-lisp/advice.el (ad-arglist): Use help-function-arglist.

* lisp/emacs-lisp/autoload.el (make-autoload): Don't burp on trivial macros.

* lisp/emacs-lisp/byte-opt.el: Use lexical binding.
(byte-inline-lapcode): Remove (to bytecomp).
(byte-compile-inline-expand): Pay attention to inlining to/from
lexically bound code.
(byte-compile-unfold-lambda): Don't handle byte-code-functions
any more.
(byte-optimize-form-code-walker): Don't handle save-window-excursion
any more and don't call compiler-macros.
(byte-compile-splice-in-already-compiled-code): Remove.
(byte-code): Don't inline any more.
(disassemble-offset): Receive `bytes' as argument rather than via
dynamic scoping.
(byte-compile-tag-number): Declare before first use.
(byte-decompile-bytecode-1): Handle new byte-codes, don't change
`return' even if make-spliceable.
(byte-compile-side-effect-and-error-free-ops): Add stack-ref, remove
obsolete interactive-p.
(byte-optimize-lapcode): Optimize new lap-codes.
Don't trip up on new form of `byte-constant' lap code.

* lisp/emacs-lisp/byte-run.el (make-obsolete): Don't set the `byte-compile'
handler any more.

* lisp/emacs-lisp/bytecomp.el: Use lexical binding instead of
a "bytecomp-" prefix.  Macroexpand everything as a separate phase.
(byte-compile-initial-macro-environment):
Handle declare-function here.
(byte-compile--lexical-environment): New var.
(byte-stack-ref, byte-stack-set, byte-discardN)
(byte-discardN-preserve-tos): New lap codes.
(byte-interactive-p): Don't use any more.
(byte-compile-push-bytecodes, byte-compile-push-bytecode-const2):
New macros.
(byte-compile-lapcode): Use them and handle new lap codes.
(byte-compile-obsolete): Remove.
(byte-compile-arglist-signature): Handle new byte-code arg"lists".
(byte-compile-arglist-warn): Check late def of inlinable funs.
(byte-compile-cl-warn): Don't silence warnings for compiler-macros
since they should have been expanded by now.
(byte-compile--outbuffer): Rename from bytecomp-outbuffer.
(byte-compile-from-buffer): Remove unused second arg.
(byte-compile-preprocess): New function.
(byte-compile-toplevel-file-form): New function to distinguish
file-form calls from outside from file-form calls from hunk-handlers.
(byte-compile-file-form): Simplify.
(byte-compile-file-form-defsubst): Remove.
(byte-compile-file-form-defmumble): Simplify now that
byte-compile-lambda always returns a byte-code-function.
(byte-compile): Preprocess.
(byte-compile-byte-code-maker, byte-compile-byte-code-unmake):
Remove, not used any more.
(byte-compile-arglist-vars, byte-compile-make-lambda-lexenv)
(byte-compile-make-args-desc): New funs.
(byte-compile-lambda): Handle lexical functions.  Always return
a byte-code-function.
(byte-compile-reserved-constants): New var, to make up room for
closed-over variables.
(byte-compile-constants-vector): Obey it.
(byte-compile-top-level): New args `lexenv' and `reserved-csts'.
(byte-compile-macroexpand-declare-function): New function.
(byte-compile-form): Call byte-compile-unfold-bcf to inline immediate
byte-code-functions.
(byte-compile-form): Check obsolescence here.
(byte-compile-inline-lapcode, byte-compile-unfold-bcf): New functions.
(byte-compile-variable-ref): Remove.
(byte-compile-dynamic-variable-op): New fun.
(byte-compile-dynamic-variable-bind, byte-compile-variable-ref)
(byte-compile-variable-set): New funs.
(byte-compile-discard): Add 2 args.
(byte-compile-stack-ref, byte-compile-stack-set)
(byte-compile-make-closure, byte-compile-get-closed-var): New funs.
(byte-compile-funarg, byte-compile-funarg-2): Remove, handled in
macroexpand-all instead.
(byte-compile-quote-form): Remove.
(byte-compile-push-binding-init, byte-compile-not-lexical-var-p)
(byte-compile-bind, byte-compile-unbind): New funs.
(byte-compile-let): Handle let* and lexical binding.
(byte-compile-let*): Remove.
(byte-compile-catch, byte-compile-unwind-protect)
(byte-compile-track-mouse, byte-compile-condition-case):
Handle a new :fun-body form, used for lexical scoping.
(byte-compile-save-window-excursion)
(byte-compile-with-output-to-temp-buffer): Remove.
(byte-compile-defun): Simplify.
(byte-compile-stack-adjustment): New fun.
(byte-compile-out): Use it.
(byte-compile-refresh-preloaded): Don't reload byte-compiler files.

* lisp/emacs-lisp/cconv.el: New file.

* lisp/emacs-lisp/cl-extra.el (cl-macroexpand-all): Properly quote CL
closures.

* lisp/emacs-lisp/cl-macs.el (cl-byte-compile-block)
(cl-byte-compile-throw): Remove.
(cl-block-wrapper, cl-block-throw): Use compiler-macros instead.

* lisp/emacs-lisp/cl.el (pushnew): Silence warning.

* lisp/emacs-lisp/disass.el (disassemble-internal): Handle new
`closure' objects.
(disassemble-1): Handle new byte codes.

* lisp/emacs-lisp/edebug.el (edebug-eval-defun)
(edebug-eval-top-level-form): Use eval-sexp-add-defvars.
(edebug-toggle): Avoid `eval'.

* lisp/emacs-lisp/eieio-comp.el: Remove.

* lisp/emacs-lisp/eieio.el (byte-compile-file-form-defmethod):
Don't autoload.
(eieio-defgeneric-form-primary-only-one): Use `byte-compile' rather
than the internal `byte-compile-lambda'.
(defmethod): Don't hide code under quotes.
(eieio-defmethod): New `code' argument.

* lisp/emacs-lisp/float-sup.el (pi): Don't declare as dynamically bound.

* lisp/emacs-lisp/lisp-mode.el (eval-last-sexp-1):
Use eval-sexp-add-defvars.
(eval-sexp-add-defvars): New fun.

* lisp/emacs-lisp/macroexp.el: Use lexical binding.
(macroexpand-all-1): Check obsolete macros.  Expand compiler-macros.
Don't convert ' to #' without checking that it's indeed quoting
a lambda.

* lisp/emacs-lisp/pcase.el: Don't use destructuring-bind.
(pcase--memoize): Rename from pcase-memoize.  Change weakness.
(pcase): Add `let' pattern.
Change memoization so it actually works.
(pcase-mutually-exclusive-predicates): Add byte-code-function-p.
(pcase--u1) <guard, pred>: Fix possible shadowing problem.
<let>: New case.

* src/alloc.c (Fmake_symbol): Init new `declared_special' field.

* src/buffer.c (defvar_per_buffer): Set new `declared_special' field.

* src/bytecode.c (Bstack_ref, Bstack_set, Bstack_set2, BdiscardN):
New byte-codes.
(exec_byte_code): New function extracted from Fbyte_code to handle new
calling convention for byte-code-functions.  Add new byte-codes.

* src/callint.c (Fcall_interactively): Preserve lexical-binding mode for
interactive spec.

* src/doc.c (Fdocumentation, store_function_docstring):
* src/data.c (Finteractive_form): Handle closures.

* src/eval.c (Fsetq): Handle lexical vars.
(Fdefun, Fdefmacro, Ffunction): Make closures when needed.
(Fdefconst, Fdefvaralias, Fdefvar): Mark as dynamic.
(FletX, Flet): Obey lexical binding.
(Fcommandp): Handle closures.
(Feval): New `lexical' arg.
(eval_sub): New function extracted from Feval.  Use it almost
everywhere where Feval was used.  Look up vars in lexical env.
Handle closures.
(Ffunctionp): Move from subr.el.
(Ffuncall): Handle closures.
(apply_lambda): Remove `eval_flags'.
(funcall_lambda): Handle closures and new byte-code-functions.
(Fspecial_variable_p): New function.
(syms_of_eval): Initialize the Vinternal_interpreter_environment var,
but without exporting it to Lisp.

* src/fns.c (concat, mapcar1): Accept byte-code-functions.

* src/image.c (parse_image_spec): Use Ffunctionp.

* src/keyboard.c (eval_dyn): New fun.
(menu_item_eval_property): Use it.

* src/lisp.h (struct Lisp_Symbol): New field `declared_special'.

* src/lread.c (lisp_file_lexically_bound_p): New function.
(Fload): Bind Qlexical_binding.
(readevalloop): Remove `evalfun' arg.
Bind Qinternal_interpreter_environment.
(Feval_buffer): Bind Qlexical_binding.
(defvar_int, defvar_bool, defvar_lisp_nopro, defvar_kboard):
Mark as dynamic.
(syms_of_lread): Declare `lexical-binding'.

* src/window.c (Ftemp_output_buffer_show): New fun.
(Fsave_window_excursion):
* src/print.c (Fwith_output_to_temp_buffer): Move to subr.el.

3 files changed, 136 insertions, 33 deletions

diff --git a/doc/lispref/ChangeLog b/doc/lispref/ChangeLog
index 1eb3cfa2556..faa5fa44e46 100644
--- a/doc/lispref/ChangeLog
+++ b/doc/lispref/ChangeLog
@@ -1,6 +1,14 @@
+2011-04-01  Stefan Monnier  <monnier@iro.umontreal.ca>
+
+        * variables.texi (Defining Variables): Mention the new meaning of `defvar'.
+        (Lexical Binding): New sub-section.
+
+        * eval.texi (Eval): Discourage the use of `eval'.
+        Document its new `lexical' argument.
+
 2011-03-28  Stefan Monnier  <monnier@iro.umontreal.ca>
 
-        * commands.texi (Command Overview): post-command-hook is not reset to
+        * commands.texi (Command Overview): `post-command-hook' is not reset to
         nil any more.
 
 2011-03-19  Stefan Monnier  <monnier@iro.umontreal.ca>
diff --git a/doc/lispref/eval.texi b/doc/lispref/eval.texi
index d44fe5bb95b..74f3d9c48b9 100644
--- a/doc/lispref/eval.texi
+++ b/doc/lispref/eval.texi
@@ -585,6 +585,11 @@ occurrence in a program being run.  On rare occasions, you may need to
 write code that evaluates a form that is computed at run time, such as
 after reading a form from text being edited or getting one from a
 property list.  On these occasions, use the @code{eval} function.
+Often @code{eval} is not needed and something else should be used instead.
+For example, to get the value of a variable, while @code{eval} works,
+@code{symbol-value} is preferable; or rather than store expressions
+in a property list that then need to go through @code{eval}, it is better to
+store functions instead that are then passed to @code{funcall}.
 
   The functions and variables described in this section evaluate forms,
 specify limits to the evaluation process, or record recently returned
@@ -596,10 +601,13 @@ to store an expression in the data structure and evaluate it.  Using
 functions provides the ability to pass information to them as
 arguments.
 
-@defun eval form
+@defun eval form &optional lexical
 This is the basic function evaluating an expression.  It evaluates
 @var{form} in the current environment and returns the result.  How the
 evaluation proceeds depends on the type of the object (@pxref{Forms}).
+@var{lexical} if non-nil means to evaluate @var{form} using lexical scoping
+rules (@pxref{Lexical Binding}) instead of the default dynamic scoping used
+historically in Emacs Lisp.
 
 Since @code{eval} is a function, the argument expression that appears
 in a call to @code{eval} is evaluated twice: once as preparation before
diff --git a/doc/lispref/variables.texi b/doc/lispref/variables.texi
index a68b2b6dd4e..7e2c32334a4 100644
--- a/doc/lispref/variables.texi
+++ b/doc/lispref/variables.texi
@@ -25,22 +25,22 @@ textual Lisp program is written using the read syntax for the symbol
 representing the variable.
 
 @menu
-* Global Variables::      Variable values that exist permanently, everywhere.
-* Constant Variables::    Certain "variables" have values that never change.
-* Local Variables::       Variable values that exist only temporarily.
-* Void Variables::        Symbols that lack values.
-* Defining Variables::    A definition says a symbol is used as a variable.
-* Tips for Defining::     Things you should think about when you
+* Global Variables::            Variable values that exist permanently, everywhere.
+* Constant Variables::          Certain "variables" have values that never change.
+* Local Variables::             Variable values that exist only temporarily.
+* Void Variables::              Symbols that lack values.
+* Defining Variables::          A definition says a symbol is used as a variable.
+* Tips for Defining::           Things you should think about when you
                             define a variable.
-* Accessing Variables::   Examining values of variables whose names
+* Accessing Variables::         Examining values of variables whose names
                             are known only at run time.
-* Setting Variables::     Storing new values in variables.
-* Variable Scoping::      How Lisp chooses among local and global values.
-* Buffer-Local Variables::  Variable values in effect only in one buffer.
-* File Local Variables::  Handling local variable lists in files.
-* Directory Local Variables:: Local variables common to all files in a directory.
-* Frame-Local Variables:: Frame-local bindings for variables.
-* Variable Aliases::      Variables that are aliases for other variables.
+* Setting Variables::           Storing new values in variables.
+* Variable Scoping::            How Lisp chooses among local and global values.
+* Buffer-Local Variables::      Variable values in effect only in one buffer.
+* File Local Variables::        Handling local variable lists in files.
+* Directory Local Variables::   Local variables common to all files in a directory.
+* Frame-Local Variables::       Frame-local bindings for variables.
+* Variable Aliases::            Variables that are aliases for other variables.
 * Variables with Restricted Values::  Non-constant variables whose value can
                                         @emph{not} be an arbitrary Lisp object.
 @end menu
@@ -437,14 +437,18 @@ this reason, user options must be defined with @code{defvar}.
 This special form defines @var{symbol} as a variable and can also
 initialize and document it.  The definition informs a person reading
 your code that @var{symbol} is used as a variable that might be set or
-changed.  Note that @var{symbol} is not evaluated; the symbol to be
-defined must appear explicitly in the @code{defvar}.
+changed.  It also declares this variable as @dfn{special}, meaning that it
+should always use dynamic scoping rules.  Note that @var{symbol} is not
+evaluated; the symbol to be defined must appear explicitly in the
+@code{defvar}.
 
 If @var{symbol} is void and @var{value} is specified, @code{defvar}
 evaluates it and sets @var{symbol} to the result.  But if @var{symbol}
 already has a value (i.e., it is not void), @var{value} is not even
-evaluated, and @var{symbol}'s value remains unchanged.  If @var{value}
-is omitted, the value of @var{symbol} is not changed in any case.
+evaluated, and @var{symbol}'s value remains unchanged.
+If @var{value} is omitted, the value of @var{symbol} is not changed in any
+case; instead, the only effect of @code{defvar} is to declare locally that this
+variable exists elsewhere and should hence always use dynamic scoping rules.
 
 If @var{symbol} has a buffer-local binding in the current buffer,
 @code{defvar} operates on the default value, which is buffer-independent,
@@ -881,7 +885,7 @@ the others.
 @cindex extent
 @cindex dynamic scoping
 @cindex lexical scoping
-  Local bindings in Emacs Lisp have @dfn{indefinite scope} and
+  By default, local bindings in Emacs Lisp have @dfn{indefinite scope} and
 @dfn{dynamic extent}.  @dfn{Scope} refers to @emph{where} textually in
 the source code the binding can be accessed.  ``Indefinite scope'' means
 that any part of the program can potentially access the variable
@@ -893,6 +897,8 @@ lasts as long as the activation of the construct that established it.
 @dfn{dynamic scoping}.  By contrast, most programming languages use
 @dfn{lexical scoping}, in which references to a local variable must be
 located textually within the function or block that binds the variable.
+Emacs can also support lexical scoping, upon request (@pxref{Lexical
+Binding}).
 
 @cindex CL note---special variables
 @quotation
@@ -901,11 +907,12 @@ dynamically scoped, like all variables in Emacs Lisp.
 @end quotation
 
 @menu
-* Scope::          Scope means where in the program a value is visible.
+* Scope::                       Scope means where in the program a value is visible.
                      Comparison with other languages.
-* Extent::         Extent means how long in time a value exists.
-* Impl of Scope::  Two ways to implement dynamic scoping.
-* Using Scoping::  How to use dynamic scoping carefully and avoid problems.
+* Extent::                      Extent means how long in time a value exists.
+* Impl of Scope::               Two ways to implement dynamic scoping.
+* Using Scoping::               How to use dynamic scoping carefully and avoid problems.
+* Lexical Binding::             Use of lexical scoping.
 @end menu
 
 @node Scope
@@ -969,12 +976,12 @@ Here, when @code{foo} is called by @code{binder}, it binds @code{x}.
 by @code{foo} instead of the one bound by @code{binder}.
 @end itemize
 
-Emacs Lisp uses dynamic scoping because simple implementations of
+Emacs Lisp used dynamic scoping by default because simple implementations of
 lexical scoping are slow.  In addition, every Lisp system needs to offer
-dynamic scoping at least as an option; if lexical scoping is the norm,
-there must be a way to specify dynamic scoping instead for a particular
-variable.  It might not be a bad thing for Emacs to offer both, but
-implementing it with dynamic scoping only was much easier.
+dynamic scoping at least as an option; if lexical scoping is the norm, there
+must be a way to specify dynamic scoping instead for a particular variable.
+Nowadays, Emacs offers both, but the default is still to use exclusively
+dynamic scoping.
 
 @node Extent
 @subsection Extent
@@ -1088,6 +1095,86 @@ for inter-function usage.  It also avoids a warning from the byte
 compiler.  Choose the variable's name to avoid name conflicts---don't
 use short names like @code{x}.
 
+
+@node Lexical Binding
+@subsection Use of Lexical Scoping
+
+Emacs Lisp can be evaluated in two different modes: in dynamic binding mode or
+lexical binding mode.  In dynamic binding mode, all local variables use dynamic
+scoping, whereas in lexical binding mode variables that have been declared
+@dfn{special} (i.e., declared with @code{defvar} or @code{defconst}) use
+dynamic scoping and all others use lexical scoping.
+
+@defvar lexical-binding
+When non-nil, evaluation of Lisp code uses lexical scoping for non-special
+local variables instead of dynamic scoping.  If nil, dynamic scoping is used
+for all local variables.  This variable is typically set for a whole Elisp file
+via file local variables (@pxref{File Local Variables}).
+@end defvar
+
+@defun special-variable-p SYMBOL
+Return whether SYMBOL has been declared as a special variable, via
+@code{defvar} or @code{defconst}.
+@end defun
+
+The use of a special variable as a formal argument in a function is generally
+discouraged and its behavior in lexical binding mode is unspecified (it may use
+lexical scoping sometimes and dynamic scoping other times).
+
+Functions like @code{symbol-value}, @code{boundp}, or @code{set} only know
+about dynamically scoped variables, so you cannot get the value of a lexical
+variable via @code{symbol-value} and neither can you change it via @code{set}.
+Another particularity is that code in the body of a @code{defun} or
+@code{defmacro} cannot refer to surrounding lexical variables.
+
+Evaluation of a @code{lambda} expression in lexical binding mode will not just
+return that lambda expression unchanged, as in the dynamic binding case, but
+will instead construct a new object that remembers the current lexical
+environment in which that lambda expression was defined, so that the function
+body can later be evaluated in the proper context.  Those objects are called
+@dfn{closures}.  They are also functions, in the sense that they are accepted
+by @code{funcall}, and they are represented by a cons cell whose @code{car} is
+the symbol @code{closure}.
+
+@menu
+* Converting to Lexical Binding:: How to start using lexical scoping
+@end menu
+
+@node Converting to Lexical Binding
+@subsubsection Converting a package to use lexical scoping
+
+Lexical scoping, as currently implemented, does not bring many significant
+benefits, unless you are a seasoned functional programmer addicted to
+higher-order functions.  But its importance will increase in the future:
+lexical scoping opens up a lot more opportunities for optimization, so
+lexically scoped code is likely to run faster in future Emacs versions, and it
+is much more friendly to concurrency, which we want to add in the near future.
+
+Converting a package to lexical binding is usually pretty easy and should not
+break backward compatibility: just add a file-local variable setting
+@code{lexical-binding} to @code{t} and add declarations of the form
+@code{(defvar @var{VAR})} for every variable which still needs to use
+dynamic scoping.
+
+To find which variables need this declaration, the simplest solution is to
+check the byte-compiler's warnings.  The byte-compiler will usually find those
+variables either because they are used outside of a let-binding (leading to
+warnings about reference or assignment to ``free variable @var{VAR}'') or
+because they are let-bound but not used within the let-binding (leading to
+warnings about ``unused lexical variable @var{VAR}'').
+
+In cases where a dynamically scoped variable was bound as a function argument,
+you will also need to move this binding to a @code{let}.  These cases are also
+flagged by the byte-compiler.
+
+To silence byte-compiler warnings about unused variables, just use a variable
+name that start with an underscore, which the byte-compiler interpret as an
+indication that this is a variable known not to be used.
+
+In most cases, the resulting code will then work with either setting of
+@code{lexical-binding}, so it can still be used with older Emacsen (which will
+simply ignore the @code{lexical-binding} variable setting).
+
 @node Buffer-Local Variables
 @section Buffer-Local Variables
 @cindex variable, buffer-local
@@ -1103,9 +1190,9 @@ local to each terminal, or to each frame.  @xref{Multiple Terminals},
 and @xref{Frame-Local Variables}.)
 
 @menu
-* Intro to Buffer-Local::      Introduction and concepts.
-* Creating Buffer-Local::      Creating and destroying buffer-local bindings.
-* Default Value::              The default value is seen in buffers
+* Intro to Buffer-Local::       Introduction and concepts.
+* Creating Buffer-Local::       Creating and destroying buffer-local bindings.
+* Default Value::               The default value is seen in buffers
                                  that don't have their own buffer-local values.
 @end menu