Variadic C functions now count arguments with ptrdiff_t.

This partly undoes my 2011-03-30 change, which replaced int with size_t.
Back then I didn't know that the Emacs coding style prefers signed int.
Also, in the meantime I found a few more instances where arguments
were being counted with int, which may truncate counts on 64-bit
machines, or EMACS_INT, which may be unnecessarily wide.
* lisp.h (struct Lisp_Subr.function.aMANY)
(DEFUN_ARGS_MANY, internal_condition_case_n, safe_call):
Arg counts are now ptrdiff_t, not size_t.
All variadic functions and their callers changed accordingly.
(struct gcpro.nvars): Now size_t, not size_t.  All uses changed.
* bytecode.c (exec_byte_code): Check maxdepth for overflow,
to avoid potential buffer overrun.  Don't assume arg counts fit in 'int'.
* callint.c (Fcall_interactively): Check arg count for overflow,
to avoid potential buffer overrun.  Use signed char, not 'int',
for 'varies' array, so that we needn't bother to check its size
calculation for overflow.
* editfns.c (Fformat): Use ptrdiff_t, not EMACS_INT, to count args.
* eval.c (apply_lambda):
* fns.c (Fmapconcat): Use XFASTINT, not XINT, to get args length.
(struct textprop_rec.argnum): Now ptrdiff_t, not int.  All uses changed.
(mapconcat): Use ptrdiff_t, not int and EMACS_INT, to count args.

1 files changed, 8 insertions, 4 deletions

diff --git a/src/callint.c b/src/callint.c
index 29bb7ccc409..dc5e6a4c37a 100644
--- a/src/callint.c
+++ b/src/callint.c
@@ -269,10 +269,9 @@ invoke it.  If KEYS is omitted or nil, the return value of
   /* If varies[i] > 0, the i'th argument shouldn't just have its value
      in this call quoted in the command history.  It should be
      recorded as a call to the function named callint_argfuns[varies[i]].  */
-  int *varies;
+  signed char *varies;
 
-  register size_t i;
-  size_t nargs;
+  ptrdiff_t i, nargs;
   int foo;
   char prompt1[100];
   char *tem1;
@@ -465,9 +464,14 @@ invoke it.  If KEYS is omitted or nil, the return value of
         break;
     }
 
+  if (min (MOST_POSITIVE_FIXNUM,
+           min (PTRDIFF_MAX, SIZE_MAX) / sizeof (Lisp_Object))
+      < nargs)
+    memory_full (SIZE_MAX);
+
   args = (Lisp_Object *) alloca (nargs * sizeof (Lisp_Object));
   visargs = (Lisp_Object *) alloca (nargs * sizeof (Lisp_Object));
-  varies = (int *) alloca (nargs * sizeof (int));
+  varies = (signed char *) alloca (nargs);
 
   for (i = 0; i < nargs; i++)
     {