More work for named queues

- Rename `WORKERS` setting to `CONCURRENT_JOBS` for more clarity. Added this
  setting to the command line options, the ini .conf & default settings conf.py
  files.
- Added support for JSON style arrays in INI config.
- Added support for weighted named queues. The style for the setting is
  [[weight, "name"], [weight, "name"]]. Configured in both the INI and command
  line for job manager. Added documentation.
- Updated the spec for the INFORM message. Weights are sent with the queue
  names. If there are no weights specified they will be given the default value
  of 0.
- updated Router.queues to a list from a deque so that it can be sorted
  by priority more easily.

3 files changed, 36 insertions, 18 deletions

diff --git a/docs/conf.py b/docs/conf.py
index 864bbcf..aa23956 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -16,6 +16,7 @@ import sys
 import os
 import shlex
 
+import eventmq   # for __version__
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
@@ -52,15 +53,15 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'EventMQ'
-copyright = u'2015, eventboard.io'
-author = u'eventboard.io'
+copyright = u'2016, eventboard.io'
+author = u'EventMQ Contributors'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = '0'
+version = eventmq.__version__
 # The full version, including alpha/beta/rc tags.
 release = '0'
 
diff --git a/docs/protocol.rst b/docs/protocol.rst
index 4c2aff0..c341403 100644
--- a/docs/protocol.rst
+++ b/docs/protocol.rst
@@ -71,7 +71,7 @@ FRAME  Value          Description
 1      eMQP/1.0       Protocol version
 2      REQUEST        command
 3      _MSGID_        A unique id for the msg
-4      _QUEUE_NAME_   the name of the queue the worker belongs to
+4      _QUEUE_NAME_   the name of the queue the request should be sent to
 5      _HEADERS_      dictionary of headers. can be an empty set
 6      _MSG_          The message to send
 ====== ============== ===========
@@ -85,7 +85,7 @@ FRAME  Value          Description
 1      eMQP/1.0       Protocol version
 2      PUBLISH        command
 3      _MSGID_        A unique id for the msg
-4      _TOPIC_NAME_   the name of the queue the worker belongs to
+4      _TOPIC_NAME_   the name of the topic this message should be published across
 5      _HEADERS_      csv list of headers
 6      _MSG_          The message to send
 ====== ============== ===========
@@ -99,7 +99,7 @@ FRAME   Value         Description
 1      eMQP/1.0       Protocol version
 2      SCHEDULE       command
 3      _MSGID_        A unique id for the msg
-4      _TOPIC_NAME_   name of queue that the job should run in
+4      _QUEUE_NAME_   name of queue that the job should run in
 5      _HEADERS_      csv list of headers for this message
 6      _MSG_          The message to send
 ====== ============== ===========
@@ -113,7 +113,7 @@ FRAME   Value         Description
 1      eMQP/1.0       Protocol version
 2      UNSCHEDULE     command
 3      _MSGID_        A unique id for the msg
-4      _TOPIC_NAME_   ignored for this command, broadcasted to all queues
+4      _QUEUE_NAME_   ignored for this command, broadcasted to all queues
 5      _HEADERS_      csv list of headers for this message
 6      _MSG_          The message to send
 ====== ============== ===========
@@ -129,7 +129,7 @@ FRAME   Value         Description
 1      eMQP/1.0       Protocol version
 2      INFORM         command
 3      _MSGID_        A unique id for the msg
-4      _QUEUE_NAME_   csv seperated names of queue the worker belongs to
+4                     Queues. Unused for scheduler
 5      scheduler      type of peer connecting
 ====== ============== ===========
 
@@ -144,7 +144,7 @@ FRAME   Value         Description
 1      eMQP/1.0       Protocol version
 2      INFORM         command
 3      _MSGID_        A unique id for the msg
-4      _QUEUE_NAME_   csv seperated names of queue the worker belongs to.
+4      _QUEUES_       csv seperated arrays containing an int and a string for weight and name. e.g. [40, 'email']
 5      worker         type of peer connecting
 ====== ============== ===========
 
@@ -203,7 +203,6 @@ Heartbeating
  * If the worker detects that the broker disconnected it SHOULD restart the conversation.
  * If the broker detects that a worker has disconnected it should stop sending it a message of any type.
  * If the scheduler detects that the broker disconnects it SHOULD restart the conversation.
- * If the broker detects that a scheduler has disconnected it should ??????????.
 
 REQUEST Headers
 ---------------
diff --git a/docs/settings_file.rst b/docs/settings_file.rst
index 07d2338..b0bc36c 100644
--- a/docs/settings_file.rst
+++ b/docs/settings_file.rst
@@ -15,16 +15,34 @@ Scheduler
 Job Manager
 ***********
 
+concurrent_jobs
+===============
+Default: 4
+
+This is the number of concurrent jobs the indiviudal job manager should execute
+at a time. If you are using the multiprocess or threading model this number
+becomes important as you will want to control the load on your server. If the
+load equals the number of cores on the server, processes will begin waiting for
+cpu cycles and things will begin to slow down.
+
+A safe number to choose if your jobs block a lot would be (2 * cores). If your
+jobs are cpu intensive you will want to set this number to the number of cores
+you have or (cores - 1) to leave cycles for the os and other processes. This is
+something that will have to be tuned based on the jobs that are
+running. Grouping similar jobs in named queues will help you tune this number.
+
 queues
 ======
-Default: default
-
-Comma seperated list of queues to process jobs for. Example:
-``queues=high,med,low,default``. The philosophy taken for this list is each job
-manager should have a single primary queue. This queue is the first in the list
-(in the case of the example ``high`` is the primary queue). Subsequent queues
-are queues that this job manager should help out with should jobs be backed up,
-and there are no primary queue jobs to take care of.
+Default: (10, default)
+
+Semi-colon seperated list of queues to process jobs for with thier
+weights. Example: ``queues=(10, data_process); (15, email)``.  With these
+weights and the ``CONCURRENT_JOBS`` setting, you should be able to tune managers
+running jobs locally pretty efficiently. If you have a larger box with a weight
+of 50 on q1 and 8 concurrent jobs and a smaller box with a weight 30 and 4
+concurrent jobs, the q1 jobs will be sent to the large box until it is no longer
+accepting jobs. At this point jobs will start to be sent to the next highest
+number until the large box is ready to accept another q1 job.
 
 .. note::