aboutsummaryrefslogtreecommitdiffstats
path: root/router.html
diff options
context:
space:
mode:
Diffstat (limited to 'router.html')
-rw-r--r--router.html125
1 files changed, 117 insertions, 8 deletions
diff --git a/router.html b/router.html
index 1cacb79..0604f89 100644
--- a/router.html
+++ b/router.html
@@ -84,6 +84,7 @@
84 84
85 <ul class="current"> 85 <ul class="current">
86<li class="toctree-l1 current"><a class="reference internal" href="api.html">API Documentation</a><ul class="current"> 86<li class="toctree-l1 current"><a class="reference internal" href="api.html">API Documentation</a><ul class="current">
87<li class="toctree-l2"><a class="reference internal" href="client.html"><code class="docutils literal"><span class="pre">client</span></code> &#8211; Client Utilities</a></li>
87<li class="toctree-l2"><a class="reference internal" href="exceptions.html"><code class="docutils literal"><span class="pre">exceptions</span></code> &#8211; Exceptions</a></li> 88<li class="toctree-l2"><a class="reference internal" href="exceptions.html"><code class="docutils literal"><span class="pre">exceptions</span></code> &#8211; Exceptions</a></li>
88<li class="toctree-l2"><a class="reference internal" href="jobmanager.html"><code class="docutils literal"><span class="pre">jobmanager</span></code> &#8211; Job Manager</a></li> 89<li class="toctree-l2"><a class="reference internal" href="jobmanager.html"><code class="docutils literal"><span class="pre">jobmanager</span></code> &#8211; Job Manager</a></li>
89<li class="toctree-l2"><a class="reference internal" href="poller.html"><code class="docutils literal"><span class="pre">poller</span></code> &#8211; Poller</a></li> 90<li class="toctree-l2"><a class="reference internal" href="poller.html"><code class="docutils literal"><span class="pre">poller</span></code> &#8211; Poller</a></li>
@@ -148,6 +149,20 @@
148<em class="property">class </em><code class="descclassname">eventmq.router.</code><code class="descname">Router</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router" title="Permalink to this definition">¶</a></dt> 149<em class="property">class </em><code class="descclassname">eventmq.router.</code><code class="descname">Router</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router" title="Permalink to this definition">¶</a></dt>
149<dd><p>A simple router of messages</p> 150<dd><p>A simple router of messages</p>
150<dl class="method"> 151<dl class="method">
152<dt id="eventmq.router.Router.add_scheduler">
153<code class="descname">add_scheduler</code><span class="sig-paren">(</span><em>scheduler_id</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.add_scheduler" title="Permalink to this definition">¶</a></dt>
154<dd><p>Adds a scheduler to the queue to receive SCHEDULE commands</p>
155<table class="docutils field-list" frame="void" rules="none">
156<col class="field-name" />
157<col class="field-body" />
158<tbody valign="top">
159<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>scheduler_id</strong> (<a class="reference external" href="https://docs.python.org/library/functions.html#str" title="(in Python v2.7)"><em>str</em></a>) &#8211; unique id of the scheduler to add</td>
160</tr>
161</tbody>
162</table>
163</dd></dl>
164
165<dl class="method">
151<dt id="eventmq.router.Router.add_worker"> 166<dt id="eventmq.router.Router.add_worker">
152<code class="descname">add_worker</code><span class="sig-paren">(</span><em>worker_id</em>, <em>queues=None</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.add_worker" title="Permalink to this definition">¶</a></dt> 167<code class="descname">add_worker</code><span class="sig-paren">(</span><em>worker_id</em>, <em>queues=None</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.add_worker" title="Permalink to this definition">¶</a></dt>
153<dd><p>Adds a worker to worker queues</p> 168<dd><p>Adds a worker to worker queues</p>
@@ -156,7 +171,7 @@
156<col class="field-body" /> 171<col class="field-body" />
157<tbody valign="top"> 172<tbody valign="top">
158<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> 173<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
159<li><strong>worker_id</strong> &#8211; unique id of the worker to add</li> 174<li><strong>worker_id</strong> (<a class="reference external" href="https://docs.python.org/library/functions.html#str" title="(in Python v2.7)"><em>str</em></a>) &#8211; unique id of the worker to add</li>
160<li><strong>queues</strong> &#8211; queue or queues this worker should be a member of</li> 175<li><strong>queues</strong> &#8211; queue or queues this worker should be a member of</li>
161</ul> 176</ul>
162</td> 177</td>
@@ -166,6 +181,13 @@
166</dd></dl> 181</dd></dl>
167 182
168<dl class="method"> 183<dl class="method">
184<dt id="eventmq.router.Router.clean_up_dead_schedulers">
185<code class="descname">clean_up_dead_schedulers</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.clean_up_dead_schedulers" title="Permalink to this definition">¶</a></dt>
186<dd><p>Loops through the list of schedulers and remove any schedulers who
187the router hasn&#8217;t received a heartbeat in HEARTBEAT_TIMEOUT</p>
188</dd></dl>
189
190<dl class="method">
169<dt id="eventmq.router.Router.clean_up_dead_workers"> 191<dt id="eventmq.router.Router.clean_up_dead_workers">
170<code class="descname">clean_up_dead_workers</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.clean_up_dead_workers" title="Permalink to this definition">¶</a></dt> 192<code class="descname">clean_up_dead_workers</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.clean_up_dead_workers" title="Permalink to this definition">¶</a></dt>
171<dd><p>Loops through the worker queues and removes any workers who haven&#8217;t 193<dd><p>Loops through the worker queues and removes any workers who haven&#8217;t
@@ -209,10 +231,7 @@ and announces itself.</p>
209<dl class="method"> 231<dl class="method">
210<dt id="eventmq.router.Router.on_receive_request"> 232<dt id="eventmq.router.Router.on_receive_request">
211<code class="descname">on_receive_request</code><span class="sig-paren">(</span><em>msg</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.on_receive_request" title="Permalink to this definition">¶</a></dt> 233<code class="descname">on_receive_request</code><span class="sig-paren">(</span><em>msg</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.on_receive_request" title="Permalink to this definition">¶</a></dt>
212<dd><p>This function is called when a message comes in from the client socket. 234<dd><table class="docutils field-list" frame="void" rules="none">
213It then calls <cite>on_command</cite>. If <cite>on_command</cite> isn&#8217;t found, then a
214warning is created.</p>
215<table class="docutils field-list" frame="void" rules="none">
216<col class="field-name" /> 235<col class="field-name" />
217<col class="field-body" /> 236<col class="field-body" />
218<tbody valign="top"> 237<tbody valign="top">
@@ -239,9 +258,70 @@ a warning is created.</p>
239</dd></dl> 258</dd></dl>
240 259
241<dl class="method"> 260<dl class="method">
261<dt id="eventmq.router.Router.queue_message">
262<code class="descname">queue_message</code><span class="sig-paren">(</span><em>msg</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.queue_message" title="Permalink to this definition">¶</a></dt>
263<dd><p>Add a message to the queue for processing later</p>
264</dd></dl>
265
266<dl class="attribute">
267<dt id="eventmq.router.Router.queues">
268<code class="descname">queues</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.queues" title="Permalink to this definition">¶</a></dt>
269<dd><p>JobManager address by queue name. The lists here are Last Recently
270Used queues where a worker is popped off when given a job, and
271appeneded when one finishes. There is one entry per available
272worker slot, so you may see duplicate addresses.</p>
273<p class="rubric">Example</p>
274<p>{&#8216;default&#8217;: [&#8216;w1&#8217;, &#8216;w2&#8217;, &#8216;w1&#8217;, &#8216;w4&#8217;]}</p>
275</dd></dl>
276
277<dl class="attribute">
278<dt id="eventmq.router.Router.received_disconnect">
279<code class="descname">received_disconnect</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.received_disconnect" title="Permalink to this definition">¶</a></dt>
280<dd><p>Set to True when the router should die.</p>
281</dd></dl>
282
283<dl class="method">
242<dt id="eventmq.router.Router.requeue_worker"> 284<dt id="eventmq.router.Router.requeue_worker">
243<code class="descname">requeue_worker</code><span class="sig-paren">(</span><em>worker_id</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.requeue_worker" title="Permalink to this definition">¶</a></dt> 285<code class="descname">requeue_worker</code><span class="sig-paren">(</span><em>worker_id</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.requeue_worker" title="Permalink to this definition">¶</a></dt>
244<dd><p>Add a worker back to the queue pool</p> 286<dd><p>Add a worker back to the pools for which it is a member of.</p>
287<div class="admonition note">
288<p class="first admonition-title">Note</p>
289<p class="last">This will (correctly) add duplicate items into the queues.</p>
290</div>
291</dd></dl>
292
293<dl class="method">
294<dt id="eventmq.router.Router.reset_heartbeat_counters">
295<code class="descname">reset_heartbeat_counters</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.reset_heartbeat_counters" title="Permalink to this definition">¶</a></dt>
296<dd><p>Reset all the counters for heartbeats back to 0</p>
297</dd></dl>
298
299<dl class="method">
300<dt id="eventmq.router.Router.router_main">
301<code class="descname">router_main</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.router_main" title="Permalink to this definition">¶</a></dt>
302<dd><p>Kick off router with logging and settings import</p>
303</dd></dl>
304
305<dl class="attribute">
306<dt id="eventmq.router.Router.scheduler_queue">
307<code class="descname">scheduler_queue</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.scheduler_queue" title="Permalink to this definition">¶</a></dt>
308<dd><p><em>Queue for schedulers to use</em></p>
309</dd></dl>
310
311<dl class="attribute">
312<dt id="eventmq.router.Router.schedulers">
313<code class="descname">schedulers</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.schedulers" title="Permalink to this definition">¶</a></dt>
314<dd><p>Scheduler clients. Clients are able to send SCHEDULE commands that
315need to be routed to a scheduler, which will keep track of time and
316run the job.
317Contains dictionaries:</p>
318<blockquote>
319<div><dl class="docutils">
320<dt>self.schedulers[&lt;scheduler_zmq_id&gt;] = {</dt>
321<dd>&#8216;hb&#8217;: &lt;last_recv_heartbeat&gt;,</dd>
322</dl>
323<p>}</p>
324</div></blockquote>
245</dd></dl> 325</dd></dl>
246 326
247<dl class="method"> 327<dl class="method">
@@ -283,14 +363,20 @@ needed when building messages</p>
283</dd></dl> 363</dd></dl>
284 364
285<dl class="method"> 365<dl class="method">
366<dt id="eventmq.router.Router.send_schedulers_heartbeats">
367<code class="descname">send_schedulers_heartbeats</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.send_schedulers_heartbeats" title="Permalink to this definition">¶</a></dt>
368<dd><p>Send HEARTBEATs to all registered schedulers</p>
369</dd></dl>
370
371<dl class="method">
286<dt id="eventmq.router.Router.send_workers_heartbeats"> 372<dt id="eventmq.router.Router.send_workers_heartbeats">
287<code class="descname">send_workers_heartbeats</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.send_workers_heartbeats" title="Permalink to this definition">¶</a></dt> 373<code class="descname">send_workers_heartbeats</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.send_workers_heartbeats" title="Permalink to this definition">¶</a></dt>
288<dd><p>Send heartbeats to all registered workers.</p> 374<dd><p>Send HEARTBEATs to all registered workers.</p>
289</dd></dl> 375</dd></dl>
290 376
291<dl class="method"> 377<dl class="method">
292<dt id="eventmq.router.Router.start"> 378<dt id="eventmq.router.Router.start">
293<code class="descname">start</code><span class="sig-paren">(</span><em>frontend_addr='tcp://127.0.0.1:47290'</em>, <em>backend_addr='tcp://127.0.0.1:47291'</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.start" title="Permalink to this definition">¶</a></dt> 379<code class="descname">start</code><span class="sig-paren">(</span><em>frontend_addr='tcp://127.0.0.1:47291'</em>, <em>backend_addr='tcp://127.0.0.1:47290'</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.start" title="Permalink to this definition">¶</a></dt>
294<dd><p>Begin listening for connections on the provided connection strings</p> 380<dd><p>Begin listening for connections on the provided connection strings</p>
295<table class="docutils field-list" frame="void" rules="none"> 381<table class="docutils field-list" frame="void" rules="none">
296<col class="field-name" /> 382<col class="field-name" />
@@ -306,6 +392,29 @@ needed when building messages</p>
306</table> 392</table>
307</dd></dl> 393</dd></dl>
308 394
395<dl class="attribute">
396<dt id="eventmq.router.Router.waiting_messages">
397<code class="descname">waiting_messages</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.waiting_messages" title="Permalink to this definition">¶</a></dt>
398<dd><p>Message buffer. When messages can&#8217;t be sent because there are no
399workers available to take the job</p>
400</dd></dl>
401
402<dl class="attribute">
403<dt id="eventmq.router.Router.workers">
404<code class="descname">workers</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.workers" title="Permalink to this definition">¶</a></dt>
405<dd><p>List of queues by workers. Meta data about the worker such as the
406queue memebership and timestamp of last message received are stored
407here.</p>
408<dl class="docutils">
409<dt><strong>Keys</strong></dt>
410<dd><ul class="first last simple">
411<li>queues: list() of queues the worker belongs to</li>
412<li>hb: monotonic timestamp of the last received message from worker</li>
413</ul>
414</dd>
415</dl>
416</dd></dl>
417
309</dd></dl> 418</dd></dl>
310 419
311</div> 420</div>