diff options
Diffstat (limited to 'router.html')
| -rw-r--r-- | router.html | 125 |
1 files changed, 117 insertions, 8 deletions
diff --git a/router.html b/router.html index 1cacb79..0604f89 100644 --- a/router.html +++ b/router.html | |||
| @@ -84,6 +84,7 @@ | |||
| 84 | 84 | ||
| 85 | <ul class="current"> | 85 | <ul class="current"> |
| 86 | <li class="toctree-l1 current"><a class="reference internal" href="api.html">API Documentation</a><ul class="current"> | 86 | <li class="toctree-l1 current"><a class="reference internal" href="api.html">API Documentation</a><ul class="current"> |
| 87 | <li class="toctree-l2"><a class="reference internal" href="client.html"><code class="docutils literal"><span class="pre">client</span></code> – Client Utilities</a></li> | ||
| 87 | <li class="toctree-l2"><a class="reference internal" href="exceptions.html"><code class="docutils literal"><span class="pre">exceptions</span></code> – Exceptions</a></li> | 88 | <li class="toctree-l2"><a class="reference internal" href="exceptions.html"><code class="docutils literal"><span class="pre">exceptions</span></code> – Exceptions</a></li> |
| 88 | <li class="toctree-l2"><a class="reference internal" href="jobmanager.html"><code class="docutils literal"><span class="pre">jobmanager</span></code> – Job Manager</a></li> | 89 | <li class="toctree-l2"><a class="reference internal" href="jobmanager.html"><code class="docutils literal"><span class="pre">jobmanager</span></code> – Job Manager</a></li> |
| 89 | <li class="toctree-l2"><a class="reference internal" href="poller.html"><code class="docutils literal"><span class="pre">poller</span></code> – Poller</a></li> | 90 | <li class="toctree-l2"><a class="reference internal" href="poller.html"><code class="docutils literal"><span class="pre">poller</span></code> – Poller</a></li> |
| @@ -148,6 +149,20 @@ | |||
| 148 | <em class="property">class </em><code class="descclassname">eventmq.router.</code><code class="descname">Router</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router" title="Permalink to this definition">¶</a></dt> | 149 | <em class="property">class </em><code class="descclassname">eventmq.router.</code><code class="descname">Router</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router" title="Permalink to this definition">¶</a></dt> |
| 149 | <dd><p>A simple router of messages</p> | 150 | <dd><p>A simple router of messages</p> |
| 150 | <dl class="method"> | 151 | <dl class="method"> |
| 152 | <dt id="eventmq.router.Router.add_scheduler"> | ||
| 153 | <code class="descname">add_scheduler</code><span class="sig-paren">(</span><em>scheduler_id</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.add_scheduler" title="Permalink to this definition">¶</a></dt> | ||
| 154 | <dd><p>Adds a scheduler to the queue to receive SCHEDULE commands</p> | ||
| 155 | <table class="docutils field-list" frame="void" rules="none"> | ||
| 156 | <col class="field-name" /> | ||
| 157 | <col class="field-body" /> | ||
| 158 | <tbody valign="top"> | ||
| 159 | <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>scheduler_id</strong> (<a class="reference external" href="https://docs.python.org/library/functions.html#str" title="(in Python v2.7)"><em>str</em></a>) – unique id of the scheduler to add</td> | ||
| 160 | </tr> | ||
| 161 | </tbody> | ||
| 162 | </table> | ||
| 163 | </dd></dl> | ||
| 164 | |||
| 165 | <dl class="method"> | ||
| 151 | <dt id="eventmq.router.Router.add_worker"> | 166 | <dt id="eventmq.router.Router.add_worker"> |
| 152 | <code class="descname">add_worker</code><span class="sig-paren">(</span><em>worker_id</em>, <em>queues=None</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.add_worker" title="Permalink to this definition">¶</a></dt> | 167 | <code class="descname">add_worker</code><span class="sig-paren">(</span><em>worker_id</em>, <em>queues=None</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.add_worker" title="Permalink to this definition">¶</a></dt> |
| 153 | <dd><p>Adds a worker to worker queues</p> | 168 | <dd><p>Adds a worker to worker queues</p> |
| @@ -156,7 +171,7 @@ | |||
| 156 | <col class="field-body" /> | 171 | <col class="field-body" /> |
| 157 | <tbody valign="top"> | 172 | <tbody valign="top"> |
| 158 | <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> | 173 | <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| 159 | <li><strong>worker_id</strong> – unique id of the worker to add</li> | 174 | <li><strong>worker_id</strong> (<a class="reference external" href="https://docs.python.org/library/functions.html#str" title="(in Python v2.7)"><em>str</em></a>) – unique id of the worker to add</li> |
| 160 | <li><strong>queues</strong> – queue or queues this worker should be a member of</li> | 175 | <li><strong>queues</strong> – queue or queues this worker should be a member of</li> |
| 161 | </ul> | 176 | </ul> |
| 162 | </td> | 177 | </td> |
| @@ -166,6 +181,13 @@ | |||
| 166 | </dd></dl> | 181 | </dd></dl> |
| 167 | 182 | ||
| 168 | <dl class="method"> | 183 | <dl class="method"> |
| 184 | <dt id="eventmq.router.Router.clean_up_dead_schedulers"> | ||
| 185 | <code class="descname">clean_up_dead_schedulers</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.clean_up_dead_schedulers" title="Permalink to this definition">¶</a></dt> | ||
| 186 | <dd><p>Loops through the list of schedulers and remove any schedulers who | ||
| 187 | the router hasn’t received a heartbeat in HEARTBEAT_TIMEOUT</p> | ||
| 188 | </dd></dl> | ||
| 189 | |||
| 190 | <dl class="method"> | ||
| 169 | <dt id="eventmq.router.Router.clean_up_dead_workers"> | 191 | <dt id="eventmq.router.Router.clean_up_dead_workers"> |
| 170 | <code class="descname">clean_up_dead_workers</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.clean_up_dead_workers" title="Permalink to this definition">¶</a></dt> | 192 | <code class="descname">clean_up_dead_workers</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.clean_up_dead_workers" title="Permalink to this definition">¶</a></dt> |
| 171 | <dd><p>Loops through the worker queues and removes any workers who haven’t | 193 | <dd><p>Loops through the worker queues and removes any workers who haven’t |
| @@ -209,10 +231,7 @@ and announces itself.</p> | |||
| 209 | <dl class="method"> | 231 | <dl class="method"> |
| 210 | <dt id="eventmq.router.Router.on_receive_request"> | 232 | <dt id="eventmq.router.Router.on_receive_request"> |
| 211 | <code class="descname">on_receive_request</code><span class="sig-paren">(</span><em>msg</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.on_receive_request" title="Permalink to this definition">¶</a></dt> | 233 | <code class="descname">on_receive_request</code><span class="sig-paren">(</span><em>msg</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.on_receive_request" title="Permalink to this definition">¶</a></dt> |
| 212 | <dd><p>This function is called when a message comes in from the client socket. | 234 | <dd><table class="docutils field-list" frame="void" rules="none"> |
| 213 | It then calls <cite>on_command</cite>. If <cite>on_command</cite> isn’t found, then a | ||
| 214 | warning is created.</p> | ||
| 215 | <table class="docutils field-list" frame="void" rules="none"> | ||
| 216 | <col class="field-name" /> | 235 | <col class="field-name" /> |
| 217 | <col class="field-body" /> | 236 | <col class="field-body" /> |
| 218 | <tbody valign="top"> | 237 | <tbody valign="top"> |
| @@ -239,9 +258,70 @@ a warning is created.</p> | |||
| 239 | </dd></dl> | 258 | </dd></dl> |
| 240 | 259 | ||
| 241 | <dl class="method"> | 260 | <dl class="method"> |
| 261 | <dt id="eventmq.router.Router.queue_message"> | ||
| 262 | <code class="descname">queue_message</code><span class="sig-paren">(</span><em>msg</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.queue_message" title="Permalink to this definition">¶</a></dt> | ||
| 263 | <dd><p>Add a message to the queue for processing later</p> | ||
| 264 | </dd></dl> | ||
| 265 | |||
| 266 | <dl class="attribute"> | ||
| 267 | <dt id="eventmq.router.Router.queues"> | ||
| 268 | <code class="descname">queues</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.queues" title="Permalink to this definition">¶</a></dt> | ||
| 269 | <dd><p>JobManager address by queue name. The lists here are Last Recently | ||
| 270 | Used queues where a worker is popped off when given a job, and | ||
| 271 | appeneded when one finishes. There is one entry per available | ||
| 272 | worker slot, so you may see duplicate addresses.</p> | ||
| 273 | <p class="rubric">Example</p> | ||
| 274 | <p>{‘default’: [‘w1’, ‘w2’, ‘w1’, ‘w4’]}</p> | ||
| 275 | </dd></dl> | ||
| 276 | |||
| 277 | <dl class="attribute"> | ||
| 278 | <dt id="eventmq.router.Router.received_disconnect"> | ||
| 279 | <code class="descname">received_disconnect</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.received_disconnect" title="Permalink to this definition">¶</a></dt> | ||
| 280 | <dd><p>Set to True when the router should die.</p> | ||
| 281 | </dd></dl> | ||
| 282 | |||
| 283 | <dl class="method"> | ||
| 242 | <dt id="eventmq.router.Router.requeue_worker"> | 284 | <dt id="eventmq.router.Router.requeue_worker"> |
| 243 | <code class="descname">requeue_worker</code><span class="sig-paren">(</span><em>worker_id</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.requeue_worker" title="Permalink to this definition">¶</a></dt> | 285 | <code class="descname">requeue_worker</code><span class="sig-paren">(</span><em>worker_id</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.requeue_worker" title="Permalink to this definition">¶</a></dt> |
| 244 | <dd><p>Add a worker back to the queue pool</p> | 286 | <dd><p>Add a worker back to the pools for which it is a member of.</p> |
| 287 | <div class="admonition note"> | ||
| 288 | <p class="first admonition-title">Note</p> | ||
| 289 | <p class="last">This will (correctly) add duplicate items into the queues.</p> | ||
| 290 | </div> | ||
| 291 | </dd></dl> | ||
| 292 | |||
| 293 | <dl class="method"> | ||
| 294 | <dt id="eventmq.router.Router.reset_heartbeat_counters"> | ||
| 295 | <code class="descname">reset_heartbeat_counters</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.reset_heartbeat_counters" title="Permalink to this definition">¶</a></dt> | ||
| 296 | <dd><p>Reset all the counters for heartbeats back to 0</p> | ||
| 297 | </dd></dl> | ||
| 298 | |||
| 299 | <dl class="method"> | ||
| 300 | <dt id="eventmq.router.Router.router_main"> | ||
| 301 | <code class="descname">router_main</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.router_main" title="Permalink to this definition">¶</a></dt> | ||
| 302 | <dd><p>Kick off router with logging and settings import</p> | ||
| 303 | </dd></dl> | ||
| 304 | |||
| 305 | <dl class="attribute"> | ||
| 306 | <dt id="eventmq.router.Router.scheduler_queue"> | ||
| 307 | <code class="descname">scheduler_queue</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.scheduler_queue" title="Permalink to this definition">¶</a></dt> | ||
| 308 | <dd><p><em>Queue for schedulers to use</em></p> | ||
| 309 | </dd></dl> | ||
| 310 | |||
| 311 | <dl class="attribute"> | ||
| 312 | <dt id="eventmq.router.Router.schedulers"> | ||
| 313 | <code class="descname">schedulers</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.schedulers" title="Permalink to this definition">¶</a></dt> | ||
| 314 | <dd><p>Scheduler clients. Clients are able to send SCHEDULE commands that | ||
| 315 | need to be routed to a scheduler, which will keep track of time and | ||
| 316 | run the job. | ||
| 317 | Contains dictionaries:</p> | ||
| 318 | <blockquote> | ||
| 319 | <div><dl class="docutils"> | ||
| 320 | <dt>self.schedulers[<scheduler_zmq_id>] = {</dt> | ||
| 321 | <dd>‘hb’: <last_recv_heartbeat>,</dd> | ||
| 322 | </dl> | ||
| 323 | <p>}</p> | ||
| 324 | </div></blockquote> | ||
| 245 | </dd></dl> | 325 | </dd></dl> |
| 246 | 326 | ||
| 247 | <dl class="method"> | 327 | <dl class="method"> |
| @@ -283,14 +363,20 @@ needed when building messages</p> | |||
| 283 | </dd></dl> | 363 | </dd></dl> |
| 284 | 364 | ||
| 285 | <dl class="method"> | 365 | <dl class="method"> |
| 366 | <dt id="eventmq.router.Router.send_schedulers_heartbeats"> | ||
| 367 | <code class="descname">send_schedulers_heartbeats</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.send_schedulers_heartbeats" title="Permalink to this definition">¶</a></dt> | ||
| 368 | <dd><p>Send HEARTBEATs to all registered schedulers</p> | ||
| 369 | </dd></dl> | ||
| 370 | |||
| 371 | <dl class="method"> | ||
| 286 | <dt id="eventmq.router.Router.send_workers_heartbeats"> | 372 | <dt id="eventmq.router.Router.send_workers_heartbeats"> |
| 287 | <code class="descname">send_workers_heartbeats</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.send_workers_heartbeats" title="Permalink to this definition">¶</a></dt> | 373 | <code class="descname">send_workers_heartbeats</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.send_workers_heartbeats" title="Permalink to this definition">¶</a></dt> |
| 288 | <dd><p>Send heartbeats to all registered workers.</p> | 374 | <dd><p>Send HEARTBEATs to all registered workers.</p> |
| 289 | </dd></dl> | 375 | </dd></dl> |
| 290 | 376 | ||
| 291 | <dl class="method"> | 377 | <dl class="method"> |
| 292 | <dt id="eventmq.router.Router.start"> | 378 | <dt id="eventmq.router.Router.start"> |
| 293 | <code class="descname">start</code><span class="sig-paren">(</span><em>frontend_addr='tcp://127.0.0.1:47290'</em>, <em>backend_addr='tcp://127.0.0.1:47291'</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.start" title="Permalink to this definition">¶</a></dt> | 379 | <code class="descname">start</code><span class="sig-paren">(</span><em>frontend_addr='tcp://127.0.0.1:47291'</em>, <em>backend_addr='tcp://127.0.0.1:47290'</em><span class="sig-paren">)</span><a class="headerlink" href="#eventmq.router.Router.start" title="Permalink to this definition">¶</a></dt> |
| 294 | <dd><p>Begin listening for connections on the provided connection strings</p> | 380 | <dd><p>Begin listening for connections on the provided connection strings</p> |
| 295 | <table class="docutils field-list" frame="void" rules="none"> | 381 | <table class="docutils field-list" frame="void" rules="none"> |
| 296 | <col class="field-name" /> | 382 | <col class="field-name" /> |
| @@ -306,6 +392,29 @@ needed when building messages</p> | |||
| 306 | </table> | 392 | </table> |
| 307 | </dd></dl> | 393 | </dd></dl> |
| 308 | 394 | ||
| 395 | <dl class="attribute"> | ||
| 396 | <dt id="eventmq.router.Router.waiting_messages"> | ||
| 397 | <code class="descname">waiting_messages</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.waiting_messages" title="Permalink to this definition">¶</a></dt> | ||
| 398 | <dd><p>Message buffer. When messages can’t be sent because there are no | ||
| 399 | workers available to take the job</p> | ||
| 400 | </dd></dl> | ||
| 401 | |||
| 402 | <dl class="attribute"> | ||
| 403 | <dt id="eventmq.router.Router.workers"> | ||
| 404 | <code class="descname">workers</code><em class="property"> = None</em><a class="headerlink" href="#eventmq.router.Router.workers" title="Permalink to this definition">¶</a></dt> | ||
| 405 | <dd><p>List of queues by workers. Meta data about the worker such as the | ||
| 406 | queue memebership and timestamp of last message received are stored | ||
| 407 | here.</p> | ||
| 408 | <dl class="docutils"> | ||
| 409 | <dt><strong>Keys</strong></dt> | ||
| 410 | <dd><ul class="first last simple"> | ||
| 411 | <li>queues: list() of queues the worker belongs to</li> | ||
| 412 | <li>hb: monotonic timestamp of the last received message from worker</li> | ||
| 413 | </ul> | ||
| 414 | </dd> | ||
| 415 | </dl> | ||
| 416 | </dd></dl> | ||
| 417 | |||
| 309 | </dd></dl> | 418 | </dd></dl> |
| 310 | 419 | ||
| 311 | </div> | 420 | </div> |