blob: 04fe451b826679c10cac9a7ecf41fed5815ee7dd (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
|
##############################
Server Settings (eventmq.conf)
##############################
EventMQ uses a standard INI style config file with the default
location of ``/etc/eventmq.conf``. If you would like to specify a custom path
you can use the ``EVENTMQ_CONFIG_FILE`` environment variable.
All of these options can be defined via environment variables by converting
them to upper case and prefixing them with ``EVENTMQ_``. For example
``EVENTMQ_MAX_SOCKETS=2048``.
******
Global
******
super_debug
===========
Default: False
Enable most verbose level of debug statements
max_sockets
===========
Default: 1024
Define the max sockets for a process/context
******
Router
******
frontend_addr
=============
Default: 'tcp://127.0.0.1:47291'
The address used to listen for client and scheduler connections
backend_addr
============
Default: 'tcp://127.0.0.1:47291'
The address used to listen for connections from workers
wal
===
Default: '/var/log/eventmq/wal.log'
Write-ahead Log for replaying messages received by the Router. Will
try to create the directory specified and append to the filename given.
Requires correct permissions to write to the given file.
wal_enabled
===========
Default: False
Enable or disable the Write-ahead Log
*********
Scheduler
*********
scheduler_addr
==============
Default: 'tcp://127.0.0.1:47291'
The address the scheduler will use to connect to the broker
rq_host
=======
Default: 'localhost'
The hostname of the redis server used to persist scheduled jobs. This is
expected to support redis' save operation which saves the contents to disk.
rq_port
=======
Default: 6379
Port of redis server to connect to.
rq_db
=====
Default: 0
Which redis database to use
rq_password
===========
Default: ''
Password to use when connecting to redis
***********
Job Manager
***********
concurrent_jobs
===============
Default: 4
This is the number of concurrent jobs the indiviudal job manager should execute
at a time. If you are using the multiprocess or threading model this number
becomes important as you will want to control the load on your server. If the
load equals the number of cores on the server, processes will begin waiting for
cpu cycles and things will begin to slow down.
A safe number to choose if your jobs block a lot would be (2 * cores). If your
jobs are cpu intensive you will want to set this number to the number of cores
you have or (cores - 1) to leave cycles for the os and other processes. This is
something that will have to be tuned based on the jobs that are
running. Grouping similar jobs in named queues will help you tune this number.
queues
======
Default: [[10, "default"]]
Comma seperated list of queues to process jobs for with their weights. This list
must be valid JSON otherwise an error will be thrown.
Example: ``queues=[[10, "data_process"], [15, "email"]]``. With these
weights and the ``CONCURRENT_JOBS`` setting, you should be able to tune managers
running jobs locally efficiently. If you have a larger server with a weight of
50 on q1 and 8 concurrent jobs and a smaller server with a weight 30 and 4
concurrent jobs, the q1 jobs will be sent to the large box until it is no longer
accepting jobs. At this point jobs will start to be sent to the next highest
number until the large box is ready to accept another q1 job.
.. note::
It is recommended that you have some workers listening for jobs on your
default queue so that anything that is not explicitly assigned will still be
run.
setup_callabe/setup_path
========================
Default: '' (Signifies no task will be attempted)
Strings containing path and callable to be run when a worker is spawned
if applicable to that type of worker. Currently the only supported worker is a
MultiProcessWorker, and is useful for pulling any global state into memory.
max_job_count
=============
Default: 1024
After a worker runs this amount of jobs, it will gracefully exit and be replaced
|
