Desynchronized Spider¶
This abstract class can be extended to produce new plugins that do not require a system-wide configuration synchronization.
pathspider.desync¶
-
class
pathspider.desync.
DesynchronizedSpider
(worker_count, libtrace_uri, args, server_mode=False)[source]¶ -
configurator
()[source]¶ Since there is no need for a configurator thread in a desynchronized spider, this thread is a no-op
-
connections
= []¶
-
worker
(worker_number)[source]¶ This function provides the logic for configuration-synchronized worker threads.
Parameters: worker_number (int) – The unique number of the worker. The workers operate as continuous loops:
- Fetch next job from the job queue
- Perform pre-connection operations
- Acquire a lock for “config_zero”
- Perform the “config_zero” connection
- Release “config_zero”
- Acquire a lock for “config_one”
- Perform the “config_one” connection
- Release “config_one”
- Perform post-connection operations for config_zero and pass the result to the merger
- Perform post-connection operations for config_one and pass the result to the merger
- Do it all again
If the job fetched is the SHUTDOWN_SENTINEL, then the worker will terminate as this indicates that all the jobs have now been processed.
-