Synchronized Spider

This abstract class can be extended to produce new plugins that require a system-wide configuration synchronization.

pathspider.sync

class pathspider.sync.SemaphoreN(value)[source]

An extension to the standard library’s BoundedSemaphore that provides functions to handle n tokens at once.

acquire_n(value=1, blocking=True, timeout=None)[source]

Acquire value number of tokens at once.

The parameters blocking and timeout have the same semantics as BoundedSemaphore.

Returns:The same value as the last call to BoundedSemaphore’s acquire() if acquire() were called value times instead of the call to this method.
empty()[source]

Acquire all tokens of the semaphore.

release_n(value=1)[source]

Release value number of tokens at once.

Returns:The same value as the last call to BoundedSemaphore’s release() if release() were called value times instead of the call to this method.
class pathspider.sync.SynchronizedSpider(worker_count, libtrace_uri, args, server_mode=False)[source]
configurations = []
configurator()[source]

Thread which synchronizes on a set of semaphores and alternates between two system states.

connect(job, config)[source]

Performs the requested connection.

classmethod register_args(subparsers)[source]
worker(worker_number)[source]

This function provides the logic for configuration-synchronized worker threads.

Parameters:worker_number (int) – The unique number of the worker.

The workers operate as continuous loops:

  • Fetch next job from the job queue
  • For each configuration:
    • Acquire a lock for the configuration (blocking)
    • Perform the connection with the configuration
    • Release the lock
    • Pass the result to the merger
  • Do it all again

If the job fetched is the SHUTDOWN_SENTINEL, then the worker will terminate as this indicates that all the jobs have now been processed.