Synchronized Spider

This abstract class can be extended to produce new plugins that require a system-wide configuration synchronization.

pathspider.sync

class pathspider.sync.SemaphoreN(value)

An extension to the standard library’s BoundedSemaphore that provides functions to handle n tokens at once.

acquire_n(value=1, blocking=True, timeout=None)

Acquire value number of tokens at once.

The parameters blocking and timeout have the same semantics as BoundedSemaphore.

Returns:The same value as the last call to BoundedSemaphore’s acquire() if acquire() were called value times instead of the call to this method.

empty()

Acquire all tokens of the semaphore.

release_n(value=1)

Release value number of tokens at once.

class pathspider.sync.SynchronizedSpider(worker_count, libtrace_uri, args, server_mode=False)

configurations = []

configurator()

Thread which synchronizes on a set of semaphores and alternates between two system states.

connect(job, config)

Performs the requested connection.

classmethod register_args(subparsers)

worker(worker_number)

This function provides the logic for configuration-synchronized worker threads.

Parameters:worker_number (int) – The unique number of the worker.

The workers operate as continuous loops:

• Fetch next job from the job queue

• For each configuration:

  • Acquire a lock for the configuration (blocking)
  • Perform the connection with the configuration
  • Release the lock
  • Pass the result to the merger

• Do it all again

If the job fetched is the SHUTDOWN_SENTINEL, then the worker will terminate as this indicates that all the jobs have now been processed.