Synchronized Spider

This abstract class can be extended to produce new plugins that require a system-wide configuration synchronization.

pathspider.sync

class pathspider.sync.SemaphoreN(value)

An extension to the standard library’s BoundedSemaphore that provides functions to handle n tokens at once.

acquire_n(value=1, blocking=True, timeout=None)

Acquire value number of tokens at once.

The parameters blocking and timeout have the same semantics as BoundedSemaphore.

Returns:The same value as the last call to BoundedSemaphore’s acquire() if acquire() were called value times instead of the call to this method.

empty()

Acquire all tokens of the semaphore.

release_n(value=1)

Release value number of tokens at once.

Returns:The same value as the last call to BoundedSemaphore’s release() if release() were called value times instead of the call to this method.

class pathspider.sync.SynchronizedSpider(worker_count, libtrace_uri, args, server_mode=False)

configurations = []

configurator()

Thread which synchronizes on a set of semaphores and alternates between two system states.

connect(job, config)

Performs the requested connection.

classmethod register_args(subparsers)

worker(worker_number)

This function provides the logic for configuration-synchronized worker threads.

Parameters:worker_number (int) – The unique number of the worker.

The workers operate as continuous loops:

• Fetch next job from the job queue
• Perform pre-connection operations
• Acquire a lock for “config_zero”
• Perform the “config_zero” connection
• Release “config_zero”
• Acquire a lock for “config_one”
• Perform the “config_one” connection
• Release “config_one”
• Perform post-connection operations for config_zero and pass the result to the merger
• Perform post-connection operations for config_one and pass the result to the merger
• Do it all again

If the job fetched is the SHUTDOWN_SENTINEL, then the worker will terminate as this indicates that all the jobs have now been processed.