Installation

Debian GNU/Linux

Note

If there has not been much time since the release, the Debian packages for the latest version may not yet be available.

PATHspider is packaged for Debian and packages are made available for the testing and stable-backports distributions. If you are running Debian stable, ensure that you have enabled the stable-backports repository in your apt sources.

To install PATHspider, simply run:

sudo apt install pathspider

Vagrant

Warning

Depending on the set up of your Vagrant virtualization provider, some tests may be affected. It is wise to test against known configurations to ensure that your networking set up has a clear path to the Internet before running larger measurement campaigns.

On systems other than Linux systems, you may use Vagrant to run PATHspider. This may also be useful during development. A Vagrantfile is provided that will create a Debian-based virtual machine with all the PATHspider dependencies installed.

In the virtual machine, the PATHspider code will be copied to /vagrant. To improve compatibility across platforms, this is not synchronized with the repository outside of the Vagrant image. Expert users may edit the Vagrantfile to achieve this. PATHspider is installed in development mode, meaning that this is also the location of the PATHspider code that will be run when running the /usr/bin/pspdr command inside the virtual machine.

Assuming that you have Vagrant and a virtualisation provider (e.g. VirtualBox) installed, you can get started with:

vagrant up
vagrant ssh

Depending on the speed of your Internet connection, this may take a long time.

virtualenv

Warning

PATHspider 2.0 depends on pycurl >= 7.43.0.1, released on the 7th December 2017. If you have errors when running PATHspider similar to AttributeError: module 'pycurl' has no attribute 'CONNECT_TO' then it is most likely the case that your version of pycurl is too old.

virtualenv is a tool to create isolated Python environments. This allows you to install the dependencies necessary for PATHspider without having them conflict with your system libraries used for other applications on the system.

The following instructions assume a Debian GNU/Linux system and may have to be modified on other systems:

sudo apt install libtrace-dev libldns-dev python3-dev python3-virtualenv
mkdir ~/psenv && cd ~/psenv
python3 -m virtualenv -p /usr/bin/python3 .
source bin/activate
export PATH=$PWD/bin:$PATH
git clone https://github.com/nevil-brownlee/python-libtrace.git
pushd python-libtrace && python3 setup.py install && popd
git clone https://github.com/mami-project/pathspider.git
pushd pathspider && \
    pip install -r requirements.txt && \
    pip install -r requirements_dev.txt && \
    python3 setup.py develop && popd
pspdr test

Ensure that all tests have passed before beginning to measure or develop with PATHspider. To re-enter the virtual environment from another shell session:

cd ~/psenv
source bin/activate
export PATH=$PWD/bin:$PATH
pspdr test

Source

Warning

PATHspider 2.0 depends on pycurl >= 7.43.0.1, released on the 7th December 2017. If you have errors when running PATHspider similar to AttributeError: module 'pycurl' has no attribute 'CONNECT_TO' then it is most likely the case that your version of pycurl is too old.

If you are working from the source distribution (e.g. cloned git repository) then you will need to install the required dependencies. On Debian GNU/Linux, assuming you have the stable-backports repository enabled if you are running stable:

sudo apt build-dep pathspider

Note

This will install both the runtime and the build dependencies required for PATHspider, its testsuite and its documentation.

On other platforms, you may install most of the dependencies required via pip:

pip3 install -r requirements.txt

Unfortunately, python-libtrace is not available on PyPI and so must be installed seperately. You will also need to ensure that for both pycurl and python-libtrace you have the build dependencies available as these are compiled CPython modules.

If you wish to build the documentation from source or to use the testsuite, and you are installing your dependencies via pip, you will also need the following dependencies:

pip3 install -r requirements_dev.txt

With the dependencies installed, you can install PATHspider with:

python3 setup.py install

cloud-init

The following cloud-config script installs PATHspider globally in the VM. The default user is ubuntu. Include your public key to ssh into the VM.

Customise this to your needs. You may want to change the hostname.

#cloud-config

# Hostname management
preserve_hostname: False
hostname: spider
fqdn: spider.local

package_update: true

package_upgrade: true

ssh_authorized_keys:
 - ssh-rsa <include your public SSH key here>

packages:
 - git
 - python3
 - python3-pip
 - python3-setuptools
 - python3-pycurl
 - libtrace-dev
 - libldns-dev

write_files:
- content: |
    #!/bin/bash
    export LC_ALL=C
    # Select the stable pathspider release, comment for github clone
    REL=2.0.1

    cd /tmp
    git clone https://github.com/nevil-brownlee/python-libtrace.git
    cd python-libtrace/
      make install-py3
    cd -

    if [ -z "$REL" ]; then
      git clone https://github.com/mami-project/pathspider
    else
      wget -q -O - https://github.com/mami-project/pathspider/archive/$REL.tar.gz | tar -xzvf -
    fi

    if [ -d pathspider ]; then
      cd pathspider # github clone
    else
      cd pathspider-$REL # release
    fi
        pip3 install -r requirements.txt
        python3 setup.py install
    cd -
  path: /root/build-psp.sh
  permissions: 0755

runcmd:
  - ls -al /root > /install-psp.log
  - /root/build-psp.sh

# Configure where output will go
output:
  all: ">> /var/log/cloud-init.log"

# configure interaction with ssh server
ssh_svcname: ssh
ssh_deletekeys: True
ssh_genkeytypes: ['rsa', 'ecdsa']