Adds rewrite rules clear api.

Fixes content type for request/response intercept and adds doc links for writing intercept code.
Adds comments clarifying intercept client test coverage.
Augments human test for url rewrite and adds clear portion.
Adds test for response intercept.
Adds bootstrap script to download tools to test with.
Refactors human tests so browser session are cleaned up when tests fail and tests run with chromium.
Adds server start script for travis.
Changes .travis.yml to use provided scripts and also run headless chrome.

Updates .travis.yml not to skip all tests.

Updates .travis.yml not to skip all tests.
Removes all but Python 2.7 for testing purposes.

Removes redundant argument from .travis.yml.
 Updates webdriver tests to use chrome with the proxy. (FF does not work with actual tests)
 Updates tests to be functional tests. (uses proper assertions)
 Removes redundant lines from the remote tests.

Reduces tests to human tests only for testing.
Fixes Chromium binary location setting.

Tweaks .travis.yml

Tweaking server start scripts.

Adding jdk version for proper functionality of Se server.

Sets oracle jdk8 as default.

Tweaks .travis.yml and bootstrap.sh

Adds Linux binaries.

Reverting to chrome stable for testing.

Adding dist spec for google-chrome-stable installation.

Adds chrome no sandbox option.

Now all tests can run on Travis.

Author: Mehmet Gerceker

.travis.yml

@@ -1,4 +1,15 @@
 language: python
+
+jdk:
+  - oraclejdk8
+
+addons:
+  apt:
+    packages:
+      - oracle-java8-installer
+      - oracle-java8-set-default
+      - google-chrome-stable
+
 sudo: false
 python:
   - 2.6
@@ -7,10 +18,12 @@ python:
   - 3.4
   - 3.5
 before_script:
-  - curl -l -L -O https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-6/browsermob-proxy-2.1.0-beta-6-bin.zip
-  - unzip browsermob-proxy-2.1.0-beta-6-bin.zip
-  - sh browsermob-proxy-2.1.0-beta-6/bin/browsermob-proxy --port=9090 &
+  - export DISPLAY=:99.0
+  - sh -e /etc/init.d/xvfb start
+  - ./bootstrap.sh
+  - ./start-servers.sh
   - sleep 5
 install: pip install pytest selenium
-script: python setup.py develop && py.test -m "not human" test
+script: python setup.py develop && py.test test
 sudo: false
+dist: trusty

bootstrap.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+if [ "$(uname)" == "Darwin" ]; then
+    echo "We're on a MAC!"
+   chromeDriver="chromedriver_mac64.zip"
+   geckoDriverVersion="v0.13.0"
+   geckoDriver="geckodriver-$geckoDriverVersion-macos.tar.gz"
+else
+    echo "We're not on a MAC!"
+   chromeDriver="chromedriver_linux64.zip"
+   geckoDriverVersion="v0.14.0"
+   geckoDriver="geckodriver-$geckoDriverVersion-linux64.tar.gz"
+fi
+rm -rf tools
+mkdir -p tools && \
+cd tools && \
+wget https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.4/browsermob-proxy-2.1.4-bin.zip && \
+unzip -o browsermob-proxy-2.1.4-bin.zip && \
+rm -rf browsermob-proxy*.zip* && \
+wget http://selenium-release.storage.googleapis.com/3.0/selenium-server-standalone-3.0.1.jar && \
+wget https://github.com/mozilla/geckodriver/releases/download/${geckoDriverVersion}/${geckoDriver} && \
+tar zxf ${geckoDriver} && \
+rm -rf ${geckoDriver}* && \
+wget https://chromedriver.storage.googleapis.com/2.27/${chromeDriver} && \
+unzip ${chromeDriver} && \
+rm -rf ${chromeDriver}* && \
+cd .. 

browsermobproxy/client.py

@@ -34,7 +34,7 @@ def __init__(self, url, params=None, options=None):
                 jcontent = json.loads(content)
             except Exception as e:
                 raise Exception("Could not read Browsermob-Proxy json\n"
-                    "Another server running on this port?\n%s..."%content[:512])
+                                "Another server running on this port?\n%s..." % content[:512])
             self.port = jcontent['port']
         url_parts = self.host.split(":")
         self.proxy = url_parts[1][2:] + ":" + str(self.port)
@@ -103,7 +103,6 @@ def har(self):
 
         return r.json()
 
-
     def new_har(self, ref=None, options=None, title=None):
         """
         This sets a new HAR to be recorded
@@ -199,24 +198,30 @@ def headers(self, headers):
 
     def response_interceptor(self, js):
         """
-        Executes the javascript against each response
-
-        :param str js: the javascript to execute
+        Executes the java/js code against each response
+        `HttpRequest request <https://netty.io/4.1/api/io/netty/handler/codec/http/HttpRequest.html>`_,
+        `HttpMessageContents contents <https://raw.githubusercontent.com/lightbody/browsermob-proxy/master/browsermob-core/src/main/java/net/lightbody/bmp/util/HttpMessageContents.java>`_,
+        `HttpMessageInfo messageInfo <https://raw.githubusercontent.com/lightbody/browsermob-proxy/master/browsermob-core/src/main/java/net/lightbody/bmp/util/HttpMessageInfo.java>`_
+        are available objects to interact with.
+        :param str js: the js/java code to execute
         """
         r = requests.post(url='%s/proxy/%s/filter/response' % (self.host, self.port),
-                  data=js,
-                  headers={'content-type': 'application/json'})
+                          data=js,
+                          headers={'content-type': 'text/plain'})
         return r.status_code
 
     def request_interceptor(self, js):
         """
-        Executes the javascript against each request
-
-        :param str js: the javascript to execute
+        Executes the java/js code against each response
+        `HttpRequest request <https://netty.io/4.1/api/io/netty/handler/codec/http/HttpRequest.html>`_,
+        `HttpMessageContents contents <https://raw.githubusercontent.com/lightbody/browsermob-proxy/master/browsermob-core/src/main/java/net/lightbody/bmp/util/HttpMessageContents.java>`_,
+        `HttpMessageInfo messageInfo <https://raw.githubusercontent.com/lightbody/browsermob-proxy/master/browsermob-core/src/main/java/net/lightbody/bmp/util/HttpMessageInfo.java>`_
+        are available objects to interact with.
+        :param str js: the js/java code to execute
         """
         r = requests.post(url='%s/proxy/%s/filter/request' % (self.host, self.port),
-                  data=js,
-                  headers={'content-type': 'application/json'})
+                          data=js,
+                          headers={'content-type': 'text/plain'})
         return r.status_code
 
     LIMITS = {
@@ -295,7 +300,7 @@ def remap_hosts(self, address=None, ip_address=None, hostmap=None):
             hostmap[address] = ip_address
 
         r = requests.post('%s/proxy/%s/hosts' % (self.host, self.port),
-                         json.dumps(hostmap),
+                          json.dumps(hostmap),
                           headers={'content-type': 'application/json'})
         return r.status_code
 
@@ -308,7 +313,7 @@ def wait_for_traffic_to_stop(self, quiet_period, timeout):
         :param int timeout: max number of milliseconds to wait
         """
         r = requests.put('%s/proxy/%s/wait' % (self.host, self.port),
-                 {'quietPeriodInMs': quiet_period, 'timeoutInMs': timeout})
+                         {'quietPeriodInMs': quiet_period, 'timeoutInMs': timeout})
         return r.status_code
 
     def clear_dns_cache(self):
@@ -334,12 +339,21 @@ def rewrite_url(self, match, replace):
                          params)
         return r.status_code
 
+    def clear_all_rewrite_url_rules(self):
+        """
+        Clears all URL rewrite rules
+        :return: status code
+        """
+
+        r = requests.delete('%s/proxy/%s/rewrite' % (self.host, self.port))
+        return r.status_code
+
     def retry(self, retry_count):
         """
         Retries. No idea what its used for, but its in the API...
 
         :param int retry_count: the number of retries
         """
         r = requests.put('%s/proxy/%s/retry' % (self.host, self.port),
-                 {'retrycount': retry_count})
+                         {'retrycount': retry_count})
         return r.status_code

start-servers.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+TOOLS=$(pwd)/tools
+cd $TOOLS/browsermob-proxy-2.1.4/bin/ && \
+./browsermob-proxy --port 9090 & \
+cd $TOOLS && \
+java -Dwebdriver.chrome.driver=chromedriver -Dwebdriver.gecko.driver=geckodriver -jar selenium-server-standalone-3.0.1.jar &

test/test_client.py

@@ -167,6 +167,8 @@ def test_close(self):
     def test_response_interceptor_with_parsing_js(self):
         """
         /proxy/:port/interceptor/response
+        This test is only checking very basic syntax rules. The snippet needs to be JAVA/JS
+        Code only gets validated if the filter/interceptor is used.
         """
         js = 'alert("foo")'
         status_code = self.client.response_interceptor(js)
@@ -175,6 +177,8 @@ def test_response_interceptor_with_parsing_js(self):
     def test_response_interceptor_with_invalid_js(self):
         """
         /proxy/:port/interceptor/response
+        This test is only checking very basic syntax rules. The snippet needs to be JAVA/JS
+        Code only gets validated if the filter/interceptor is used.
         """
         js = 'alert("foo"'
         status_code = self.client.response_interceptor(js)
@@ -183,6 +187,8 @@ def test_response_interceptor_with_invalid_js(self):
     def test_request_interceptor_with_parsing_js(self):
         """
         /proxy/:port/interceptor/request
+        This test is only checking very basic syntax rules. The snippet needs to be JAVA/JS
+        Code only gets validated if the filter/interceptor is used.
         """
         js = 'alert("foo")'
         status_code = self.client.request_interceptor(js)
@@ -191,6 +197,8 @@ def test_request_interceptor_with_parsing_js(self):
     def test_request_interceptor_with_invalid_js(self):
         """
         /proxy/:port/interceptor/request
+        This test is only checking very basic syntax rules. The snippet needs to be JAVA/JS
+        Code only gets validated if the filter/interceptor is used.
         """
         js = 'alert("foo"'
         status_code = self.client.request_interceptor(js)

test/test_remote.py

@@ -1,31 +1,55 @@
+from os import environ
+
 from selenium import webdriver
 import selenium.webdriver.common.desired_capabilities
 import os
 import sys
-import time
 import pytest
 
+
 def setup_module(module):
     sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
 
+
 class TestRemote(object):
     def setup_method(self, method):
         from browsermobproxy.client import Client
         self.client = Client("localhost:9090")
+        chrome_binary = environ.get("CHROME_BIN", None)
+        self.desired_caps = selenium.webdriver.common.desired_capabilities.DesiredCapabilities.CHROME
+        if chrome_binary is not None:
+            self.desired_caps.update({
+                "chromeOptions": {
+                    "binary": chrome_binary,
+                    "args": ['no-sandbox']
+                }
+            })
+        self.driver = webdriver.Remote(
+            desired_capabilities=self.desired_caps,
+            proxy=self.client)
 
     def teardown_method(self, method):
         self.client.close()
+        self.driver.quit()
 
     @pytest.mark.human
-    def test_i_want_my_remote(self):
-        driver = webdriver.Remote(desired_capabilities=selenium.webdriver.common.desired_capabilities.DesiredCapabilities.FIREFOX,
-                                  proxy=self.client)
+    def test_set_clear_url_rewrite_rule(self):
+        targetURL = "https://www.saucelabs.com/versions.js"
+        assert 200 == self.client.rewrite_url(
+            "https://www.saucelabs.com/versions.+", "https://www.saucelabs.com/versions.json"
+        )
+        self.driver.get(targetURL)
+        assert "Sauce Connect" in self.driver.page_source
+        assert self.client.clear_all_rewrite_url_rules() == 200
+        self.driver.get(targetURL)
+        assert "Sauce Connect" not in self.driver.page_source
 
-        self.client.new_har("mtv")
-        targetURL = "http://www.mtv.com"
-        self.client.rewrite_url(".*american_flag-384x450\\.jpg", "http://www.foodsubs.com/Photos/englishmuffin.jpg")
-
-        driver.get(targetURL)
-        time.sleep(5)
-
-        driver.quit()
+    @pytest.mark.human
+    def test_response_interceptor(self):
+        content = "Response successfully intercepted"
+        targetURL = "https://saucelabs.com/versions.json?hello"
+        self.client.response_interceptor(
+            """if(messageInfo.getOriginalUrl().contains('?hello')){contents.setTextContents("%s");}""" % content
+        )
+        self.driver.get(targetURL)
+        assert content in self.driver.page_source

test/test_webdriver.py

@@ -1,10 +1,11 @@
+from os import environ
+
 from selenium import webdriver
 import selenium.webdriver.common.desired_capabilities
 from selenium.webdriver.common.proxy import Proxy
 import os
 import sys
 import copy
-import time
 import pytest
 
 def setup_module(module):
@@ -14,39 +15,28 @@ class TestWebDriver(object):
     def setup_method(self, method):
         from browsermobproxy.client import Client
         self.client = Client("localhost:9090")
+        self.driver = None
 
     def teardown_method(self, method):
         self.client.close()
+        if self.driver is not None:
+            self.driver.quit()
 
     @pytest.mark.human
     def test_i_want_my_by_capability(self):
-        capabilities = selenium.webdriver.common.desired_capabilities.DesiredCapabilities.FIREFOX
+        capabilities = selenium.webdriver.common.desired_capabilities.DesiredCapabilities.CHROME
         self.client.add_to_capabilities(capabilities)
-        driver = webdriver.Firefox(capabilities=capabilities)
-
-        self.client.new_har("mtv")
-        targetURL = "http://www.mtv.com"
-        self.client.rewrite_url(".*american_flag-384x450\\.jpg", "http://www.foodsubs.com/Photos/englishmuffin.jpg")
-
-        driver.get(targetURL)
-
-        time.sleep(5)
+        # sets self.driver for proper clean up
+        self._create_webdriver(capabilites=capabilities)
 
-        driver.quit()
+        self._run_url_rewrite_test()
 
     @pytest.mark.human
     def test_i_want_my_by_proxy_object(self):
-        driver = webdriver.Firefox(proxy=self.client)
+        self._create_webdriver(capabilites=selenium.webdriver.common.desired_capabilities.DesiredCapabilities.CHROME,
+                               proxy=self.client)
 
-        self.client.new_har("mtv")
-        targetURL = "http://www.mtv.com"
-        self.client.rewrite_url(".*american_flag-384x450\\.jpg", "http://www.foodsubs.com/Photos/englishmuffin.jpg")
-
-        driver.get(targetURL)
-
-        time.sleep(5)
-
-        driver.quit()
+        self._run_url_rewrite_test()
 
     def test_what_things_look_like(self):
         bmp_capabilities = copy.deepcopy(selenium.webdriver.common.desired_capabilities.DesiredCapabilities.FIREFOX)
@@ -58,3 +48,28 @@ def test_what_things_look_like(self):
         proxy.add_to_capabilities(proxy_capabilities)
 
         assert bmp_capabilities == proxy_capabilities
+
+    def _create_webdriver(self, capabilites, proxy=None):
+        chrome_binary = environ.get("CHROME_BIN", None)
+        if chrome_binary is not None:
+            capabilites.update({
+                "chromeOptions": {
+                    "binary": chrome_binary,
+                    "args": ['no-sandbox']
+                }
+            })
+        if proxy is None:
+            self.driver = webdriver.Remote(desired_capabilities=capabilites)
+        else:
+            self.driver = webdriver.Remote(desired_capabilities=capabilites, proxy=proxy)
+
+    def _run_url_rewrite_test(self):
+        targetURL = "https://www.saucelabs.com/versions.js"
+        assert 200 == self.client.rewrite_url(
+            "https://www.saucelabs.com/versions.+", "https://www.saucelabs.com/versions.json"
+        )
+        self.driver.get(targetURL)
+        assert "Sauce Connect" in self.driver.page_source
+        assert self.client.clear_all_rewrite_url_rules() == 200
+        self.driver.get(targetURL)
+        assert "Sauce Connect" not in self.driver.page_source