Seleniumrunner Tutorial

intro

selenium is great tool for writing tests to web applications. and with Selenium IDE, it is a breeze to write tests. here i will describe how to run your selenium tests from the command line (instead of manually starting the server, opening a browser, selecting the test file…). if you dont use selenium yet you should take a look at it first.

i will not go over the advantages over having regression tests… but to be effective it should be easy to run the tests. it is very boring to manually run the tests, so once in a while when i decide to run it i found out that some regression bugs appeared, and they were not easy to spot because they might got into the code a while ago. even worse is when somebody else on team introduced the bug ;)

basic set-up

first lets set up a simple example.
i am using python 2.5 and twisted 2.5 (web2 0.2.0 svn20070403 - from ubuntu package) on a ubuntu 7.10 machine.

i am using twisted on this example but you could use anything.

1) sampleserver.py (complete file below)

- create a top level resource that sends a page with the body text "I am here".

class Toplevel(resource.Resource):
  addSlash = True
 
  def render(self, request):
    return http.Response(
      200, 
      {'content-type': http_headers.MimeType('text', 'html')},
      """<html><body>
      I am here.
      </body></html>""")
 
site = server.Site(Toplevel())

to start the server.

twistd --ny sampleserver.py

point your browser to http://localhost:8080. You should see the text "I am here".

2) TestSuccessfulSample.html

- lets create a test case with the Selenium IDE. i put two asserions

  1. the text "I am here" is present of the page
  2. the text "you" is NOT present

you should get something like this:

<html>
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <title>TestSample</title>
</head>
 
<body>
  <table cellpadding="1" cellspacing="1" border="1">
    <thead>
      <tr><td rowspan="1" colspan="3">TestSample</td></tr>
    </thead><tbody>
      <tr>
    <td>open</td>
    <td>/</td>
    <td></td>
      </tr>
      <tr>
    <td>assertTextPresent</td>
    <td>I am here</td>
    <td></td>
      </tr>
      <tr>
    <td>assertTextNotPresent</td>
    <td>you</td>
    <td></td>
      </tr>
</tbody></table>
</body>
</html>

pretty simple. but this way you can run the test only using the selenium IDE.

3) TestSuccessfulSuite.html

- the next step is to run the selenium core test runner.
to use the test runner the test must be part of a test suite written in "selense". it is actually just a html file with a table.

<html>
<head>
  <meta content="text/html; charset=ISO-8859-1"
    http-equiv="content-type">
  <title>Test Suite</title>
</head>
 
<body>
  <table cellpadding="1" cellspacing="1" border="1">
    <tbody>
      <tr><td><b>Test Suite</b></td></tr>
      <tr><td><a href="./TestSuccessfulSample.html">Test Sample</a></td></tr>
    </tbody>
  </table>
</body>
</html>

- we also need to serve selenium core from our server.
so download it and put the "core" on the same direcotry of the server.
update our server to serve static content.
The complete file looks like this:


SampleServer.py

import os
from twisted.web2 import server, http, resource, channel
from twisted.web2 import static, http_headers
 
class Toplevel(resource.Resource):
  addSlash = True
  #serve static content from cwd 
  child_static = static.File(os.getcwd())
 
  def locateChild(self,request,segments):
    # root, show dumb test page
    if segments == [""]:
      return self, ()
    # anything else should be static files.
    return (self.child_static, segments)
 
  def render(self, request):
    return http.Response(
      200, 
      {'content-type': http_headers.MimeType('text', 'html')},
      """<html><body>
      I am here.
      </body></html>""")
 
site = server.Site(Toplevel())
 
# Standard twisted application Boilerplate
from twisted.application import service, strports
application = service.Application("demoserver")
s = strports.service('tcp:8080', channel.HTTPFactory(site))
s.setServiceParent(application)

now your folder should have:

  • sampleserver.py (twisted server)
  • TestSuccessSample.html (selenium test)
  • TestSuccessSuite.html (selenium test suite)
  • core (selenium core folder)

restart the server. point to: http://localhost:8080/core/TestRunner.html . select the file '../TestSuccessfulSuite.html' and hit the run button.

we can give extra parameters to the url to run the tests automatically.
http://localhost:8080/core/TestRunner.html?test=../TestSuccessSuite.html&auto=true

not bad. saving this address in the favorites we could run this test with one click from the browser. but i am not happy yet ;)

running from the command line

now that we have a sample test we can target our problem. how to run it from the cmd line?

all i want is to run a process that will print a single line on the console if the test was successful and exit with "0" if tests passed or "1" if it fails.

when we use the "auto" and "resultsUrl" parameter in the URL the test runner will post the results to the given url.
As explained in Continuous Integration section on the selenium core usage page, there are two ways to get the posted data:

  1. use Selenium Remote Control
  2. write your own web application to handle the POSTed data

Using Selenium Remote Control

using selenium RC is pretty straight forward. just download it, start your server and type this in your shell:

java -jar selenium-server.jar -htmlSuite "*firefox" "http://localhost:8080" "/path/to/folder/TestSuccessfulSuite.html" "/path/to/folder/result.html"

selenium RC starts a new firefox instace and saves the result in a html page (specified in the URL as a parameter).

selenium documentation suggests creating a script to parse the html result file. apart from this there some other problems in using selenium RC for this task. the server has to be started and stopped manually, it doesn't close the opened browser window. and selenium RC is an overkill for this task.

seleniumrunner.py

i wrote a small web app just to handle the post data. i wrote it in twisted, so you will need twisted to execute the test runner. note that your own web app under test can use any other web server, the test runner is a different process.

class Toplevel(resource.PostableResource):
    """ Twisted web2 Server. process selenium TestRunner.html POSTed result"""
    addSlash = True
 
    def render(self, request):
        # process POST
        # result 0 => successful, 1=> fail
        if request.args['result']==['passed']:
            print "ok"
            runner.result = 0
        else:
            print "fail"
            runner.result = 1
 
        runner.terminate()
 
        return http.Response(
            200, 
            {'content-type': http_headers.MimeType('text', 'html')},
            """<html><body> got result, thanks. </body></html>""")

this application is designed to handle a single post.
first it will look at the value of "result" parameter posted. it will save the value on the 'runner' instance, call the runner terminate method and give a polite http reply.

the runner is more than just this server. it will start the application/server under test, and open firefox on the selenium TestRunner.html page with arguments to open the test suite and run it automatically. after the test is complete the runner will collect the result and kill all subprocess and terminate its own execution. after stoping the server the script will exit with the result code 0 = success, 1 = failure.

ok. we are almost there. but first i need to talk about starting firefox from the command line… we want to execute firefox in a different session. take a look in this blog post and this page.
just giving this profile as argument is not enough, you also need to tell firefox not to use any running instance:

firefox -no-remote -P selrunner

so putting everything together i got this:
seleniumrunner.py

import os, sys, signal
import subprocess
from optparse import OptionParser
 
from twisted.web2 import server, http, resource, channel
from twisted.web2 import http_headers
from twisted.internet import reactor
 
usage = """usage: %prog [options] app testFile
 app \t command line used to start the application/server under test
 testFile \t selenium test suite file (path relative to TestRunner.html)"""
parser = OptionParser(usage=usage)
parser.add_option("-p","--app-path",dest="serverPath",default=".",
                  help="path from where the server should be started [default: %default]")
parser.add_option("-r","--runner-url",dest="runnerUrl",
                  default="http://localhost:8080/core/TestRunner.html",
                  help="relative path of the html selenium runner [default: %default]")
parser.add_option("-x","--extra-args",dest="optParam",default="",
                  help="additional URL parameter to runner")
 
# parse args
(options, args) = parser.parse_args()
if len(args) != 2:
    parser.error("incorrect number of arguments")
else:
    serverCmd = args[0].split()
    testFile = args[1]
 
class Runner(object):
    """ Selenium runner.
 
    Start and terminate subprocess:
    - application/server under test
    - web browser on selenium TestRunner.html page
 
    """
 
    def __init__(self,serverCmd,testFile,options):
        # 0=> ok, 1=>fail
        self.result = 1
        self.serverCmd = serverCmd
        self.testFile = testFile
        self.options = options
        # we dont want to kill the parent process(if it exist) in self.terminate
        # so change our process group pid.
        PID = os.getpid()
        os.setpgid(PID,PID)
 
    def start(self):
        """ start subprocess"""
        # execute server on specified path
        os.chdir(self.options.serverPath)
 
        # twisted process
        print "starting app server..."
        subprocess.Popen(self.serverCmd)
 
        # FIXME there is no guarantee the server will be ready to receive requests.
        # web browser process
        print "starting browser..."
        self.wp = subprocess.Popen(['firefox','-no-remote','-P','selrunner',
                                    self.options.runnerUrl + "?test=" + 
                                    self.testFile +
                                    "&auto=true&resultsUrl=http://localhost:8082"+
                                    self.options.optParam])
 
    def terminate(self):
        """kill all subprocesses, stop own twisted reactor"""
        os.killpg(os.getpgid(self.wp.pid),signal.SIGTERM)
 
class Toplevel(resource.PostableResource):
    """ Twisted web2 Server. process selenium TestRunner.html POSTed result"""
    addSlash = True
 
    def render(self, request):
        # process POST
        # result 0 => successful, 1=> fail
        if request.args['result']==['passed']:
            print "ok"
            runner.result = 0
        else:
            print "fail"
            runner.result = 1
 
        runner.terminate()
 
        return http.Response(
            200, 
            {'content-type': http_headers.MimeType('text', 'html')},
            """<html><body> got result, thanks. </body></html>""")
 
if __name__ == "__main__":
    runner = Runner(serverCmd,testFile,options)
    site = server.Site(Toplevel())
    reactor.listenTCP(8082,channel.HTTPFactory(site))
    # start runner after starting twisted reactor
    reactor.callWhenRunning(runner.start)
    reactor.run()
 
    # exit process with the test result
    sys.exit(runner.result)

Now in a single shell command you can start the server/app under test and the browser. Run the selenium tests and get the result. Now it is easy to integrate this to your continuous integration scheme.

python seleniumrunner.py "twistd -ny sampleserver.py --pidfile sampleserver.pid --logfile sampleserver.log" "../TestSuccessfulSuite.html"

caveats

it only works on linux and firefox.

opening and closing the firefox window is a bit annoying… but you can get rid of it

And that's it

page_revision: 16, last_edited: 1200764402|%e %b %Y, %H:%M %Z (%O ago)
EditTags History Files Print Site tools+ Options
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License