1 Getting started

3 ways of execution:

robot init.robot or
python3 robot -m init.robot or
python3 xx/python3.7/site-packages/robot/run.py init.robot

2 Creating test data

2.1 Test data syntax

2.1.1 Files and directories

TestSuites(a test project) contain:

Name	Description
Test files	contains test cases
Initialization file	configure the created test suite dir
Test library	lowest level keywords
Resource files	variables and keywords
Variable files	able to dynamically create variables, which is more flexible than resource files

2.1.2 Test data section

A test file contains:

Section	Description
Settings	importing test libraries, resource files and variable files, Documentation, Metadata, Suite Setup/Teardown
Variables	defining variables (global to each test case)
Test cases	creating test cases using keywords
Tasks (new feature)	like test cases. a single file can only contain either test cases or task
Keywords	Creating user keywords

Sometimes Section is also called table ! e.g. variable table

2.1.3 Supported file formats

html, tsv, txt, robot(recommended), rst

2.1.4 Rules for parsing the data

Ignore data: lots of rules, basically # is used for comment
Escaping: \
Special sequences: \n, \r, \t, \x, \u, \U
Some useful BuiltIn libraries. e.g. Convert To Bytes, ${\n}, ${EMPTY}, ${SPACE}
Multiple line: use … It is useful for keywords, documentation, variables
1
2
@{LIST} 1 2 3 4
... 5 6

2.2 Creating test cases

2.2.1 Test cases syntax

Keywords can be imported from Library, Resource or keyword table
Each test case can have:
- [Documentation]
- [Tags]
- [Setup], [Teardown]
- [Timeout]
- [Template]: data driven test, mentioned in 2.2.7

2.2.2 Arguments

Behaves like python:

Positional arguments
Default value arguments
Variable number of arguments
Named arguments
**kargs

2.2.3 Failures

If a test case fails, rest test cases will still run
We can specify failure reason e.g.
- Should Be True ${result} xx is not true
Continue on failure: 3.2.2

2.2.4 Test case name and documentation

${TEST_NAME} refer to the test name. More in 2.6.4 Built-in variables
The [Documentation] in test case shows in the command line and test reports.

2.2.5 Tagging test cases

–include, –exclude to run test cases

Name	Location	Description
Force Tags	Setting table	Set tags all test cases in this file. If it is in init file, all sub test suite dir are tags as it
Default Tags	Setting table	Set tags for test cases without [Tag]. Cannot use in init file
Tags	Test Case table	set tag for test case. Can override Default Tags. Can also be NONE
BuiltIn keywords		Dynamically set tags: Set Tags, Remove Tags, Fail and Pass Execution keywords,

2.2.6 Test setup and teardown

Test Setup, Teardown can take care of only 1 task:
- Use user defined keywords
- Use builtIn “Run Keywords”
Each test case can have its own [Setup], [Teardown] to override the one in Settings
Detail is in 3.2.1 Execution Flow

2.2.7 Test templates

Convert keyword-driven test cases into data-driven tests.
It’s like I have a method that need to be tested for lots of pairs of different arguments and expected output.

Example

*** Test Cases ***
Template with embedded arguments
    [Template]    The result of ${calculation} should be ${expected}
    1 + 1    2
    1 + 2    3

*** Keywords ***
The result of ${calculation} should be ${expected}
    ${result} =    Calculate    ${calculation}
    Should Be Equal    ${result}     ${expected}

2.2.8 Different test case styles

Keyword-driven: normal
Data-driven: using template
Behavior-driven: using Given, When, Then, And, But
- Given: initial state
- When: operation
- Then: expected result
- And, But: connect keywords

2.3 Creating Tasks

new feature after Robotframework 3.1

2.4 Creating test suites

2.4.1 Test case files

In Setting table, we can have

Documentation
Metadata
Suite Setup
Suit Teardown

2.4.2 Test suite directories

Initialization files:

For specifying test suite related settings
Similar as test case files, but cannot have test case tables and not all settings are supported.
Library, resource, variable, keywords cannot applied to other files !!!

2.4.3 Test suite name and documentation

Test suite name
- name is constructed from fire name
- underscore -> space
- e.g. some_tests.robot -> Some Tests
- Execution order: by file name. Detail in 3.2.1 Execution flow

2.4.4 Free test suite metadata

2.4.5 Suite setup and teardown

Suite setup is executed before any of suite’s test cases or child test suites. Suite setup fails, all testcase fails
Suite teardown is executed after everything, even suite setup fails.
They can be set in init file

2.5 Using test libraries

2.5.1 Importing libraries

Library: used in Settings table
Import Library: used in test cases dynamically

2.5.2 Specifying library to import

System tries to find the library from the module search path including PYTHONPATH

Using relative, absolute path

*** Settings ***
Library    PythonLibrary.py
Library    /absolute/path/JavaLibrary.java
Library    relative/path/PythonDirLib/    possible    arguments
Library    ${RESOURCES}/Example.class

2.5.3 Setting custom name to test library

We can initialize 2 python class using this way. e.g.

1 2	Library xx arg1 WITH NAME TestLib1 Library xx arg2 WITH NAME TestLib2

2.5.4 Standard libraries

2.5.5 External libraries

2.6 Variables

2.6.1 Introduction

&{} dictionary
${} variable
@{} list
%{} environment

2.6.2 Using variables

Naming rules:
- Case insensitive, ignore underscore and space
- Global variables naming rule: upper case
${} variables
- Can contain any object, including numbers, lists, dictionaries, or even custom objects.
- With other string or variable, this variable will convert to its unicode | str e.g.
  1
  2
  3
  ${VAR} is an object
  Method ${VAR} # Method gets an object
  Method Hi ${VAR} # Method gets a string

@{} list

Not a true list argument !!! We should use ${} to parse a list variable as argument, since these are equivalent.
1
2
3
@{USER} robot secret
Login robot secret
Login @{USER}

Accessing list items

1
2
3

@{STRINGS}    one    two    kolme    four
${STRINGS}[2:-1:2]  # Start from 2 until last one, step 2
@{STRINGS}[index] # Old syntax. Do not use it

&{} dictionary
- Not a true dictionary argument !!! We should use ${} to parse a list variable as argument, since these are equivalent.
  1
  2
  3
  &{USER} name=robot password=secret
  Login name=robot password=secret
  Login &{USER}
- Accessing list items
  ${USER.name} or ${USER}[name]
%{} environment variables
- Keyword: Set Environment Variable
- Keyword: Delete Environment Variable

2.6.3 Creating variables

Variable table in test case file or resource file

Cannot create dynamic variables

Syntax can be either

${NAME}    robot
${NAME} =    robot
@{LIST}         one         two
&{DICT}    first=one    second=two

Variable file
- robot --variablefile xx.py test.robot
- Global to all testcases
Command line variables
- robot --variable NAME:robot
- Global to all testcases

BuiltIn keywords for local variables

@{list} =    Create List    1    2
${list} =    Create List    1    2
${dict} =    Create Dictionary    1    2
&{dict} =    Create Dictionary    1    2
${a}    ${b} =    Create List    1    2

Set Test/Suite/Global Variable keywords
- Set Test Variable: visible in this test case
- Set Suite Variable: visible in this test suite (all test cases in this file)
- Set Global Variable: visible in global, like -variable

2.6.4 Built-in variables

Operating-system variables
- ${CURDIR}
- ${TEMPDIR}
- ${EXECDIR}
- ${/}: path separator
- ${:}: path separator
- ${\n}: system line separator
Number variables
- int ${80}
- float ${3.14}
- scientific ${-1e-4}
- binary ${0b1011}
- octal ${0o10}
- hex ${0xff}
Boolean
- case insensitive
- ${true} ${false}
Null
- case insensitive
- ${none} ${null}
Space and empty variables
- Should Be Equal {SPACE * 4} \ \ \ \
- ${EMPTY} @{EMPTY LIST} &{EMPTY DICT}
Automatic variables
- It is like metadata of robot test
  ${TEST NAME} ${SUITE NAME} ${PREV TEST NAME} ${OUTPUT DIR}

2.6.5 Variable priorities and scopes

Priorities

Variable set during test execution
- e.g. Create List or return value
- e.g. Set Test/Suite/Global Variable keywords
- Highest priority
- Cannot affect variables not in its scope
–variable:
- Second priority
- Last one override others
- Override all others
–variablefile:
- Third priority
- Variable in first file override others
Variable table
- Forth priority
- visible to all test case in this file
Imported resource and variable files
- Lowest
Built-in variables
- This is special case. I don’t think it should be discussed here

Scope

Global scope: –variable, –variablefile, Built-in variables, Set Global Variable
Test suite scope
- Not recursive: higher-level test suite are not available in lower-level suites
- Set Suite Variable, Imported resource and variable files, Variable table
Test case scope: return value from keywords

2.6.6 Advanced variable features (interesting)

Extended variable syntax

The expression inside the curly brackets is evaluated as a Python expression

Keyword1    ${OBJECT.eat('Cucumber')}
Keyword2    ${OBJECT.name}
Keyword3    ${DICT[2]}

${string} =    Set Variable    abc
Log    ${string.upper()}

Extended variable syntax assignment
${OBJECT.name} = Set Variable New name

Variables inside variables

${JOHN HOME}    /home/john
${JANE HOME}    /home/jane

${name} =    Get Name
Keyword    ${${name} HOME}

2.7 Creating user keywords

2.7.1 User keyword syntax

Keywords in init file are not visible in other files, neither files in same directory or sub-directory
Can only resource another file to use its keywords
Settings in Keyword:
- [Documentation]
- [Tags]
- [Arguments]
- [Return]
- [Teardown]
- [Timeout]

2.7.2 User keyword name and documentation

2.7.3 User keyword tags

2.7.4 User keyword arguments

Positional
Default values

Variable number of arguments

*** Test Cases ***
Example
	Two Arguments With Defaults    arg2=abcd
*** Keywords ***
One Required And One With Default
    [Arguments]    ${required}    ${optional}=default
Two Arguments With Defaults
    [Arguments]    ${arg1}=default 1    ${arg2}=${VARIABLE}
Required, Default, Varargs
    [Arguments]    ${req}    ${opt}=42    @{others}

Free named arguments: like **kargs in python
[Arguments] ${required} &{kargs}
Named-only arguments with user keywords
- New feature after 3.1
- Specify named argument after @{varargs} or after @{}
- With Varargs
  [Arguments] @{varargs} ${named}
- Without Varargs
  [Arguments] @{} ${named}

2.7.5 Embedding arguments into keyword name

Confused space with sequential word

Select ${city} ${name}
Select Los Angeles Lakers will be confused

Have to use “” to wrap it

*** Keywords ***
Select "${city}" "${team}" from list
    Log    ${city} ${team}
	
Select "Los Angeles" "Lakers" from list

Embedded arguments matching rule

Regular expressions: every argument in the name is replaced with a pattern .*? that basically matches any string.
- Example
  1
  2
  3
  I execute "ls" with "-lh", will match both
  I execute "${cmd}"
  I execute "${cmd}" with "${opts}"
- Solution
  I execute "${cmd:[^"]+}"

Escaping special characters

like }, \

2.7.6 User keyword return values

Using [Return] setting

${a}    ${b}    ${c} =    Return Three Values
@{list} =    Return Three Values
${scalar}    @{rest} =    Return Three Values

Return Three Values
	[Return]    foo    bar    zap

Using special keywords to return
- Return From Keyword
- Return From Keyword If

2.7.7 User keyword teardown

2.8 Resource and variable files

User keywords and variables in test case files and test suite initialization files can only be used in files where they are created

2.8.1 Resource files

Searching path
- Absolute path
- Otherwise
  - Relatively to the directory
  - PYTHONPATH
Resource file structure
- Settings can only have Library, Resource, Variables
  Same keywords name from different resource file
Distinguish by file name without extension
- myresources.Keyword
- common.Keyword

2.8.2 Variable files

Taking variable files into use

Settings

1 2	* Settings * Variables myvariables.py

Command lines

---variablefile myvariables.py

Creating variables in python

VAR = "abcde"
VAR2 = method()
def method():
	return xxxxx

Selecting which variables to include

all variables not start with _ will be include as variables
- even import math as _math def get_area()
Or we can specify __all__ = [‘VARNAME2’, ‘VARNAME2’]. Only variables in __all__ can be used

Getting variables from a special function

def get_variables(arg):
    if arg == 'a':
        variables = {"BAR": 7}
    else:
        variables = {"BAR": 77}
    return variables

robot --variablefile ./varfile.py:a test.robot

Implementing variable file as Python or Java class: weird

Variable file as YAML: weird

2.9.1 Handling keywords with same names

Two kinds of keywords:
1. Library keywords (from standard/external libraries)
2. User keywords (user created/resourced)
Priority:
- Highest - Keywords created in current file
- Second - Resourced
- Third - External library
- Lowest - Standard library
Solution:
- Specifying a keyword explicitly. e.g. Libraryname.Run
- Specifying explicit priority using keyword. e.g. Set Library Search Order

2.9.2 Timeouts

We can set [Timeout] in Settings(default timeout for all test cases), Test Cases, Keywords

2.9.3 For loops

Simple for loop
Nested for loops
- implemente by keyword

Using several loop variables

FOR    ${index}    ${english}    ${finnish}    IN
   ...     1           cat           kissa
   ...     2           dog           koira
   ...     3           horse         hevonen
FOR    ${index}    ${english}    ${finnish}    IN

For-in-range loop:

for i in range(xx, xx):
For-in-enumerate loop

for index, item in enumerate(my_list):
For-in-zip loop

for i, j in zip(list1, list2):
Exiting for loop
- Exit For Loop
  - Exit For Loo If
Continuing for loop
- Continue For Loop
- Continue For Loop If

Removing unnecessary keywords from outputs

Using command line option: –RemoveKeywords FOR

Repeating single keyword

Repeat Keyword 5 Some Keyword arg1 arg2

2.9.4 Conditional execution

Do not use if statement in Test Cases
Run Keyword: can set variable dynamically

If statement:

1 2	Run Keyword If '${status}' == 'good' action Run Keyword Unless '${status}' == 'good' another action

Run Keyword If '${status}' == 'good' action ELSE another action

2.9.5 Parallel execution of keywords

Start Process
Read Process Output

3 Executing test cases

3.1 Basic usage

3.1.1 Starting test execution

3.1.2 Using command line options

Short and long options
- robot --name test.robot
- robot -N test.robot
Setting option values
- robot --include tag
- robot -i tag
- robot --include=tag
- robot -itag
Disabling options accepting no values
- robot --dryrun
- robot --nodryrun
Simple patterns
- pattern: *, ?
- robot --include f??
Tag patterns
- AND, &
  - robot --include xx&yy
  - robot --include xxANDyy
- OR
  - robot --include xxORyy
- NOT
  - robot --include xxNOTyy
- priority: AND > OR > NOT
ROBOT_OPTIONS and REBOT_OPTIONS environment variables
- ROBOT_OPTIONS: default option for test execution
- REBOT_OPTIONS: default option for result post-processing

3.1.3 Test results

Command line output
Generated output files
Return codes
- Return code can be access by ‘echo $?’
- --NoStatusRC: can set return code to be 0
Errors and warnings during execution

3.1.4 Argument files

Extract all options and arguments into an external file

--argumentfile xxxx
--argumentfile STDIN: we are able to read argumentfile from standard input, so we can
- generate_argument.sh | robot --argumentfile STDIN

3.1.5 Getting help and version information

3.1.6 Creating start-up scripts

Modifying Java startup parameters

3.1.7 Debugging problems

command line output
log file
- decrease loglevel, which shows in log.html
- FAIL, WARN, INFO, DEBUG, TRACE
- --loglevel DEBUG:INFO
  add Log, Log Variables in BuiltIn Keyword
Robot Framework bug: set env ROBOT_INTERNAL_TRACES to anything
Enable syslog
Using the Python debugger (pdb): never mind
Using Pycharm debugger. Edit configuration
- Script Path: /Users/xxx/.pyenv/versions/3.7.2/lib/python3.7/site-packages/robot/run.py
- Parameters: we can also add other parameters here like
  - e.g. –loglevel DEBUG /xxx/xxx/xxx.robot

3.2.1 Execution flow

Select test: --test, --suite, --include, --exclude

Suite setup:

if fail, all testcases, child suites won’t execute and will be marked failed as fail
Run before all tests and child suites

Suite teardown:

if fail, all testcase marked failed.
Run even suite setup fail
Run all keywords in it, even some keywords in it failed
Test setup, Test teardown
Keyword teardown, No Keyword setup

Execution order:

by filename
depth-first
if specify several tests, executed in the order they are given
add prefix to specify order
- 01__my_suite.html -> My Suite
- 02__another_suite.html -> Another Suite
randomize: –randomize

Passing execution:

Pass Execution/Pass Execution If
- Pass execution of testcases directly
- Better to set testcase non-critical

3.2.2 Continue on failure

Continue by keywords
- Run Keyword And Ignore Error
- Run Keyword And Expect Error
- Run Keyword And Continue On Failure
Special failures
- by creating exception
Execution continues on teardowns automatically
Execution continues on all data in templates

3.2.3 Stopping test execution gracefully

Pressing Ctrl-C
Using signals: kill INT or kill TERM
Using keywords: Fatal Error or fatal exceptions
Stopping when first test case fails or execution error
- –exitonfailure: if any case fail, execution stop
- –exitonerror: if any error, execution stop
Handling teardowns
- –skipteardownonexit can skip teardown if execution stoped by above

3.3 Task execution

3.4 Post-processing outputs

Integral part of Robot Framework for handling output files
command: rebot

3.5 Configuring execution

3.5.1 Selecting files to parse

--extension robot:text

3.5.2 Selecting test cases

Select robot files to run

--test *test
--suite *test.xx
--include test*
--exclude test*

Re-executing failed test suites

Grab failed test and run them again

robot --rerunfailed output.xml tests
robot --rerunfailedsuites

When no tests match selection

Usually, we are given an error, but we can also run it by

–RunEmptySuite
–ProcessEmptySuite

3.5.3 Setting criticality

All testcase are critical by default
Failed noncritical test case will not affect overall status
- –critical
- –noncritical

3.5.4 Setting metadata

3.5.5 Configuring where to search libraries and other extensions

PYTHONPATH, JYTHONPATH and IRONPYTHONPATH
Using –pythonpath option
Configuring sys.path in program
Java classpath

3.5.6 Setting variables

3.5.7 Dry run

For validating syntax

3.5.8 Randomizing execution order

--randomize <what>[:<seed>]

3.5.9 Programmatic modification of test data

If the provided built-in features to modify test data before execution are not enough, Robot Framework 2.9 and newer makes it possible to do custom modifications programmatically

A python class extends SuiteVisitor
--prerebotmodifier xxx.py:arg1:arg2

3.5.10 Controlling console output

Console output type
Console width
Console colors

3.5.11 Setting listeners

For monitoring the test execution

–listener xxx
Listener interface

3.6 Created outputs

4 Extending Robot Framework

4.1 Creating test libraries

4.1.1 Introduction

4.1.2 Creating test library class or module

Test library names

When libraring a library, if classname == filename:
    Library    filename
else:
    Library    filename.classname

Test library scope

ROBOT_LIBRARY_SCOPE: TEST CASE; TEST SUITE; GLOBAL
ROBOT_LIBRARY_SCOPE = 'TEST SUITE'
Global and Test are for sharing state between tests

Specifying library version

ROBOT_LIBRARY_VERSION = '0.1'
or __version__ = 0.1
They will be written into the syslog

4.1.3 Creating static keywords

Create keyword in python file !

ignore methods begin with _

Keyword names

ignore space, underscore, case insensitive
custom keyword name by @keyword(‘Login As User’)

Keyword tags

@keyword(tags=['tag1', 'tag2'])

Argument types

Manually: int(port)
Decorator
@keyword(types={'count': int, 'case_insensitive': bool})
Implicit argument type
def example_keyword(count=-1, case_insensitive=True)

If we specify as 2 and 3, argument type will be automatically converted.

Embedding arguments into keyword names

Awesome!!

1 2	@keyword('Add ${quantity:\d+} copies of ${item} to cart') def add_copies_to_cart(quantity, item):

4.1.4 Communicating with Robot Framework

Reporting keyword status

Only AssertionError, Exception, RuntimeError shows exception messages
Self created Error should specify
- ROBOT_SUPPRESS_NAME = True
Stopping/Continue test execution in exception class
- ROBOT_EXIT_ON_FAILURE = TRUE
- ROBOT_CONTINUE_ON_FAILURE = TRUE
Logging information into console
1. ERROR and WARN will be written in console
  print('*WARN*', ...)
2. stderr will be written in console
  - print('aaa', file=sys.stderr)
  - sys.__stdout__.write('Got arg %s\n' % arg)
3. use robot logging API
  1
  2
  3
  from robot.api import logger
  logger.console('Got arg %s' % arg)
  logger.info('Got arg %s' % arg, also_console=True)
4. use python logging API
  - import logging
  - logging.info('')

Communication when using threads

Supposed only main thread talk with robotframework
BackgroundLogger helps with not-main thread

4.1.5 Distributing test libraries

4.1.6 Dynamic library API

@keyword
get_keyword_names(self):
    return [name for name in dir(self) if hasattr(getattr(self, name), 'robot_name')]
run_keyword(self, name, args)
get_keyword_arguments 
get_keyword_documentation 
get_keyword_tags
get_keyword_types

4.1.7 Hybrid library API

Getting keyword names
Running keywords
Getting keyword arguments and documentation

4.1.8 Using Robot Framework’s internal modules

Available APIs: https://robot-framework.readthedocs.io/en/v3.1.1/
Using BuiltIn library

4.1.9 Extending existing test libraries

Modifying original source code: don’t do this
Using inheritance: cool
Using other libraries directly

Getting active library instance from Robot Framework

from robot.libraries.BuiltIn import BuiltIn

def title_should_start_with(expected):
    seleniumlib = BuiltIn().get_library_instance('SeleniumLibrary')
    title = seleniumlib.get_title()
    if not title.startswith(expected):
        raise AssertionError("Title '%s' did not start with '%s'"
                             % (title, expected))

Libraries using dynamic or hybrid API

4.2. Remote library

4.2.1 Introduction

Allows distributed testing: have test library on different machine

4.2.2 Putting Remote library to use

Importing Remote library
Library Remote http://10.0.0.2/example 1 minute WITH NAME Example3
Starting and stopping remote servers
- start: by Process or SSH library
- stop: remote server should supply Stop Remote Server keyword
```
* ctrl-c on server
* kill
```

4.2.3 Supported argument and return value types

XML-RPC protocol does not support all possible object types

4.2.4 Remote protocol

Executing remote keywords: it is just make a server

Server side: RobotRemoteServer(Library(), host=xx, port=xx)
Client side: Library Remote http://10.0.0.2/example 1 minute WITH NAME Example3
So Robot framework is able to use it directly

4.3.1 Taking listeners into use

robot --listener listener.py:arg1:arg2 tests.robot

4.3.2 Listener interface versions

ROBOT_LISTENER_API_VERSION = 2: only gets information about the execution but cannot directly affect it
ROBOT_LISTENER_API_VERSION = 3: gets data and is able to alter execution

4.3.3 Listener interface methods

During the test execution different listener methods are called when test suites, test cases and keywords start and end
Listener version 2
- start_suite, end_suite, start_test, end_test, start_keyword, end_keyword, log_message, message, output_file, log_file, report_file, debug_file
Listener version 3
- library_import, resource_import, variables_import, xunit_file, close

4.3.4 Listeners logging

4.3.5 Listener examples

4.3.6 Test libraries as listeners

It is like combine library with listener

class PythonLibraryAsListenerItself(object):
    ROBOT_LIBRARY_SCOPE = 'TEST SUITE'
    ROBOT_LISTENER_API_VERSION = 2

    def __init__(self):
        self.ROBOT_LIBRARY_LISTENER = self

    def _end_suite(self, name, attrs):
        print 'Suite %s (%s) ending.' % (name, attrs['id'])

    # actual library code here ...