Authoring Layers

What is a Layer?

Prefab reverse engineers an interface by examining pixels and constructing a tree of identified elements. Prefab Layers breaks down this process into a series of tree transformations. A Layer is a python script that performs a set of transformations needed for your particular application. In your application, you can implement layers of your own, and you can import and parameterize existing layers. Each layer is designed to be relatively simple, and to achieve complex behaviors layers are composed into a chain where the tree output by one layer is the input to the next.

Check out the template_layer.py file for a full example.

Implementation

The easiest way to create a python script that implements either or both of the two methods:

def interpret(interpret_data)
def generalize_annotations(annotation_data)

interpret(interpret_data)

This method performs a set of tree transformations on a reverse engineered interface hierarchy. Implement this method to interpret a tree that results from the execution of layers preceding this layer.

interpret_data.tree:
This is a read-only object that represents the current interpretation of a screenshot.

Each node has a dictionary of tags:
tagvalue = tree['tagname']

A bounding rectangle:
width = tree.width
height = tree.height
top = tree.top
left = tree.left

A list of children that are contained spatially within the node's boundary: 
children = tree.get_children()
	
interpret_data.runtime_storage:
This is a dict where you can store any information generalized from your annotations. See the generalize_annotations method.

interpret_data.tree_transformer:
This is an object that will queue transformations that will be executed after this layer completes.
enqueue_set_tag(node, tagname, tagvalue)
enqueue_set_ancestor(node, ancestor_node)
enqueue_delete(node)
newnode = create_node(x, y, width, height, tags = {} )

Example (tagging leaf elements)

def interpret(interpret_data):
	find_leaves_and_tag_as_text(interpret_data, interpret_data.tree)
	
	
def find_leaves_and_tag_as_text(interpret_data, currnode):
	if len( currnode.get_children() ) == 0:
		interpret_data.enqueue_set_tag(currnode, 'is_text', True)
		
	for child in currnode.get_children():
		find_leaves_and_tag_as_text(interpret_data, child)

Example (matching stored path descriptors to tag nodes with annotation data)

def interpret(interpret_data):
	recursively_apply_corrections(interpret_data, interpret_data.tree)

def recursively_apply_corrections(interpret_data, currnode):
	'''
	This method recursively tags nodes
	that match path descriptors stored in 
	the runtime_storage
	'''	
	transformer = interpret_data.tree_transformer
	from path_utils import get_path 
	#import the method to retrieve path descriptors
	
	path = get_path(currnode, interpret_data.tree)
	if path in interpret_data.runtime_storage:
		#check if there is a corresponding path descriptor stored
		data = interpret_data.runtime_storage[path]
		for key in data:
			transformer.enqueue_set_tag(currnode, key, data[key])
	
	for child in currnode.get_children():
		recursively_apply_corrections(interpret_data, child)

generalize_annotations(args)

This method is used to process any annotations that are stored in the annotation libraries given to the layer. The typical use of this method is to compute something using the annotations and store the results in the layer's runtime storage. The layer can then use these results in its interpret method.

annotation_data.annotated_nodes:
This argument is a list of annotations.
An annotation consists of a tree node and 
the associated data. The associated data is 
a dict containing key/value pairs. It also
It also contains the root node to the entire 
tree containing this annotation. See the
interpret method for the specifications of
the tree node object.
	
annotation = annotated_nodes[0]
is_text = annotation.data['is_text']
node = annotation.node
root = annotation.root
	
annotation_data.runtime_storage:
This is a dict where you can store any information
generalized from your annotations, such as path
descriptors or learned classifiers. This data will be
passed into the interpret method, and it will also be
serialized to disk so that the layer does not need to
recompute everything if Prefab exits and restarts.

annotation = annotated_nodes[0]
path = get_path( annotation.node )
runtime_storage[path] = annotation.data

annotation_data.parameters:
This is a dict of developer-provided parameters
the layer can access to parameterize its behavior.
The same parameters also available in the interpret method.
For example, if you are translating the language of inter
the layer might accept the specific language as a parameter.
translation_language = parameters['language']

Example (computing path descriptors and storing annotation data for runtime)

def generalize_annotations(annotation_data):
	runtime_storage = annotation_data.runtime_storage
	runtime_storage.clear()
	annotated_nodes = annotation_data.annotated_nodes
	#importing the script for computing path descriptors
	from path_utils import get_path 
	for anode in annotated_nodes:
		path = get_path( anode.node, anode.root)
		runtime_storage[path] = anode.data