Introduction to Algorithms
Week 11: Notes

Some of this week's topics are covered in Problem Solving with Algorithms:

Here are some additional notes on the following topics:

expression trees

In the last lecture we learned about prefix, infix and postfix syntaxes for arithmetic expressions.

We may store expressions using an expression tree, which is a form of abstract syntax tree. An expression tree reflects an expression’s hierarchical structure. For example, here is an expression tree for the infix expression ((3 + 4) * (2 + 5)):

tree
Note that this expression tree also corresponds to the prefix expression * + 3 4 + 2 5, or the postfix expression 3 4 + 2 5 + *. Equivalent expressions in infix, prefix or postfix form will always have the same tree! That’s because this tree reflects the abstract syntax of the expression, which is independent of concrete syntax which defines how expressions are written as strings of symbols.

We can easily store expression trees using objects in Python. Here is a class for arithmetic expressions:

class Expr:
  def __init__(self, op, left, right):
    self.op, self.left, self.right = op, left, right

This may remind you of the class for binary search trees that we saw a few weeks ago.

We can build the expression tree in the picture above as follows:

l = Expr('+', 3, 4)

r = Expr('+', 2, 5)
top = Expr('*', l, r)

If we have an expression tree in memory, we may wish to evaluate it. To evaluate an expression means to determine its value. For example, when we evaluate the infix expression (2 + (3 * 4)), we get the value 14.

We first write a function that can apply an operator to two numbers:

def evalOp(op, x, y):
  if op == '+':
    return x + y
  elif op == '-':
    return x - y
  elif op == '*':
    return x * y
  elif op == '/':
    return x // y

  else:

    assert False, 'unknown operator'

Now we can evaluate an expression tree with a straightforward recursive function:

def eval(e):
  if not isinstance(e, Expr):
    return e
    
  l = eval(e.left)
  r = eval(e.right)
  return evalOp(e.op, l, r)

Given an expression tree, it’s not hard to convert it to a string representation in either infix, prefix or postfix form. For example, we can generate a prefix expression as follows:

def toPrefix(e):
  if not isinstance(e, Expr):
    return str(e)
  
  l = toInfix(e.left)
  r = toInfix(e.right)
  return f'{e.op} {l} {r}'

Or we may generate an infix expression from a tree:

def toInfix(e):
  if not isinstance(e, Expr):
    return str(e)
  
  l = toInfix(e.left)
  r = toInfix(e.right)
  return f'({l} {e.op} {r})'

Notice that we fully parenthesize each subexpression in a generated infix expression. If we did not do this, then we might generate a string such as '3 + 4 * 5 + 2' when we really meant '((3 + 4) * (5 + 2))'. In a prefix expression, parentheses are unnecessary since prefix expressions are unambiguous.

grammars for arithmetic expressions

Our discussion of expression syntax to this point has been somewhat informal. We will now formally define the syntax of arithmetic expressions, which will help us in the next section. For simplicity, we will assume that

We can define the syntax of infix arithmetic expressions using the following context-free grammar:

digit → '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'
op → '+' | '-' | '*' | '/'
expr → digit | '(' expr op expr ')'

Context-free grammars are commonly used to define programming languages. A grammar contains non-terminal symbols (such as digit, op, expr) and terminal symbols (such as '0', '1', '2', …, '+', '-', '*', '/'), which are characters. A grammar contains a set of production rules that define how each non-terminal symbol can be constructed from other symbols. These rules collectively define which strings of terminal symbols are syntactically valid.

For example, ((4 + 5) * 9) is a valid expression in the syntax defined by this grammar, because it can be constructed from the top-level non-terminal expr by successively replacing symbols using the production rules:

expr
→ ( expr op expr )
→ ( expr op digit )
→ (expr op 9)
→ (expr * 9)
→ ((expr op expr) * 9)
→ ((expr op digit) * 9)
→ ((expr op 5) * 9)
→ ((expr + 5) * 9)
→ ((digit + 5) * 9)
→ ((4 + 5) * 9)

You will learn more about context-free grammars in more advanced courses, but it’s worth taking this first look at them now because they are so commonly used to specify languages.

To modify our grammar to specify the syntax of prefix expressions, we need change only the last production rule above:

expr → digit | op expr expr

Or, for postfix expressions:

expr → digit | expr expr op

Note again that the prefix and postfix expression languages have no parentheses.

parsing expressions

We would now like to parse arithmetic expressions defined by the grammar above. Given a string containing a prefix, infix, or postfix expression, we'd like to determine the expression's structure and (a) generate an expression tree or (b) evaluate the expression directly.

Let's first write a function that can parse an arithmetic expression in prefix notation and generate an expression tree. We will use a Python iterator to read each character of a string in turn. (If you are unfamiliar with iterators, read the Programming 1 lecture notes on that topic.)

def _parsePrefix(it):
  c = next(it)
  if c.isdigit():  # base case
    return int(c)
    
  left = _parsePrefix(it)
  right = _parsePrefix(it)
  return Expr(c, left, right)

def parsePrefix(s):
  return _parsePrefix(iter(s))

Notice that the structure of the function _parsePrefix follows the grammar rule for prefix expressions:

expr → digit | op expr expr

If we wish to evaluate a prefix expression directly rather than generating a tree, we can do so by changing the last line of _parsePrefix above to

return evalOp(c, left, right)

Parsing an infix expression is not much more difficult. Recall the grammar rule for producing infix expressions:

expr → digit | '(' expr op expr ')'

We now write a recursive function that mirrors this rule:

def _parseInfix(it):
  c = next(it)
  if c.isdigit():
    return int(c)
  
  assert c == '('
  left = _parseInfix(it)
  c = next(it)
  right = _parseInfix(it)
  assert next(it) == ')'
  
  return Expr(c, left, right)

def parseInfix(s):
  return _parseInfix(iter(s))

As with prefix expressions, we can make a trivial change to this function to evaluate an infix expression directly rather than generating a tree:

return evalOp(c, left, right)

To parse a postfix expression, we cannot easily use recursion. Instead, the most straightforward approach is to use a stack. As we read a postfix expression from left to right, each time we see a number we push it onto the stack. When we see an operator, we pop two numbers from the stack, apply the operator and then push the result back onto the stack. For example, in evaluating the postfix expression 4 5 + 2 1 + * we would perform the following operations:

When we finish reading the expression, if it is valid then the stack will contain a single number, which is the result of our expression evaluation.

To generate a expression tree from a postfix expression, we can use a stack of pointers to expression trees. When we read an operand, we pop two subtrees from the stack, combine them into an Expr object, then push it back on the stack.

representing graphs

A graph consists of a set of vertices (often called nodes in computer science) and edges. Each edge connects two vertices. In a undirected graph, edges have no direction: two vertices are either connected by an edge or they are not. In a directed graph, each edge has a direction: it points from one vertex to another.

Graphs are fundamental in computer science. Many problems can be expressed in terms of graphs, and we can answer lots of interesting questions using various graph algorithms.

We can represent graphs in a computer program in either of two ways. Suppose that a graph has N vertices numbered 0 through (N - 1). We can represent the graph using adjacency-matrix representation as a two-dimensional array A of booleans, with dimensions N x N. In this representation, A[i][j] is true if and only there is an edge from vertex i to vertex j. (If the graph is undirected, then A[i][j] = A[j][i] for all i and j, and so we may optionally save space by storing only the matrix elements above the main diagonal, i.e. elements for which i < j.)

In Python, we may store a graph in adjacency-matrix representation using a list of lists of booleans, which is essentially a two-dimensional array.

Alternatively, we can use adjacency-list representation, in which for each vertex u we store a list of its adjacent vertices.

If vertices are numbered 0 through (N - 1), then in Python we can store a graph in adjacency-list representation using a list of lists of integers. For a graph g, g[i] is a list of the integer ids of its adjacent vertices. Or, if vertices have identifiers such as strings, then we can store a graph in adjacency-list representation using a dictionary that maps each vertex id to a list of adjacent vertex ids.

When we store an undirected graph in adjacency-list representation, then if there is an edge from u to v we store u in v’s adjacency list and also store v in u’s adjacency list.

Adjacency-matrix representation is more compact if a graph is dense. It allows us to immediately tell whether two vertices are connected, but it may take O(V) time to enumerate a given vertex’s edges. On the other hand, adjacency-list representation is more compact for sparse graphs. With this representation, if a vertex has e edges then we can enumerate them in time O(e), but determining whether two vertices are connected may take time O(e), where e is the number of edges of either vertex. Thus, the running time of some algorithms may differ depending on which representation we use.

In this course we will usually represent graphs using adjacency-list representation.

a graph of Europe

Here is an undirected graph where each node is a country in the European Union. There is an edge between every two neighboring countries, and also between any two countries connected by a car ferry (in this case appearing as a dashed line). Thus, two countries are connected by an edge if it is possible to drive from one to the other. We will use this graph to illustrate various algorithms in the following sections.

map


reading a graph

We will now write a function read_graph that can read an undirected graph from a text representation in which nodes are named by strings. Each input line should contain a node name, followed by a colon and a space-separated list of nodes that are connected to that node. For example:

austria: germany czech slovakia hungary slovenia italy
belgium: netherlands germany luxembourg france
bulgaria: romania greece
...

read_graph returns a graph in adjacency-list representation as a Python dictionary. Here is the implementation:

# read an undirected graph
def read_graph(filename):
    g = defaultdict(list)
    with open(filename) as f:
        for line in f:
            names = line.split()
            first = names[0][0:-1]  # remove trailing ':'
            for name in names[1:]:
                g[first].append(name)
                g[name].append(first)
    return g

Here is a text file europe.txt containing the graph of Europe depicted above. You can read it using read_graph:

europe = read_graph('europe.txt')

depth-first search

In this course we will study two fundamental algorithms that can search a graph, i.e. explore it by visiting its nodes in some order.

These two algorithms (and some other graph search algorithms as well) have some common elements. As they run, at each moment in time the graph’s vertices fall into three sets:

At the beginning all vertices are undiscovered. A vertex joins the frontier when the algorithm first sees it. After the algorithm has followed all of the vertex's edges, the vertex becomes explored. By convention, when we draw pictures illustrating graph algorithms we draw vertices as either white (undiscovered), gray (on the frontier) or black (explored).

Note that these algorithms will not usually mark vertices as belonging to one of these three states explicitly. Instead, these states are concepts that help us understand how these algorithms work.

A depth-first search is like exploring a maze by walking through it. Each time we hit a dead end, we walk back to the previous intersection, and try the next unexplored path from that intersection. If we have already explored all paths from the intersection, we walk back to the intersection before that, and so on.

It is easy to implement a depth-first search using a recursive function. In fact we have already done so in this course! For example, we wrote a function to add up all values in a binary tree, like this:

def sum(n):
    if n == None:
        return 0
    return sum(n.left) + n.val + sum(n.right)

This function actually visits all tree nodes using a depth-first tree search.

For a depth-first search on arbitrary graphs which may not be trees, we must avoid walking through a cycle in an infinite loop. To accomplish this, when we first visit a vertex we mark it as visited. In other words, all vertices in the frontier and explored sets are considered to be visited. Whenever we follow an edge, if it leads to a visited vertex then we ignore it.

Here is a picture of a depth-first search in progress on our Europe graph, starting from Austria:

map
As the depth-first search progresses it traverses a depth-first tree that spans the original graph. If we like, we may store this tree in memory as an product of the depth-first search. Here is a depth-first tree for the Europe graph:

map
Here is Python code that implements a depth-first search using a nested recursive function. It takes a graph g in adjacency-list representation and a node start where the search should begin:

# recursive depth-first search
def dfs(g, start):

    def visit(node):
        if node in visited:
            return
        print('visiting ' + node)
        visited.add(node)
        for n in g[node]:
            visit(n)

    visited = set()
    visit(start)

This function merely prints out the visited nodes. But we can modify it to perform useful tasks. For example, to determine whether a graph is connected, we can run a depth-first search and then check whether the search visited every node. As we will see in later lectures and in Programming II, a depth-first search is also a useful building block for other graph algorithms: determining whether a graph is cyclic, topological sorting, discovering strongly connnected components and so on.

A depth-first search does a constant amount of work for each vertex (making a recursive call) and for each edge (following the edge and checking whether the vertex it points to has been visited). So it runs in time O(V + E), where V and E are the numbers of vertices and edges in the graph.

tutorial solutions

Here are the solutions we saw to a couple of exercises in the tutorial:

Tree Walk

Write a procedure that prints out all values in a binary search tree in increasing order. 

class Node:
    def __init__(self, left, val, right):
        self.left, self.val, self.right = left, val, right

def printAll(n):
    if n != None:
        printAll(n.left)
        print(n.val)
        printAll(n.right)

Adjacency Matrix to Adjacency List

Write a function that takes a undirected graph in adjacency matrix representation, and returns the same graph in adjacency list representation. Assume that the graph's vertices are numbered from 0 to (N – 1).

def matrixToList(m):
    n = len(m)
    g = [[] for _ in range(n)]
    for i in range(n):
        for j in range(n):
            if m[i][j]:
                g[i].append(j)
    return g

Note that we could replace the double for loop above with a list comprehension. I will leave that as an additional exercise for you.