Introduction to Algorithms, 2020-1
Week 12: Notes

Some of this week's topics are covered in Problem Solving with Algorithms:

Here are some additional notes.

a graph of Europe

Here is an undirected graph where each node is a country in the European Union. There is an edge between every two neighboring countries, and also between any two countries connected by a car ferry (in this case appearing as a dashed line). Thus, two countries are connected by an edge if it is possible to drive from one to the other. We will use this graph to illustrate various algorithms in the following sections.

map
depth-first search

In this course we will study two fundamental algorithms that can search a graph, i.e. explore it by visiting its nodes in some order.

These two algorithms (and some other graph search algorithms as well) have some common elements. As they run, at each moment in time the graph’s vertices fall into three sets:

At the beginning all vertices are undiscovered. A vertex joins the frontier when the algorithm first sees it. After the algorithm has followed all of the vertex's edges, the vertex becomes explored. By convention, when we draw pictures illustrating graph algorithms we draw vertices as either white (undiscovered), gray (on the frontier) or black (explored).

Note that these algorithms will not usually mark vertices as belonging to one of these three states explicitly. Instead, these states are concepts that help us understand how these algorithms work.

A depth-first search is like exploring a maze by walking through it. Each time we hit a dead end, we walk back to the previous intersection, and try the next unexplored path from that intersection. If we have already explored all paths from the intersection, we walk back to the intersection before that, and so on.

It is easy to implement a depth-first search using a recursive function. In fact we have already done so in this course! For example, we wrote a function to add up all values in a binary tree, like this:

def sum(n):
    if n == None:
        return 0
    return sum(n.left) + n.val + sum(n.right)

This function actually visits all tree nodes using a depth-first tree search.

For a depth-first search on arbitrary graphs which may not be trees, we must avoid walking through a cycle in an infinite loop. To accomplish this, when we first visit a vertex we mark it as visited. In other words, all vertices in the frontier and explored sets are considered to be visited. Whenever we follow an edge, if it leads to a visited vertex then we ignore it.

Here is a picture of a depth-first search in progress on our Europe graph, starting from Austria:

map
As the depth-first search progresses it traverses a depth-first tree that spans the original graph. If we like, we may store this tree in memory as an product of the depth-first search. Here is a depth-first tree for the Europe graph:

map
Here is Python code that implements a depth-first search using a nested recursive function. It takes a graph g in adjacency-list representation and a node start where the search should begin. It assumes that vertices are numbered from 0 to (N – 1), where N is the total number of vertices.

# recursive depth-first search
def dfs(graph, start):
    def visit(v):
        print('visiting', v)
        visited[v] = True
        for w in graph[v]:
            if not visited[w]:
                visit(w)

    visited = [False] * len(graph)
    visit(start)

This function merely prints out the visited nodes. But we can modify it to perform useful tasks. For example, to determine whether a graph is connected, we can run a depth-first search and then check whether the search visited every node. As we will see in later lectures and in Programming II, a depth-first search is also a useful building block for other graph algorithms: determining whether a graph is cyclic, topological sorting, discovering strongly connnected components and so on.

A depth-first search does a constant amount of work for each vertex (making a recursive call) and for each edge (following the edge and checking whether the vertex it points to has been visited). So it runs in time O(V + E), where V and E are the numbers of vertices and edges in the graph.

breadth-first search

Starting from some node N, a breadth-first search first visits nodes adjacent to N, i.e. nodes of distance 1 from N. It then visits nodes of distance 2, and so on. In this way it can determine the shortest distance from N to every other node in the graph.

We can implement a breadth-first search using a queue. Just like with depth-first graph search, we must remember all nodes that we have visited to avoid walking in circles. We begin by adding the start node to the queue and marking it as visited. In a loop, we repeatedly remove nodes from the queue. Each time we remove an node, we mark all of its adjacent unvisited nodes as visited and add them to the queue. The algorithm terminates once the queue is empty, at which point we will have visited all reachable nodes.

The queue represents the frontier. When we remove a node from the queue, it moves to the explored set. Just like with depth-first graph search, the visited nodes are the frontier nodes and the nodes in the explored set.

As the algorithm runs, all nodes in the queue are at approximately the same distance from the start node. To be more precise, at every moment in time there is some value d such that all nodes in the queue are at distance d or (d + 1) from the start node.

Let's revisit the Europe graph that we saw in the last lecture. Here is a breadth-first search in progress, starting from Austria:

map
To implement a breadth-first search in Python, we need a queue data structure. Let's suppse that we have a Queue class with operations enqueue() and dequeue(). We can implement this class using a linked list or a circular array, as we' ve seen in earlier lectures. Alternatively, if we know that the graph is small or we are just experimenting, we can implement a Queue using a simple Python list:

class Queue:
    def __init__(self):
        self.q = []
       
    def enqueue(self, x):
        self.q.append(x)
    
    def dequeue(self):
        return self.q.pop(0)    # O(N) !
        
    def is_empty(self):
        return len(self.q) == 0

Beware, however, that the dequeue() algorithm here runs in O(N), so this queue will perform poorly if N is not small.

Here is a Python function bfs() that performs a breadth-first search. It takes a graph in adjacency-list representation, plus a start vertex:

# breadth-first search
def bfs(graph, start):
    visited = [False] * len(graph)
    q = Queue()
    
    visited[start] = True
    q.enqueue(start)
    
    while not q.is_empty():
        v = q.dequeue()
        print('visiting', v)
        
        for w in graph[v]:
            if not visited[w]:
                visited[w] = True
                q.enqueue(w)

Note that we must mark nodes as visited when we add them to the queue, not when we remove them. (If we marked them as visited only when removing them, then our algorithm could add the same node to the queue more than once.)

Like a depth-first search, a breadth-first search does a constant amount of work for each vertex and edge, so it also runs in time O(V + E).

non-recursive depth-first search

Suppose that we replace the queue in our preceding breadth-first search function with a stack. The function will now perform a depth-first search!

# iterative depth-first search
def dfs_iter(graph, start):
    visited = [False] * len(graph)
    s = Stack()
    
    visited[start] = True
    s.push(start)
    
    while not s.is_empty():
        v = s.pop()
        print('visiting', v)
        
        for w in graph[v]:
            if not visited[w]:
                visited[w] = True
                s.push(w)

Specifically, this is a non-recursive depth-first search, or a depth-first search with an explicit stack.

This shows that there is a close relationship between stacks and depth-first search. Specifically, a stack is a LIFO (last in first out) data structure. And when we perform a depth-first search, the last frontier node we discover is the first that we will expand by following its edges. Similarly, a queue is a FIFO (first in first out) data structure, and in a breadth-first search the first frontier node we discover is the first that we will expand.

It is sometimes wise to implement a depth-first search non-recursively in a situation where the search may be very deep. This will avoid running out of call stack space, which is fixed on most operating systems.

tutorial exercises

Here are a couple of exercises that we solved in the tutorial:

# Return the shortest distance from (start) to (goal)
def distance(graph, start, goal):
    if start == goal:
        return 0
        
    visited = [False] * len(graph)
    q = Queue()
    
    visited[start] = True
    q.enqueue(  (start, 0)  )
    
    while not q.is_empty():
        v, dist = q.dequeue()
        
        for w in graph[v]:
            if not visited[w]:
                if w == goal:
                    return dist + 1
                    
                visited[w] = True
                q.enqueue(  (w, dist + 1)   )
    
    return -1

# Return a shortest path from (start) to (goal), or None
# if there is none.
def shortest_path(graph, start, goal):
    visited = [False] * len(graph)
    
    # Record the parent of each vertex, i.e. the vertex through
    # which we discovered it
    parent = [None] * len(graph)
    
    q = Queue()
    
    visited[start] = True
    parent[start] = None
    q.enqueue(start)
    
    while not q.is_empty():
        v = q.dequeue()
        
        for w in graph[v]:
            if not visited[w]:
                parent[w] = v
                if w == goal:
                    p = w
                    path = []
                    while p != None:
                        path.append(p)
                        p = parent[p]
                    return path[::-1]
                
                visited[w] = True
                q.enqueue(w)