Introduction to Algorithms
Week 10: Notes

Some of this week's topics are covered in Problem Solving with Algorithms. See sections 4.9 Infix, Prefix and Postfix Expressions, 7.8. Priority Queues with Binary Heaps, 7.9. Binary Heap Operations, 7.10. Binary Heap Implementation.

Here are some additional notes.

priority queues

In recent lectures we learned about stacks and queues, which are abstract data types that we can implement in various ways, such as using an array or a linked list.

A priority queue is another abstract data type. At a minimum, a priority queue might provide the following methods:

q.add(value): Add a value fo a priority queue.
q.isEmpty(): Return true if the queue is empty.
q.removeLargest(): Remove the largest value from a priority queue and return it.

A priority queue differs from a stack and an ordinary queue in the order in which elements are removed. A stack is last in first out: the pop function removes the element that was added most recently. An ordinary queue is first in first out: the dequeue function removes the element that was added least recently. In a priority queue, the removeLargest method removes the element with the largest value.

The interface above describes a max-queue, in which we can efficiently remove the smallest value. Alternatively we can build a min-queue, which has a removeSmallest method that removes the largest value; this is more convenient for some applications. Generally any data structure that implements a max-queue can be trivially modified to produce a min-queue, by changing the direction of element comparisons.

In theory we could implement a priority queue using a binary search tree. If we did so and the tree was balanced, then add and removeLargest would run in time O(log N), where N is the number of elements in the queue. But there are more efficient data structures for implementing priority queues, such as binary heaps, to be discussed next.

binary heaps

A binary heap is a binary tree that satisfies two properties.

If a node with value p has a child with value c, then p ≥ c.
All levels of the tree are complete except possibly the last level, which may be missing some nodes on the right side.

For example, here is a binary heap:

tree

This heap looks like a complete binary tree with three nodes missing on the right in the last level.

The height of a binary heap with N nodes is ⌊log₂(N)⌋ = O(log N). In other words, a binary heap is always balanced.

Typically we don’t store a binary heap using dynamically allocated nodes and pointers. Instead, we use an array, which is possible because of the shape of the binary heap tree structure. The binary heap above can be stored in an array a like this:

Notice that the array values are in the same order in which they appear in the tree, reading across tree levels from top to bottom and left to right.

In Python, we can store a heap elements in an ordinary Python list (which, as we know, is actually a dynamically sizeable array):

class BinaryHeap:
    def __init__(self):
        self.a = []    # list will hold heap elements

Here is a tree showing the indices at which heap nodes are stored:

tree

From this tree we can see the following patterns. If a heap node N has index i, then

N’s left child (if any) has index 2i + 1
N’s right child (if any) has index 2i + 2
N’s parent (if any) has index (i - 1) // 2

So we can easily move between related tree nodes by performing index arithmetic:

def left(i):
    return (2 * i + 1)
    
def right(i):
    return (2 * i + 2)
    
def parent(i):
    return ((i - 1) // 2)

heap operations

We will now describe operations that will let us use a heap as a priority queue.

Suppose that a heap structure satisfies the heap properties, except that one node has a value v which is larger than its parent. An operation called up heap can move v upward to restore the heap properties. Suppose that v’s parent has value v₁. That parent may have a second child with value v₂. We begin by swapping v and its parent v₁. Now v’s children are v₁ and v₂ (if present). We know that v > v₁. If v₂ is present then v₁ > v₂, so v > v₂ by transitivity. Thus the heap properties have been restored for the node that now contains v. And now we continue this process, swapping v upward in the heap until it reaches a position where it is not larger than its parent, or until it reaches the root of the tree.

Now suppose that a heap structure satisfies the heap properties, except that one node N has a value v which is smaller than one or both of its children. We can restore the heap properties by performing a down heap operation, in which the value v moves downward in the tree to an acceptable position. Let v₁ be the value of the largest of N’s children. We swap v with v₁. Now N’s value is v₁, which restores the heap property for this node, since v₁ > v and v₁ is also greater than or equal to the other child node (if there is one). We then continue this process, swapping v downward as many times as necessary until v reaches a point where it has no larger children. The process is guaranteed to terminate successfully, since if v eventually descends to a leaf node there will be no children and the condition will be satisfied.

We can use the up heap and down heap operations to implement priority queue operations.

We first consider inserting a value v into a heap. To do this, we first add v to the end of the heap array, expanding the array size by 1. Now v is in a new leaf node at the end of the last heap level. We next perform an up heap operation on the value v, which will bring it upward to a valid position. An insert operation will always run in time O(log N).

Now we consider removing the largest value from a max-heap. Since the largest value in such a heap is always in the root node, we must place some other value there. So we take the value at the end of the heap array and place it in the root, decreasing the array size by 1 as we do so. We now perform a down heap operation on this value, which will lower it to a valid position in the tree. If there are N values in the heap, the tree height is ⌊log₂(N)⌋, so this process is guaranteed to run in O(log N) time.

Here is removeLargest in Python:

    def removeLargest(self):
        a = self.a
        if len(a) == 1:
            return a.pop()
        x = a[0]
        a[0] = a.pop()
        self.downHeap(0)
        return x

heapsort

We’ve already seen several sorting algorithms in this course: bubble sort, insertion sort, merge sort and quicksort. We can use a binary heap to implement another sorting algorithm: heapsort. The essence of heapsort is simple: given a array of numbers to sort, we put them into a heap, and then we pull out the numbers one by one in sorted order.

Let’s examine these steps in a bit more detail. For heapsort, we want a max-heap, which lets us remove the largest element by calling a method removeLargest. Given an array of N values, we first need to heapify it, i.e. turn it into a heap. We could simply perform N insert operations at a total cost of O(N log N), but actually there is a way to heapify N values in time O(N); see the section Binary Heap implementation in the Problem Solving text for details.

Once we have heapified the array, sorting it is easy: we repeatedly call removeLargest to remove the largest element, which we place at the end of the array:

def heapsort(a):
    heap = BinaryHeap()
    heap.buildHeap(a)

    for i in range(len(a) – 1, 0, -1):
        a[i] = heap.removeLargest()

The total time for heapsort to sort N values is the time to heapify the array plus the time to remove each of the N values from the heap in turn. This is

O(N) + O(N log N) = O(N log N)

Thus heapsort has the same asymptotic running time as merge sort and quicksort. Some advantages of heapsort are that it runs on an array in place (unlike merge sort) and that it is guaranteed to run in time O(N log N) for any input (unlike quicksort).

prefix, infix and postfix expressions

Arithmetic expressions are composed from numbers and operators that act on those numbers. In this section we will use the operators +, -, * and /, the last of which denotes integer division.

In traditional mathematical notation, these operators are infix operators: they are written between the values that they act on (which are called operands). For example, we write 2 + 2, or 8 - 7. In this last expression, 8 and 7 are the operands.

Here are some arithmetic expressions written using infix notation:

((4 + 5) * (2 + 1))

((7 / 2)-1)

We may choose an alternative syntax that uses prefix notation, in which we write each operator before its operands. For example, we write + 2 4 to mean the sum of 2 and 4, or / 8 2 to mean 8 divided by 2. Here are the above expressions rewritten in prefix notation:

* + 4 5 + 2 1

- / 7 2 1

Or we may use postfix notation, in which operators come after both operands: we might write 4 5 + to mean the sum of 4 and 5. Here are the above expressions in postfix notation:

4 5 + 2 1 + *

7 2 / 1 -

In infix notation we must write parentheses to distinguish expressions that would otherwise be ambiguous. For example, 4 + 5 * 9 could mean either ((4 + 5) * 9) or (4 + (5 * 9)). (Another way to disambiguate expressions is using operator precedence. For example, * is generally considered to have higher precedence than +. But in this discussion we will assume that no such precedence exists.)