Lecture 9: Notes

Some topics from this lecture are also covered in the Introduction to Algorithms textbook:

• 6 Heapsort (including section 6.5 Priority queues)

priority queues

In recent lectures we learned about stacks and queues, which are abstract data types that we can implement in various ways, such as using an array or a linked list.

A priority queue is another abstract data type. Like a stack and an ordinary queue, there is one function for adding an element to a priority queue, and another function for removing an element. Here is an interface for a priority queue that holds integers:

type
  priority_queue = …

procedure init(var q: priority_queue);
function isEmpty(q: priority_queue);
procedure insert(var q: priority_queue; i: integer);
function removeLargest(var q: priority_queue): integer;

A priority queue differs from a stack and an ordinary queue in the order in which elements are removed. A stack is last in first out: the pop function removes the element that was added most recently. An ordinary queue is first in first out: the dequeue function removes the element that was added least recently. In a priority queue, the removeLargest function removes the element with the largest value.

Alternatively we can implement a priority queue of objects that have both a key and satellite (or auxiliary) data. In such a queue, removeLargest will remove the element with the largest key.

In theory we could implement a priority queue using a binary search tree. If we did so and the tree was balanced, then insert and removeLargest would run in time O(log N), where N is the number of elements in the tree. But there are more efficient data structures for implementing priority queues, such as binary heaps, to be discussed next.

binary heaps

A binary heap is a binary tree that satisfies two properties.

1. If a node with value p has a child with value c, then p >= c.

2. All levels of the tree are complete except possibly the last level, which may be missing some nodes on the right side.

For example, here is a binary heap:

This heap looks like a complete binary tree with three nodes missing on the right in the last level.

The height of a binary heap with N nodes is floor(log₂(N)) = O(log N). In other words, a binary heap is always balanced.

Typically we don’t store a binary heap using dynamically allocated nodes and pointers. Instead, we use an array, which is possible because of the shape of the binary heap tree structure. The binary heap above can be stored in an array a like this:

Notice that the array values are in the same order in which they appear in the tree, reading across tree levels from top to bottom and left to right.

Here is a tree showing the indices at which heap nodes are stored:

From this tree we can see the following patterns. If a heap node N has index i, then

• N’s left child (if any) has index i * 2 + 1

• N’s right child (if any) has index i * 2 + 2

• N’s parent (if any) has index (i - 1) div 2

So we can easily move between related tree nodes by performing index arithmetic.

In Pascal, we can store a heap like this:

type
  heap = record
    a: array of integer;  // elements a[0 .. n - 1] store the heap
    n: integer;
  end;

procedure init(var h: heap);
begin
  setLength(h.a, 1);
  h.n := 0;
end;

We store the heap size in an integer field n, which will sometimes be less than the size of the underlying array a.

Here are functions for moving through a heap using index arithmetic:

function parent(i: integer): integer;
begin
  parent := (i - 1) div 2;
end;

function leftChild(i: integer): integer;
begin
  leftChild := 2 * i + 1;
end;

function rightChild(i: integer): integer;
begin
  rightChild := 2 * i + 2;
end;

heap operations

We will now describe operations that will let us use a heap as a priority queue.

Suppose that a heap structure satisfies the heap properties, except that one node N has a value v which is smaller than one or both of its children. We can restore the heap properties by performing a down heap operation, in which the value v moves downward in the tree to an acceptable position. Let v₁ be the value of the largest of N’s children. We swap v with v₁. Now N’s value is v₁, which restores the heap property for this node, since v₁ > v and v₁ is also greater than or equal to the other child node (if there is one). We then continue this process, swapping v downward as many times as necessary until v reaches a point where it has no smaller children. The process is guaranteed to terminate successfully, since if v eventually descends to a leaf node there will be no children and the condition will be satisfied.

Here is Pascal code that implements the down heap operation:

procedure swap(var i, j: integer);
var
  k: integer;
begin
  k := i;
  i := j;
  j := k;
end;

// Move the value at index i downward until its children are smaller than it
procedure down(var h: heap; i: integer);
var
  l, r, largest: integer;
begin
  l := leftChild(i);
  r := rightChild(i);
  largest := i;
  if (l < h.n) and (h.a[l] > h.a[i]) then
    largest := l;
  if (r < h.n) and (h.a[r] > h.a[largest]) then
    largest := r;
  if largest <> i then  // some child is larger
    begin
      swap(h.a[i], h.a[largest]);
      down(h, largest);
    end;
end;

Now suppose that a heap structure satisfies the heap properties, except that one node has a value v which is larger than its parent. A complementary operation called up heap can move v upward to restore the heap properties. Suppose that v’s parent has value v₁. That parent may have a second child with value v₂. We begin by swapping v and its parent v₁. Now v’s children are v₁ and v₂ (if present). We know that v > v₁. If v₂ is present then v₁ > v₂, so v > v₂ by transitivity. Thus the heap properties have been restored for the node that now contains v. And now we continue this process, swapping v upward in the heap until it reaches a position where it is less than its parent, or until it reaches the root of the tree.

You will implement up heap in Pascal for a homework exercise.

We can use the up heap and down heap operations to implement priority queue operations.

We first consider removing the largest value from a heap. Since the largest value in a heap is always in the root node, we must place some other value there. So we take the value at the end of the heap array and place it in the root, decreasing the array size by 1 as we do so. We now perform a down heap operation on this value, which will lower it to a valid position in the tree. If there are N values in the heap, the tree height is floor(log₂(N)), so this process is guaranteed to run in O(log N) time.

Here is removeLargest in Pascal:

function removeLargest(var h: heap): integer;
begin
  removeLargest := h.a[0];
  h.a[0] := h.a[h.n - 1];
  h.n := h.n - 1;
  down(h, 0);
end;

Now we consider inserting a value v into a heap. To do this, we first add v to the end of the heap array, expanding the array size by 1. Now v is in a new leaf node at the end of the last heap level. We next perform an up heap operation on the value v, which will bring it upward to a valid position. Just like removeLargest, insert will always run in time O(log N).

You’ll implement the insert operation in a homework exercise.

heapsort

We’ve already seen several sorting algorithms in this course: bubble sort, insertion sort, merge sort and quicksort. We can use a binary heap to implement another sorting algorithm: heapsort. The essence of heapsort is simple: given a array of numbers to sort, we put them into a heap, and then we pull out the numbers one by one in sorted order.

Let’s examine these steps in a bit more detail. Given an array of N values, we first need to heapify it, i.e. turn it into a heap. One way to do this is to simply insert each number in turn using the insert operation described above. This will time O(N log N).

But there is actually a more efficient way to heapify the array: we walk backward through the array, running the down heap operation on each value in turn. Here is Pascal code to do that:

type
  intarray = array of integer;

procedure heapify(out h: heap; var a: intarray);
var
  i: integer;
begin
  h.a := a;
  h.n := length(a);
  
  for i := h.n - 1 downto 0 do
    down(h, i);
end;

How long will heapify take to run? The depth of each tree node N is the length of the longest path from N downward to a leaf. The depth of each leaf is 0. Given a node N with depth d, down(N) will perform at most d swaps, so it will run in time O(d). Thus the running time of heapify is proportional to the total of the depths of all nodes in the tree.

Let D(N) be the total depth of all nodes in a binary heap with N nodes. Suppose that binary heaps H₁ and H₂ both consist of complete binary trees and have heights t -1 and t respectively. Then H₁ has 2^t – 1 nodes and H₂ has 2^{t + 1} – 1 nodes. The leaves of H₂ contribute nothing to its total depth. Every other node in H₂ has a height that equals one plus the height of the corresponding node in H₁. So

D(2^{t + 1} – 1) = D(2^t – 1) + 2^t – 1

Letting N = 2^{t + 1} – 1, we have

D(N) = D(floor(N / 2)) + floor(N / 2)

This looks very similar to the recurrence

E(N) = E(N / 2) + N / 2

whose solution is

E(N) = N / 2 + N / 4 + N / 8 + … + 1 = O(N)

As you might expect it is also true that D(N) = O(N) (though we will not prove that formally here).

Once we have heapified the array, sorting it is easy: we repeatedly call removeLargest to remove the largest element, which we place at the end of the array. Here is our Pascal implementation:

procedure heapsort(var a: intarray);
var
  h: heap;
  i: integer;
begin
  heapify(h, a);

  while h.n > 0 do
    begin
      i := h.n;
      h.a[i] := removeLargest(h);
    end;
end;

The total time for heapsort to sort N values is the time to heapify the array plus the time to remove each of the N values from the heap in turn. This is

O(N) + O(N log N) = O(N log N)

Thus heapsort has the same asymptotic running time as merge sort and quicksort. Some advantages of heapsort are that it runs on an array in place (unlike merge sort) and that it is guaranteed to run in time O(N log N) for any input (unlike quicksort).

arithmetic expressions

Arithmetic expressions are composed from numbers and operators that act on those numbers. In this section we will use the operators +, -, * and /, where / denotes integer division.

In traditional mathematical notation, these operators are infix operators: they are written between the values that act on (which are called operands). For example, we write 2 + 2, or 8 - 7. In this last expression, 8 and 7 are the operands.

Here are some arithmetic expressions written using infix notation:

((4 + 5) * (2 + 1))

((7 / 2) – 1)

We may choose an alternative syntax that uses prefix notation, in which we write each operator before its operands. For example, we write + 2 4 to mean the sum of 2 and 4, or / 8 2 to mean 8 divided by 2. Here are the above expressions rewritten in prefix notation:

* + 4 5 + 2 1

- / 7 2 1

Or we may use postfix notation, in which operators come after both operands: we might write 4 5 + to mean the sum of 4 and 5. Here are the above expressions in postfix notation:

4 5 + 2 1 + *

7 2 / 1 -

In infix notation we must write parentheses to distinguish expressions that would otherwise be ambiguous. For example, 4 + 5 * 9 could mean either ((4 + 5) * 9) or (4 + (5 * 9)). (Another way to disambiguate expressions is using operator precedence. For example, * is generally considered to have higher precedence than +. But in this discussion we will assume that no such precedence exists.)

In prefix or postfix notation there is no need either for parentheses or operator precedence, because expressions are inherently unambiguous. For example, the prefix expression * + 4 5 9 is equivalent to ((4 + 5) * 9), and the prefix expression + 4 * 5 9 is equivalent to (4 + (5 * 9)). There is no danger of confusing these in prefix form even without parentheses.

grammars for arithmetic expressions

Let’s formally define several forms of arithmetic expressions. To keep things simple, we will work with expressions that contain only single-digit numbers.

We can define a language of infix arithmetic expressions using the following context-free grammar:

digit → ‘0’ | ‘1’ | ‘2’ | ‘3’ | ‘4’ | ‘5’ | ‘6’ | ‘7’ | ‘8’ | ‘9’

op → ‘+’ | ‘-’ | ‘*’ | ‘/’

expr → digit

expr → ‘(‘ expr op expr ‘)’

Context-free grammars are commonly used to define programming languages. A grammar contains non-terminal symbols (such as digit, op, expr) and terminal symbols (such as ‘0’, ‘1’, ‘2’, …, ‘+’, ‘-’, ‘*’, ‘/’). It contains a set of production rules that define how each non-terminal symbol can be constructed from other symbols. These rules collectively define which strings of terminal symbols are valid, i.e. which strings belong to the language being defined.

For example, “((4 + 5) * 9)” is a valid expression in the language defined by this grammar, because it can be constructed from the top-level non-terminal expr by successively replacing symbols using the production rules:

expr

→ ( expr op expr )

→ ( expr op digit )

→ (expr op 9)

→ (expr * 9)

→ ((expr op expr) * 9)

→ ((expr op digit) * 9)

→ ((expr op 5) * 9)

→ ((expr + 5) * 9)

→ ((digit + 5) * 9)

→ ((4 + 5) * 9)

You will learn much more about context-free grammars in more advanced courses, but it’s worth taking this first look at them now because they are so commonly used to specify languages.

To modify our language to use prefix operators, we need change only the last production rule above:

expr → op expr expr

Or, for postfix operators:

expr → expr expr op

Note again that the prefix and postfix expression languages have no parentheses.

evaluating arithmetic expressions

It is not difficult to write Pascal functions that evaluate arithmetic expressions, i.e. compute the numeric value that each expression represents. Because arithmetic expressions have a recursive structure, these functions will often be recursive as well.

First, here are some useful helper functions including evalOp, which applies an operator to two integer arguments:

uses character, sysutils;

procedure error(s: string);
begin
  writeln(s);
  halt;
end;

function readChar(): char;
begin
  if seekEoln then error('unexpected end of input');
  read(readChar);
end;

function isOp(c: char): boolean;
begin
  case c of
    '+', '-', '*', '/': exit(true);
    else exit(false);
  end;
end;

function evalOp(c: char; i, j: integer): integer;
begin
  case c of
    '+': exit(i + j);
    '-': exit(i - j);
    '*': exit(i * j);
    '/': exit(i div j);
  end;
end;

We can evaluate a prefix expression using a recursive function. Its structure reflects the grammar for prefix expressions defined in the previous section.

function evalPrefix(): integer;
var
  c: char;
  i, j: integer;
begin
  c := readChar();
  if isDigit(c) then exit(strToInt(c));   // expr → digit
  if not isOp(c) then error('operator expected');
  i := evalPrefix();
  j := evalPrefix();
  exit(evalOp(c, i, j));    // expr → op expr expr
end;

We can evaluate an infix expression similarly:

function evalInfix(): integer;
var
  c: char;
  i, j: integer;
begin
  c := readChar();
  if isDigit(c) then exit(strToInt(c));   // expr → digit
  if c <> '(' then error('expected (');
  
  i := evalInfix();
  c := readChar();
  if not isOp(c) then error('operator expected');
  j := evalInfix();
  
  if readChar() <> ')' then error('expected )');
  exit(evalOp(c, i, j));  // expr → ‘(‘ expr op expr ‘)’
end;

The most straightforward way to evaluate a postfix expression is using a stack. As we read a postfix expression from left to right, each time we see a number we push it onto the stack. When we see an operator, we pop two numbers from the stack, apply the operator and then push the result back onto the stack. For example, in evaluating the postfix expression 4 5 + 2 1 + * we would perform the following operations:

• push 4

• push 5

• pop 4 and 5, push 9

• push 2

• push 1

• pop 2 and 1, push 3

• pop 9 and 3, push 27

When we finish reading the expression, if it is valid then the stack will contain a single number, which is the result of our expression evaluation.

You will write code to evaluate a postfix expression using a stack in an exercise this week.

variant records

We would now like to store and manipulate arithmetic expressions in memory as our program runs. We will do this using a tree structure, but we will first need to learn about one more feature of the Pascal language.

A variant record is a record whose field set can vary depending on the value of a tag field in the record. For example:

type
  shapeType = (square, rectangle, circle);

  shape = record
    centerX, centerY: real;

    case kind: shapeType of
      square: (side: real);
      rectangle: (length, height: real);
      circle: (radius: real);
    end;

In the example above, kind is the tag field, and has type shapeType which is an enumerated type. A tag field may alternatively have any ordinal type, such as integer or boolean.

To use a variant record, simply set its tag field and associated field values:

var
  s: shape;

begin
  s.kind := rectangle;
  s.length := 5.0;
  s.height := 3.0;
  ...

Code that processes a variant record will often want to use a case statement to branch based on its tag field:

  case s.kind of
    square:    area := s.side * s.side;
    rectangle: area := s.length * s.height;
    circle:    area := pi * s.radius * s.radius;
  end;

expression trees

We will store expressions in our program using an expression tree, which is a form of abstract syntax tree. An expression tree reflects an expression’s hierarchical structure. For example, here is an expression tree for the infix expression ((3 + 4) * (2 + 5)):

Note that this expression tree also corresponds to the prefix expression * + 3 4 + 2 5, or the postfix expression 3 4 + 2 5 + *. Equivalent expressions in infix, prefix or postfix form will always have the same tree! That’s because this tree reflects the abstract syntax of the expression, which is independent of concrete syntax which defines how expressions are written as strings of symbols.

We can easily store expression trees using variant records in Pascal. Here is a variant record type for arithmetic expressions:

uses character, sysutils;

type
  expType = (int, op);
  
  expression = record
    case kind: expType of
      int: (i: integer);
      op: (c: char;
           exp1, exp2: ^expression);
    end;
  
  pexp = ^expression;

It is not difficult to modify our prefix evaluator to generate an expression tree:

function parsePrefix(): pexp;
var
  c: char;
  e: ^expression;
begin
  c := readChar();
  if isDigit(c) then
    begin
      new(e);
      e^.kind := int;
      e^.i := strToInt(c);
    end
  else
    begin
      new(e);
      e^.kind := op;
      e^.c := c;
      e^.exp1 := parsePrefix();
      e^.exp2 := parsePrefix();
    end;
  exit(e);
end;

We could similarly modify the infix evaluator we saw above to generate an expression tree.

To generate a expression tree from a postfix expression, we can use a stack of pointers to expression trees. When we read an operand, we pop two subtrees from the stack, combine them into an operator node, then push it back on the stack.

Once we have an expression tree in memory, it’s not hard to print out its expression in either infix, prefix or postfix form. Here is a procedure to print out an infix expression:

procedure printExpression(e: pexp);
begin
  case e^.kind of
    int:
      write(e^.i);
    op:
      begin
        write('(');
        printExpression(e^.exp1);
        write(' ', e^.c, ' ');
        printExpression(e^.exp2);
        write(')');
      end;
  end;
end;

We can also evaluate an expression tree with a straightforward recursive function:

function evalExpression(e: pexp): integer;
var
  i, j: integer;
begin
  case e^.kind of
    int: exit(e^.i);
    op:
      begin
        i := evalExpression(e^.exp1);
        j := evalExpression(e^.exp2);
        exit(evalOp(e^.c, i, j));
      end;
  end;
end;