Some topics from this lecture are also covered in the Introduction to Algorithms textbook:
6 Heapsort (including section 6.5 Priority queues)
In recent lectures we learned about stacks and queues, which are abstract data types that we can implement in various ways, such as using an array or a linked list.
A priority queue is another abstract data type. Like a stack and an ordinary queue, there is one function for adding an element to a priority queue, and another function for removing an element. Here is an interface for a priority queue that holds integers:
type priority_queue = … procedure init(var q: priority_queue); function isEmpty(q: priority_queue); procedure insert(var q: priority_queue; i: integer); function removeLargest(var q: priority_queue): integer;
A priority queue differs from a stack and an ordinary queue in the
order in which elements are removed. A stack is last in first out:
the pop
function removes the element that was added most
recently. An ordinary queue is first in first out: the dequeue
function removes the element that was added least recently. In a
priority queue, the removeLargest
function removes the
element with the largest value.
Alternatively we can implement a priority queue of objects that
have both a key and satellite (or
auxiliary)
data. In such a queue, removeLargest
will remove
the element with the largest key.
In theory we could implement a priority queue using a binary search tree. If we did so and the tree was balanced, then insert and removeLargest would run in time O(log N), where N is the number of elements in the tree. But there are more efficient data structures for implementing priority queues, such as binary heaps, to be discussed next.
A binary heap is a binary tree that satisfies two properties.
If a node with value p has a child with value c, then p >= c.
All levels of the tree are complete except possibly the last level, which may be missing some nodes on the right side.
For example, here is a binary heap:
This heap looks like a complete binary tree with three nodes missing on the right in the last level.
The height of a binary heap with N nodes is floor(log_{2}(N)) = O(log N). In other words, a binary heap is always balanced.
Typically we don’t store a binary heap using dynamically
allocated nodes and pointers. Instead, we use an array, which is
possible because of the shape of the binary heap tree structure. The
binary heap above can be stored in an array a
like this:
Notice
that the array values are in the same order in which they appear in
the tree, reading across tree levels from top to bottom and left to
right.
Here is a tree showing the indices at which heap nodes are stored:
From
this tree we can see the following patterns. If a heap node N has
index i, then
N’s left child (if any) has index i * 2 + 1
N’s right child (if any) has index i * 2 + 2
N’s parent (if any) has index (i - 1) div 2
So we can easily move between related tree nodes by performing index arithmetic.
In Pascal, we can store a heap like this:
type heap = record a: array of integer; // elements a[0 .. n - 1] store the heap n: integer; end; procedure init(var h: heap); begin setLength(h.a, 1); h.n := 0; end;
We store the heap size in an integer field n
, which will
sometimes be less than the size of the underlying array a
.
Here are functions for moving through a heap using index arithmetic:
function parent(i: integer): integer; begin parent := (i - 1) div 2; end; function leftChild(i: integer): integer; begin leftChild := 2 * i + 1; end; function rightChild(i: integer): integer; begin rightChild := 2 * i + 2; end;
We will now describe operations that will let us use a heap as a priority queue.
Suppose that a heap structure satisfies the heap properties, except that one node N has a value v which is smaller than one or both of its children. We can restore the heap properties by performing a down heap operation, in which the value v moves downward in the tree to an acceptable position. Let v_{1} be the value of the largest of N’s children. We swap v with v_{1}. Now N’s value is v_{1}, which restores the heap property for this node, since v_{1} > v and v_{1} is also greater than or equal to the other child node (if there is one). We then continue this process, swapping v downward as many times as necessary until v reaches a point where it has no smaller children. The process is guaranteed to terminate successfully, since if v eventually descends to a leaf node there will be no children and the condition will be satisfied.
Here is Pascal code that implements the down heap operation:
procedure swap(var i, j: integer); var k: integer; begin k := i; i := j; j := k; end; // Move the value at index i downward until its children are smaller than it procedure down(var h: heap; i: integer); var l, r, largest: integer; begin l := leftChild(i); r := rightChild(i); largest := i; if (l < h.n) and (h.a[l] > h.a[i]) then largest := l; if (r < h.n) and (h.a[r] > h.a[largest]) then largest := r; if largest <> i then // some child is larger begin swap(h.a[i], h.a[largest]); down(h, largest); end; end;
Now suppose that a heap structure satisfies the heap properties, except that one node has a value v which is larger than its parent. A complementary operation called up heap can move v upward to restore the heap properties. Suppose that v’s parent has value v_{1}. That parent may have a second child with value v_{2}. We begin by swapping v and its parent v_{1}. Now v’s children are v_{1} and v_{2} (if present). We know that v > v_{1}. If v_{2} is present then v_{1} > v_{2}, so v > v_{2} by transitivity. Thus the heap properties have been restored for the node that now contains v. And now we continue this process, swapping v upward in the heap until it reaches a position where it is less than its parent, or until it reaches the root of the tree.
You will implement up heap in Pascal for a homework exercise.
We can use the up heap and down heap operations to implement priority queue operations.
We first consider removing the largest value from a heap. Since the largest value in a heap is always in the root node, we must place some other value there. So we take the value at the end of the heap array and place it in the root, decreasing the array size by 1 as we do so. We now perform a down heap operation on this value, which will lower it to a valid position in the tree. If there are N values in the heap, the tree height is floor(log_{2}(N)), so this process is guaranteed to run in O(log N) time.
Here is removeLargest in Pascal:
function removeLargest(var h: heap): integer; begin removeLargest := h.a[0]; h.a[0] := h.a[h.n - 1]; h.n := h.n - 1; down(h, 0); end;
Now we consider inserting
a value v into a heap. To do this,
we first add v to the end of the heap array, expanding the array size
by 1. Now v is in a new leaf node at the end of the last heap level.
We next perform an up heap operation on the value v, which will bring
it upward to a valid position. Just like removeLargest
,
insert
will always run in time O(log N).
You’ll implement the insert operation in a homework exercise.
We’ve already seen several sorting algorithms in this course: bubble sort, insertion sort, merge sort and quicksort. We can use a binary heap to implement another sorting algorithm: heapsort. The essence of heapsort is simple: given a array of numbers to sort, we put them into a heap, and then we pull out the numbers one by one in sorted order.
Let’s examine these steps in a bit more detail. Given an array
of N values, we first need to heapify it, i.e. turn it into a
heap. One way to do this is to simply insert each number in turn
using the insert
operation described above. This will
time O(N log N).
But there is actually a more efficient way to heapify the array: we walk backward through the array, running the down heap operation on each value in turn. Here is Pascal code to do that:
type intarray = array of integer; procedure heapify(out h: heap; var a: intarray); var i: integer; begin h.a := a; h.n := length(a); for i := h.n - 1 downto 0 do down(h, i); end;
How long will heapify
take to run? The depth of
each tree node N is the length of the longest path from N downward to
a leaf. The depth of each leaf is 0. Given a node N with depth d,
down(N)
will perform at most d swaps, so it will run in
time O(d). Thus the running time of heapify is proportional to the
total of the depths of all nodes in the tree.
Let D(N) be the total depth of all nodes in a binary heap with N nodes. Suppose that binary heaps H_{1} and H_{2} both consist of complete binary trees and have heights t -1 and t respectively. Then H_{1} has 2^{t} – 1 nodes and H_{2} has 2^{t + 1} – 1 nodes. The leaves of H_{2} contribute nothing to its total depth. Every other node in H_{2} has a height that equals one plus the height of the corresponding node in H_{1}. So
D(2^{t + 1} – 1) = D(2^{t} – 1) + 2^{t} – 1
Letting N = 2^{t + 1} – 1, we have
D(N) = D(floor(N / 2)) + floor(N / 2)
This looks very similar to the recurrence
E(N) = E(N / 2) + N / 2
whose solution is
E(N) = N / 2 + N / 4 + N / 8 + … + 1 = O(N)
As you might expect it is also true that D(N) = O(N) (though we will not prove that formally here).
Once we have heapified the array, sorting it is easy: we
repeatedly call removeLargest
to remove the largest
element, which we place at the end of the array. Here is our Pascal
implementation:
procedure heapsort(var a: intarray); var h: heap; i: integer; begin heapify(h, a); while h.n > 0 do begin i := h.n; h.a[i] := removeLargest(h); end; end;
The total time for heapsort to sort N values is the time to heapify the array plus the time to remove each of the N values from the heap in turn. This is
O(N) + O(N log N) = O(N log N)
Thus heapsort has the same asymptotic running time as merge sort and quicksort. Some advantages of heapsort are that it runs on an array in place (unlike merge sort) and that it is guaranteed to run in time O(N log N) for any input (unlike quicksort).
Arithmetic expressions are composed from numbers and operators that act on those numbers. In this section we will use the operators +, -, * and /, where / denotes integer division.
In traditional mathematical notation, these operators are infix operators: they are written between the values that act on (which are called operands). For example, we write 2 + 2, or 8 - 7. In this last expression, 8 and 7 are the operands.
Here are some arithmetic expressions written using infix notation:
((4 + 5) * (2 + 1))
((7 / 2) – 1)
We may choose an alternative syntax that uses prefix notation, in which we write each operator before its operands. For example, we write + 2 4 to mean the sum of 2 and 4, or / 8 2 to mean 8 divided by 2. Here are the above expressions rewritten in prefix notation:
* + 4 5 + 2 1
- / 7 2 1
Or we may use postfix notation, in which operators come after both operands: we might write 4 5 + to mean the sum of 4 and 5. Here are the above expressions in postfix notation:
4 5 + 2 1 + *
7 2 / 1 -
In infix notation we must write parentheses to distinguish expressions that would otherwise be ambiguous. For example, 4 + 5 * 9 could mean either ((4 + 5) * 9) or (4 + (5 * 9)). (Another way to disambiguate expressions is using operator precedence. For example, * is generally considered to have higher precedence than +. But in this discussion we will assume that no such precedence exists.)
In prefix or postfix notation there is no need either for parentheses or operator precedence, because expressions are inherently unambiguous. For example, the prefix expression * + 4 5 9 is equivalent to ((4 + 5) * 9), and the prefix expression + 4 * 5 9 is equivalent to (4 + (5 * 9)). There is no danger of confusing these in prefix form even without parentheses.
Let’s formally define several forms of arithmetic expressions. To keep things simple, we will work with expressions that contain only single-digit numbers.
We can define a language of infix arithmetic expressions using the following context-free grammar:
digit → ‘0’ | ‘1’ | ‘2’ | ‘3’ | ‘4’ | ‘5’ | ‘6’ | ‘7’ | ‘8’ | ‘9’
op → ‘+’ | ‘-’ | ‘*’ | ‘/’
expr → digit
expr → ‘(‘ expr op expr ‘)’
Context-free grammars are commonly used to define programming languages. A grammar contains non-terminal symbols (such as digit, op, expr) and terminal symbols (such as ‘0’, ‘1’, ‘2’, …, ‘+’, ‘-’, ‘*’, ‘/’). It contains a set of production rules that define how each non-terminal symbol can be constructed from other symbols. These rules collectively define which strings of terminal symbols are valid, i.e. which strings belong to the language being defined.
For example, “((4 + 5) * 9)” is a valid expression in the language defined by this grammar, because it can be constructed from the top-level non-terminal expr by successively replacing symbols using the production rules:
expr
→ ( expr op expr )
→ ( expr op digit )
→ (expr op 9)
→ (expr * 9)
→ ((expr op expr) * 9)
→ ((expr op digit) * 9)
→ ((expr op 5) * 9)
→ ((expr + 5) * 9)
→ ((digit + 5) * 9)
→ ((4 + 5) * 9)
You will learn much more about context-free grammars in more advanced courses, but it’s worth taking this first look at them now because they are so commonly used to specify languages.
To modify our language to use prefix operators, we need change only the last production rule above:
expr → op expr expr
Or, for postfix operators:
expr → expr expr op
Note again that the prefix and postfix expression languages have no parentheses.
It is not difficult to write Pascal functions that evaluate arithmetic expressions, i.e. compute the numeric value that each expression represents. Because arithmetic expressions have a recursive structure, these functions will often be recursive as well.
First, here are some useful helper functions including evalOp
,
which applies an operator to two integer arguments:
uses character, sysutils; procedure error(s: string); begin writeln(s); halt; end; function readChar(): char; begin if seekEoln then error('unexpected end of input'); read(readChar); end; function isOp(c: char): boolean; begin case c of '+', '-', '*', '/': exit(true); else exit(false); end; end; function evalOp(c: char; i, j: integer): integer; begin case c of '+': exit(i + j); '-': exit(i - j); '*': exit(i * j); '/': exit(i div j); end; end;
We can evaluate a prefix expression using a recursive function. Its structure reflects the grammar for prefix expressions defined in the previous section.
function evalPrefix(): integer; var c: char; i, j: integer; begin c := readChar(); if isDigit(c) then exit(strToInt(c)); // expr → digit if not isOp(c) then error('operator expected'); i := evalPrefix(); j := evalPrefix(); exit(evalOp(c, i, j)); // expr → op expr expr end;
We can evaluate an infix expression similarly:
function evalInfix(): integer; var c: char; i, j: integer; begin c := readChar(); if isDigit(c) then exit(strToInt(c)); // expr → digit if c <> '(' then error('expected ('); i := evalInfix(); c := readChar(); if not isOp(c) then error('operator expected'); j := evalInfix(); if readChar() <> ')' then error('expected )'); exit(evalOp(c, i, j)); // expr → ‘(‘ expr op expr ‘)’ end;
The most straightforward way to evaluate a postfix expression is using a stack. As we read a postfix expression from left to right, each time we see a number we push it onto the stack. When we see an operator, we pop two numbers from the stack, apply the operator and then push the result back onto the stack. For example, in evaluating the postfix expression 4 5 + 2 1 + * we would perform the following operations:
push 4
push 5
pop 4 and 5, push 9
push 2
push 1
pop 2 and 1, push 3
pop 9 and 3, push 27
When we finish reading the expression, if it is valid then the stack will contain a single number, which is the result of our expression evaluation.
You will write code to evaluate a postfix expression using a stack in an exercise this week.
We would now like to store and manipulate arithmetic expressions in memory as our program runs. We will do this using a tree structure, but we will first need to learn about one more feature of the Pascal language.
A variant record is a record whose field set can vary depending on the value of a tag field in the record. For example:
type shapeType = (square, rectangle, circle); shape = record centerX, centerY: real; case kind: shapeType of square: (side: real); rectangle: (length, height: real); circle: (radius: real); end;
In the example above, kind
is the tag field, and has
type shapeType
which is an enumerated type. A tag field
may alternatively have any ordinal type, such as integer or boolean.
To use a variant record, simply set its tag field and associated field values:
var s: shape; begin s.kind := rectangle; s.length := 5.0; s.height := 3.0; ...
Code that processes a variant record will often want to use a case
statement to branch based on its tag field:
case s.kind of square: area := s.side * s.side; rectangle: area := s.length * s.height; circle: area := pi * s.radius * s.radius; end;
We will store expressions in our program using an expression tree, which is a form of abstract syntax tree. An expression tree reflects an expression’s hierarchical structure. For example, here is an expression tree for the infix expression ((3 + 4) * (2 + 5)):
Note
that this expression tree also corresponds to the prefix expression *
+ 3 4 + 2 5, or the postfix expression 3 4 + 2 5 + *. Equivalent
expressions in infix, prefix or postfix form will always have the
same tree! That’s because this tree reflects the abstract
syntax of the expression, which
is independent of concrete syntax which
defines how expressions are written as strings of symbols.
We can easily store expression trees using variant records in Pascal. Here is a variant record type for arithmetic expressions:
uses character, sysutils; type expType = (int, op); expression = record case kind: expType of int: (i: integer); op: (c: char; exp1, exp2: ^expression); end; pexp = ^expression;
It is not difficult to modify our prefix evaluator to generate an expression tree:
function parsePrefix(): pexp; var c: char; e: ^expression; begin c := readChar(); if isDigit(c) then begin new(e); e^.kind := int; e^.i := strToInt(c); end else begin new(e); e^.kind := op; e^.c := c; e^.exp1 := parsePrefix(); e^.exp2 := parsePrefix(); end; exit(e); end;
We could similarly modify the infix evaluator we saw above to generate an expression tree.
To generate a expression tree from a postfix expression, we can use a stack of pointers to expression trees. When we read an operand, we pop two subtrees from the stack, combine them into an operator node, then push it back on the stack.
Once we have an expression tree in memory, it’s not hard to print out its expression in either infix, prefix or postfix form. Here is a procedure to print out an infix expression:
procedure printExpression(e: pexp); begin case e^.kind of int: write(e^.i); op: begin write('('); printExpression(e^.exp1); write(' ', e^.c, ' '); printExpression(e^.exp2); write(')'); end; end; end;
We can also evaluate an expression tree with a straightforward recursive function:
function evalExpression(e: pexp): integer; var i, j: integer; begin case e^.kind of int: exit(e^.i); op: begin i := evalExpression(e^.exp1); j := evalExpression(e^.exp2); exit(evalOp(e^.c, i, j)); end; end; end;