Introduction to Algorithms
Week 7: Notes

Some of today's topics are covered in Problem Solving with Algorithms:

And in Introduction to Algorithms:

Here are some more notes on the following topics:

Quicksort

In recent lectures we've seen several sorting algorithms (bubble sort, insertion sort, mergesort). Quicksort is another sorting algorithm. It is efficient (typically at least twice as fast as mergesort) and is very commonly used.

Quicksort sorts values in an array in place (unlike mergesort, which needs to use extra memory to perform merges). Given a slice a[start:end] of values to sort in an array a, quicksort first arranges the slice elements into two non-empty partitions a[start:k] and a[k:end] for some integer k. It does this in such a way that every element in the first partition (a[start:k]) is less than or equal to every element in the second partition (a[k:end]). After partitioning, quicksort recursively calls itself on each of the two partitions.

The high-level structure of quicksort is similar to mergesort: both algorithms divide the array in two and call themselves recursively on both halves. With quicksort, however, there is no more work to do after these recursive calls. Sorting both partitions completes the work of sorting the array that contains them. Another difference is that merge sort always divides the array into two equal-sized pieces, but in quicksort the two partitions are generally not the same size, and sometimes one may be much larger than the other.

Two different partitioning algorithms are in common use; in this course we will use Hoare partitioning, which works as follows. To partition an array slice, we first choose a random element in the slice to use as the pivot. The pivot has some value v. Once the partition is complete, all elements in the first partition will have values less than or equal to v, and elements in the right partition will have values greater than or equal to v.

We define integer variables i and j representing array indexes. Initially i is positioned at the left end of the array and j is positioned at the right. We move i rightward, looking for a value that is greater than or equal to v (i.e. it can go into the second partition). And we move j leftward, looking for a value that is less than or equal to v. Once we have found these values, we swap a[i] and a[j]. After the swap, a[i] and a[j] now hold acceptable values for the left and right partitions, respectively. Now we continue the process, moving i to the right and j to the left. Eventually i and j meet. The point where they meet is the division point between the partitions, i.e. is the index k mentioned above.

Note that the pivot element itself may move to a different position during the partitioning process! There is nothing wrong with that. The pivot really just serves as an arbitrary value to use for separating the partitions.

For example, consider quicksort’s operation on this array:

%3

The array has n = 8 elements. Suppose that we choose 3 as our pivot. We begin by setting i = 0 and j = 7. a[0] = 6, which is greater than the pivot value 3. We move j leftward, looking for a value that is less than or equal to 3. We soon find a[6] = 2 ≤ 3. So we swap a[0] and a[6]:

%3

Now a[1] = 5 ≥ 3. j moves leftward until j = 3, since a[3] = 3 ≤ 3. So we swap a[1] and a[3]:

%3

Now i moves rightward until it hits a value that is greater than or equal to 3; the first such value is a[3] = 5. Similarly, j moves leftward until it hits a value that is less than or equal to 3; the first such value is a[2] = 1. So i = 3 and j = 2, and j < i. i and j have crossed, so the partitioning is complete. The first partition is a[0:3] and the second partition is a[3:8]. Every element in the first partition is less than every element in the second partition. Quicksort will now sort both partitions, recursively.

Notice that in this example the pivot element itself moved during the partitioning process. As mentioned above, this is common.

Here is an implementation of Hoare partitioning in Python:

# We are given that end - start >= 2, and that the pivot appears in the array
# somewhere before the end.  Return a value k in the range start < k < end
# such that a[start:k] are all <= (pivot), and a[k:end] are all >= (pivot).
# The pivot element itself may move!

def partition(a, start, end, pivot):
  i = start
  j = end - 1
  
  while True:
    while a[i] < pivot:
      i += 1
    while a[j] > pivot:
      j -= 1
    
    if j <= i:
      return j + 1
      
    a[i], a[j] = a[j], a[i]
    i += 1
    j -= 1


Note that the pivot element must appear before the last element of the array. (If it were present in the array only at the end, then partition() could return the value len(a), and then the second partition would be empty, violating property (2) above.)

With partition in place, we can easily write our top-level quicksort function:

def qsort(a, start, end):
  if end - start < 2:
    return
    
  pivot = a[randint(start, end - 2)]  # avoid choosing last element
  k = partition(a, start, end, pivot)
  qsort(a, start, k)
  qsort(a, k, end)

def quicksort(a):
  qsort(a, 0, len(a))

performance of Quicksort

To analyze the performance of quicksort, we first observe that partition(a, start, end, pivot) runs in time O(n), where n = end - start. This is because i and j stay within bounds so their values will increment or decrement at most n times.

In the best case, quicksort divides the input array into two equal-sized partitions at each step. Then we have

T(n) = 2 T(n / 2) + O(n)

This is the same recurrence we saw when we analyzed mergesort in a previous lecture. Its solution is

T(n) = O(n log n)

In the worst case, at each step one partition has a single element and the other partition has all remaining elements. Then

T(n) = T(n – 1) + O(n)

This yields

T(n) = O(n2)

So how will quicksort typically perform?

This depends critically on the choice of pivot elements. But if we choose pivot elements randomly as we have done here, then quicksort’s expected performance will be O(n log n) for any input array. (We will not prove this fact in this class, however.)

Stacks

A stack is a data type that provides two operations called push and pop. push(x) pushes a value x onto a stack, and pop() removes the value that was most recently pushed. This is like a stack of sheets of paper on a desk, where sheets can be added or removed at the top.

In other words, a stack is a last in first out (LIFO) data structure: the last element that was added is the first to be removed.

Often a stack will provide an additional operation isEmpty that returns true if it contains no elements.

Before we implement a stack, let's look at how one can be used:

s = Stack();
s.push(4);
s.push(8);
for i in range(1, 6) do
  s.push(i);
  
while not s.isEmpty() do
  print(pop(s), end = ' ')
print()

This code will write

5 4 3 2 1 8 4

Implementing a stack with an array

Stacks are an abstract data type, meaning that they can be implemented in various ways. A straightforward way to implement a stack in Python is using a list, i.e. an array:

class Stack:
  def __init__(self):
    self.vals = []

  def isEmpty(self):
    return len(self.vals) == 0
  
  def push(self, x):
    self.vals.append(x)
  
  def pop(self):
    assert not self.isEmpty(), 'stack is empty'
    return self.vals.pop()

The implementation is straightforward. The list contains all stack elements, with the top of the stack (i.e. the most recently pushed element) at the end of the list.

With this implementation, push will run in O(1) on average, since that is the running time of append(). pop will always run in O(1).

Queues

Queues are another important abstract data type. A queue provides two operations called enqueue and dequeue. enqueue(x) adds an element to the tail of a queue, and dequeue() removes the element at the head. A queue is something like people waiting in a line: you must join at the back of the line, and the person at the front of the line is served next.

In other words, queues are a first in first out (FIFO) data structure: the first value added to a queue will be the first one to be removed.

A queue can be used like this:

q = Queue()

q.enqueue(4)
q.enqueue(77)
q.enqueue(12)

print(q.dequeue())  # writes 4
print(q.dequeue())  # writes 77

Implementing a queue with a circular array

One possible way to implement a queue is using a Python list, i.e. an array. In addition to the array itself, we keep variables head and tail that indicate the array positions of the first and next-to-be-added elements. For example, if head = 3 and tail = 6 then there are currently three elements in the queue, with values a[3], a[4] and a[5].

As we add elements to the queue, they may wrap past the end of the array back to the beginning. For example, if len(a) is 8, head is 6 and tail is 2 then there are four elements in the queue: a[6], a[7], a[0] and a[1].

In our implementation, the queue has a fixed maximum size, which is the size of the array. If the tail reaches the head, then the array is full. Here is our implementation:

class Queue:
  def __init__(self, size):
    self.size = size
    self.vals = size * [None]
    self.head = 0
    self.tail = 0
  
  def count(self):
    return (self.tail - self.head) % self.size

  def isEmpty(self):
    return self.count() == 0
  
  def enqueue(self, x):
    assert self.count() < self.size - 1, 'queue is full'
    self.vals[self.tail] = x
    self.tail = (self.tail + 1) % self.size

  def dequeue(self):
    v = self.vals[self.head]
    self.head = (self.head + 1) % self.size
    return v

With this implementation, the enqueue and dequeue operations will run in O(1).

linked lists

A linked list is a common and useful data structure. It looks like this:

%3

Like an array, a linked list can hold a sequence of elements. But it performs quite differently from an array. We can access the j-th element of an array in constant time for any j, but inserting or deleting an element at the beginning of an array or in the middle takes time O(N), where N is the length of the array. Conversely, accessing the j-th element of a linked list takes time O(j), but insertions and deletions take O(1).

An element of a linked list is called a node. A node contains one or more values, plus a pointer to the next node in the list. The first node of a linked list is called its head. The last node of a linked list is its tail. The tail always points to None (in Python, or its equivalent such as nil in other languages).

By the way, we will sometimes illustrate a linked list more compactly:

2 → 4 → 7 → None

The two pictures above denote the same structure; the first is simply more detailed.

Note that a Python list is not a linked list! A Python list is an array. :)

Here is a node type for a linked list in Python:

class Node:
  def __init__(self, val, next):
    self.val = val
    self.next = next

We can build the 3-element linked list pictured above as follows:

r = Node(7, None)
q = Node(4, r)
p = Node(2, q)

Now p refers to the head of the list.

Let's build a simple class LinkedList that illustrates common operations on linked lists. A LinkedList will have a single attribute head that points to the head of the list. Initially it is None:

class LinkedList:
  def __init__(self):
    self.head = None

We can easily prepend a value to a linked list. We allocate a new node to hold the value, and set the node's next pointer to the current head of the list. Then we update the head of the list to point to the new node:

  def prepend(self, val):
    n = Node(val, self.head)
    self.head = n

Typically we combine the preceding operations into a single step, so we can write the method in a single line:

  def prepend(self, val):
    self.head = Node(val, self.head)

We can traverse a linked list using a local variable that initially points to the head of the list. In a loop, we advance this variable to point to each node in turn. Here is a method that will print all values in a LinkedList:

  def printAll(self):
    p = self.head
    while p != None:
      print(p.val)
      p = p.next

Using the preceding methods, let's create a new LinkedList, prepend some values, and print them all out:

>>> l = LinkedList()
>>> for i in range(1, 6):
...   l.prepend(i)
>>> l.printAll()
5
4
3
2
1

To append a value to a linked list, if we only have a pointer to the head (as in the LinkedList class), then we generally need to traverse to the end of the list to find the last node. We then set the last node's next pointer to a new node containing the value to append.

However, that will not work if the LinkedList is empty, since then there is no last node. In this case, we set self.head to point to a new node with the new value:

  def append(self, val):
    n = Node(val, None)
    if self.head == None:
      self.head = n
    else:
      p = self.head
      while p.next != None:
        p = p.next
      p.next = n

This method runs in O(N) in the worst case, where N is the number of elements in the list. Alternatively, if we keep a pointer to the tail (last node) in the list, then we can append in O(1). (Below, we will see a class LinkedQueue that implements this idea.)

To delete a value from a linked list, we generally need to traverse down the list to find the node p that precedes the node we wish to delete. We then assign

p.next = p.next.next

This makes the node chain exclude the node we are deleting. (And then Python's garbage collector will realize that there are no more pointers to that node, and will reclaim its memory.)

If the node we are deleting is the first in the list, then there is no preceding node, so we must handle this as a special case.

Here is a LinkedList method to delete a value from a list:

  # Delete the node (if any) with the given value.
  def delete(self, val):
    if self.head == None:
      return    # list is empty
    if self.head.val == val:
      self.head = self.head.next   # delete first node
      return
    p = self.head
    while p.next != None:
      if p.next.val == val:
        p.next = p.next.next   # delete the node after p
        return
      p = p.next

implementing a stack using a linked list

Above, we learned about stacks, which are an abstract data type with the operations push and pop. We also saw how to implement a stack using an array.

Alternatively we can implement a stack using a linked list. To accomplish this, we will create a class LinkedStack with an attribute head that points to the head of the list. Now our stack operations are quite straightforward:

class LinkedStack:
  def __init__(self):
    self.head = None
    
  def isEmpty(self):
    return self.head == None
  
  def push(self, x):
    self.head = Node(x, self.head)
  
  def pop(self):
    assert self.head != None, 'stack is empty'
    x = self.head.val
    self.head = self.head.next
    return x

Different implementations of an abstract type may have different performance characteristics. If we implement a stack using an array (i.e. a Python list), the push operation will take O(N) in the worst case, where N is the current stack size. Our linked list-based implementation performs differently: push always runs in O(1).

implementing a queue using a linked list

Above, we learned about queues, which are an abstract data type with operations enqueue and dequeue, and saw how to implement a queue using a circular array.

We may alternatively implement a queue using a linked list. To do so, we keep pointers to both the head (first node) and tail (last node) in the list. This allows us to enqueue or dequeue elements in O(1). Here is an implementation:

class LinkedQueue:
  def __init__(self):
    self.head = self.tail = None
  
  def isEmpty(self):
    return self.head == None
  
  def enqueue(self, x):
    n = Node(x, None)
    if self.head == None:
      self.head = self.tail = n
    else:
      self.tail.next = n
      self.tail = n
      
  def dequeue(self):
    assert self.head != None, 'queue is empty'
    x = self.head.val
    self.head = self.head.next
    return x