Introduction to Algorithms
Lecture 4: Notes

Many of the topics we discussed today are covered in these sections of Problem Solving with Algorithms:

Searching and Sorting
- Searching
  - The Sequential Search
  - The Binary Search (also discussed below)
- Sorting
  - The Bubble Sort
  - The Insertion Sort (also discussed below)

2. Getting Started
- 2.1 Insertion sort

Here are some additional notes.

sieve of Eratosthenes

We've seen in previous lectures that we can determine whether an integer n is prime using trial division, in which we attempt to divide n by successive integers. Because we must only check integers up to `sqrt(n)`, this primality test runs in time O(`sqrt n`).

Sometimes we may wish to generate all prime numbers up to some limit N. If we use trial division on each candidate, then we can find all these primes in time `O(N sqrt N`). But there is a faster way, using a classic algorithm called the Sieve of Eratosthenes.

It's not hard to carry out the Sieve of Eratosthenes using pencil and paper. It works as follows. First we write down all the integers from 2 to N in succession. We then mark the first integer on the left (2) as prime. Then we cross out all the multiples of 2. Now we mark the next unmarked integer on the left (3) as prime and cross out all the multiples of 3. We can now mark the next unmarked integer (5) as prime and cross out its multiples, and so on. Just as with trial division, we may stop when we reach an integer that is as large as `sqrt N`.

Here is the result of using the Sieve of Eratosthenes to generate all primes between 2 and 30:

23456789101112131415161718192021222324252627282930

We can easily implement the Sieve of Eratosthenes in Python:

# Generate all prime numbers up to (but not including) n

def sieve(n):
  isPrime = n * [True]

  for i in range(2, int(sqrt(n)) + 1):
    if isPrime[i]:
      for j in range(2 * i, n, i):
        isPrime[j] = False

  return isPrime

How long does the Sieve of Eratosthenes take to run?

The inner loop, in which we set isPrime[j] =False, runs N/2 times when we cross out all the multiples of 2, then N/3 times when we cross out multiples of 3, and so on. So its total number of iterations will be

`N(1/2 + 1/3 + 1/5 + ... + 1/p)`

where p <= `sqrt N`.

The series

`sum_(p prime) 1/p = 1/2 + 1/3 + 1/5 + …`

is called the prime harmonic series. How can we approximate its sum through a given element 1/p?

Euler was the first to demonstrate that the prime harmonic series diverges: its sum grows to infinity, through extremely slowly. Its partial sum through 1/p is close to ln (ln n):

`lim_(n->oo) [sum_(p<=n) 1/p - ln (ln n)] = 0.261...`

This shows in turn that

`N(1/2 + 1/3 + 1/5 + ... + 1/p) = N * O(log log (sqrt N)) = N * O(log log N) = O(N log log N)`

This is very close to O(N). (In fact more advanced algorithms can generate all primes through N in time O(N).)

binary search

We may wish to search for a value (sometimes called the “key”) in a sorted array. We can use an algorithm called binary search to find the key efficiently.

Binary search is related to the number-guessing game that we saw earlier. In that game, one player thinks of a number N, say between 1 and 1,000,000. If we are trying to guess it, we can first ask "Is N greater than, less than, or equal to 500,000?" If we learn that N is greater, then we now know that N is between 500,001 and 1,000,000. In our next guess we can ask to compare N to 750,000. By always guessing at the midpoint of the unknown interval, we can divide the interval in half at each step and find N in a small number of guesses.

Similarly, suppose that we are searching for the key in a sorted array of 1,000,000 elements from a[0] through a[999,999]. We can first compare the key to a[500,000]. If it is not equal, then we know which half of the array contains the key. We can repeat this process until the key is found.

To implement this in Python, we will use two integer variables lo and hi that keep track of the current unknown range. Initially lo = -1 and hi = length(a). At all times, we know that all array elements with indices <= lo are less than the key, and all elements with indices >= hi are greater than the key. That means that the key, if it exists in the array, must be in the range a[lo + 1] … a[hi – 1]. As the binary search progresses, lo and hi move toward each other. If eventually hi = lo + 1, then the unknown range is empty, meaning that the key does not exist in the array.

Here is a binary search in Python. (Recall that a Python list is actually an array.)

# Search for value k in the list a.
# Return its index, or -1 if not found.
def binarySearch(k, a):
  lo = -1
  hi = len(a)

  while hi - lo > 1:
    mid = (lo + hi) // 2
    if a[mid] == k:
      return mid
    elif a[mid] < k:
      lo = mid
    else:
      hi = mid

  return -1  # not found

Suppose that the input array has 1,000,000,000 elements. How many times might this loop iterate?

After the first iteration, the unknown interval (hi – lo) has approximately 500,000,000 elements. After two iterations, it has 250,000,000 elements, and so on. After k iterations it has 1,000,000,000 / 2^k elements. We reach the end of the loop when 1,000,000,000 / 2^k = 1, so k = log₂(1,000,000,000) ≈ 30. This means that the loop will iterate only about 30 times, which will take only a few microseconds at most.

binary search for a boundary

Suppose that we know that a given array consists of a series of values with some property P followed by some series of values that do not have that property. Then we can use a binary search to find the boundary point dividing the values with property P from those without it.

For example, suppose that we know that an array contains a series of non-decreasing integers. We might want to find the index of the first value in the array that is greater than or equal to k, for some value k. The array must contain a series of values that are less than k, followed by a series of values that are at least k. We want to find the boundary point between these series.

# Given a array of non-decreasing values, find the first one that is >= k.

def firstAtLeast(a, k):
  lo = -1
  hi = len(a)

  while hi - lo > 1:
    mid = (lo + hi) // 2
    if a[mid] < k:
      lo = mid
    else:
      hi = mid

  return hi

At every moment as the function runs:

for i ≤ lo: a[i] < k
for lo < i < hi: we don't know whether a[i] is less than or greater than k
for i ≥ hi: a[i] ≥ k

When the while loop finishes, hi == lo + 1. So there are no unknown elements, and we know that a[lo] < k and a[hi] ≥ k. And so hi is the index of the first value greater than or equal to k.

insertion sort

Bubble sort is the easiest sorting algorithm to write, but is also relatively inefficient. Let's now learn a more efficient algorithm called insertion sort.

Insertion sort is similar to how you might sort a deck of cards by hand. The sort loops through array elements from left to right. Assume that elements are indexed from 0 as in a Python list. For each element a[i], we find all elements to the left of a[i] that are greater than a[i] and shift them rightward one position. This makes room so that we can insert a[i] to the left of those elements. And now the subarray a[0..i] is in sorted order. After we repeat this for all i, the entire array is sorted.

For example, consider an insertion sort on this array:

We shift 6 rightward and insert 5 to its left. Now a[0..1] is sorted:

Now we shift 5 and 6 rightward, and insert 3 before them. Now a[0..2] is sorted:

Now we shift 3, 5, and 6 to the right, and insert 1 before them:

All elements to the left of 8 are less than it, so we can leave it in its place for the moment:

Now we shift 8 rightward and insert 7:

And so on. Here is an animation of insertion sort in action on the above array.

More concretely, to insert a[i] into the sorted subarray a[0 .. (i - 1)], insertion sort first saves the value of a[i] in a variable v. It then walks backwards through the subarray, shifting elements forward by one position as it goes. When it sees an element that is less than or equal to v, it stops, and inserts v to the right of that element. At this point the entire subarray a[0 .. i] is sorted.

Here is a Python implementation of insertion sort:

def insertionSort(a):
  for i in range(len(a)):
    v = a[i]
    j = i - 1
    while j >= 0 and a[j] > v:
      a[j + 1] = a[j]
      j -= 1
    a[j + 1] = v

What is the running time of insertion sort? In the best case, the input array is already sorted. Then no elements are shifted or modified at all and the algorithm runs in time O(n).

The worst case is when the input array is in reverse order. Then to insert each value we must shift elements to its left, so the total number of shifts is 1 + 2 + … + (n – 1) = O(n²). If the input array is ordered randomly, then on average we will shift half of the subarray elements on each iteration. Then the time is still O(n²).

Insertion sort has the same worst-case asymptotic running time as bubble sort, i.e. O(n²). But it generally outperforms bubble sort by a factor of 3 or more. It is a reasonable choice for a simple sorting algorithm when n is not large.

Introduction to Algorithms Lecture 4: Notes

sieve of Eratosthenes

binary search

binary search for a boundary

insertion sort

Introduction to Algorithms
Lecture 4: Notes