Intro to Algorithms, 2019-20
Lecture 3 – Notes

big-O notation and running time analysis

This topic is covered in the following chapters of Problem Solving with Algorithms:

3.2 What Is Algorithm Analysis?
3.3 Big-O Notation

In addition, here are my own notes on the subject.

We would like to consider how long our programs take to run as a function of their input size.

As a first example, consider this code, which computes a certain function of an input number n:

n = int(input('Enter n: '))

s = 0

for i in range(n):
  s += i

  for j in range(n):
    s += i * j

print('The answer is', s)

Here the input size is n. This code will run quickly when n is small, but as n increases the running time will certainly increase. How can we quantify this?

Here is the same code again, with comments indicating how many times each line will run:

n = int(input('Enter n: '))

s = 0               # 1 time

for i in range(n):
  s += i            # n times

  for j in range(n):
    s += i * j      # n * n times

print('The answer is', s)

Suppose that the first commented statement above runs in time a, the second runs in time b and the third runs in time c, where a, b, and c are constants representing numbers of nanoseconds. Then the total running time is a + b n + c n².

In theoretical computer science we don't care at all about the constant factors in an expression such as this. They are dependent on the particular machine on which a program runs, and may not even be so easy to determine. In fact, we don't even care about the terms (a) or (b n). That's because as n grows, these terms are dominated by the quadratic term c n², which becomes larger by an arbitrary factor. To be more precise, lim_n→∞ [c n² / (a + b n)] = ∞.

So we write:

running time = a + b n + c n² = O(n²)

Here, the O (which stands for "order") is sometimes pronounced "big O". We could give a formal mathematical definition of big O, but informally it looks like the form of an expression after discarding all constant factors and lower-order terms which are dominated by the highest-order term. For example:

3 n + 4 = O(n)

5 n² + 6 n + 4 = O(n²)

2 n + log(n) = O(n)

2ⁿ + 5 n⁴ + n = O(2ⁿ)

To know which lower-order terms we can discard we must know which functions grow faster than others. Here is a table of common growth rates in order from smallest to largest:

notation	name
O(1)	constant
O(log n)	logarithmic
O(n)	linear
O(n log n)	log linear
O(n²)	quadratic
O(n³)	cubic
O(n^k)	polynomial
O(kⁿ)	exponential
O(n!)	factorial

A running time in big-O notation is said to be asymptotic. In other words, it is not exact, but the actual running time approaches the big-O running time asymptotically to within a constant factor as N grows large.

Notice that in O(log n) above we did not specify any base for the logarithm. Are O(log₂ n) and O(log₁₀n) different orders of growth?

Actually they are the same. Recall from algebra that

log_ab · log_bc = log_ac

And so

log₂10 · log₁₀n = log₂n

This shows that log₂ n and log₁₀ n differ only by a constant factor. Big-O does not care about a constant factor, and so

O(log₂ n) = O(log₁₀ n)

Coming back to our previous example, let's make a slight change to the code:

n = int(input('Enter n: '))

s = 0

for i in range(n):
  s += i

  for j in range(i):  # previously this was range(n)
    s += i * j

print('The answer is', s)

What is its running time now?

Once again the statement "s = 0" will execute once, and "s += i" will execute n times. The statement "s += i * j" will execute 0 times on the first iteration of the outer loop (when i = 0), 1 time on the next iteration (when i = 1) and so on. The total number of times it will execute is

0 + 1 + 2 + … + (n – 1) = n (n – 1) / 2 = n² / 2 – n / 2 = O(n²)

Thus, the code's asymptotic running time is still O(n²).

This example illustrates a principle that we will see many times in the course, namely that

1 + 2 + 3 + … + N = O(N²)

running time of integer algorithms

Let's consider the running time of the algorithms on integers that we have already studied in this course.

primality testing

First let's look at primality testing using trial division, which we saw in the last lecture:

n = int(input('Enter n: '))

prime = True
i = 2
while i * i <= n:    # i.e. while i <= sqrt(n)
  if n % i == 0:
    prime = False
    break
  i += 1

if prime:
  print('n is prime')
else:
  print('not prime')

Notice that for this algorithm some values of n are harder than others. For example, if n is even then the function will exit immediately, even if n is very large. On the other hand, if n is prime then the function will need to iterate all the way up to sqrt(n) in order to determine its primality.

In general we are interested in determining an algorithm's best-case and worst-case running times as the input size n increases. For some algorithms these will be the same, and for others they will be different. The best-case running time is a lower bound on how long it might take the algorithm to run, and the worst-case running time is an upper bound.

Let's annotate the code above with the number of times each statement might run:

n = int(input('Enter n: '))

prime = True
i = 2
while i * i <= n:
  if n % i == 0:   # min = 1, max = sqrt(n) - 1
    prime = False
    break
  i += 1           # min = 0, max = sqrt(n) - 1

if prime:
  print('n is prime')
else:
  print('not prime')

The statement "ifn % i ==0" might run only once, or as many as sqrt(n) – 1 times. The number of executions of "i += 1" is similar. This function runs in O(1) in the best case, and in O(sqrt(n)) in the worst case.

Next, let's revisit our integer factorization algorithm from the last lecture:

n = int(input('Enter n: '))

i = 2
  
while i * i <= n:
  if n % i == 0:
    print(i, end = ' ')
    n //= i
  else:
    i += 1
    
print(n)

This is a bit trickier to analyze. In the worst case, n is prime, and the test 'if n % i == 0' will execute sqrt(n) – 1 times. Thus the worst-case running time is O(sqrt(n)).

Now suppose that n is a power of 2. Then the test 'if n % i == 0' will run log₂n – 1 times, and the program will run in O(log₂ n) = O(log n). In fact this is the best-case running time, though we will not prove that formally here.

greatest common divisor

For integers a and b, the greatest common divisor of a and b, also written as gcd(a, b), is the largest integer that evenly divides both a and b. For example:

gcd(20, 6) = 2
gcd(12, 18) = 6
gcd(35, 15) = 5

The greatest common divisor is useful in various situations, such as simplifying fractions. For example, suppose that we want to simplify 35/15. gcd(35, 15) = 5, so we can divide both the numerator and denominator by 5, yielding 7/3.

Naively, we can find the gcd of two values by trial division:

a = int(input('Enter a: '))
b = int(input('Enter b: '))

for i in range(min(a, b), 0, -1):
  if a % i == 0 and b % i == 0:
    break

print('The gcd is', i)

But this may take time O(N), where N = min(a, b). We'd like to be more efficient, especially since in some situations (e.g. cryptography) we may want to take the gcd of numbers that are very large.

Another way to find the greatest common divisor of two numbers is to factor the numbers into primes, then look for common primes. For example consider finding gcd(252, 90). 252 = 9 * 28 = 2² * 3² * 7¹. And 90 = 9 * 10 = 2¹ * 3² * 5¹. So the common primes are 2¹* 3²= 18, and we have gcd(252, 90) = 18.

Euclid's algorithm

Euclid’s algorithm is a much more efficient way to find the gcd of two numbers than prime factorization. It is based on the fact that for all positive integers a and b, gcd(a, b) = gcd(b, a mod b). (We will not prove this here, though the proof is not difficult.) So, for example,

gcd(252, 90)
= gcd(90, 72)
= gcd(72, 18)
= gcd(18, 0)
= 18

We can implement Euclid's algorithm in Python like this:

a = int(input('Enter a: '))
b = int(input('Enter b: '))

while b > 0:
  a, b = b, a % b

print('The gcd is', a)