Some of our topics this week are covered in the following chapters of Problem Solving with Algorithms:
3.3 Big-O Notation
The Fundamental Theorem of Arithmetic states that
Every positive integer has a unique prime factorization.
For example:
24 = 2 ⋅ 2 ⋅ 2 ⋅ 3 = 23 ⋅ 31
30 = 2 ⋅ 3 ⋅ 5 = 21 ⋅ 31 ⋅ 51
64 = 2 ⋅ 2 ⋅ 2 ⋅ 2 ⋅ 2 ⋅ 2 = 26
100 = 2 ⋅ 2 ⋅ 5 ⋅ 5 = 22 ⋅ 52
You can see a proof of this theorem (which is actually not completely trivial) in an introductory number theory course.
We can use trial division to factor an integer. This is similar to our primality testing algorithm, which also used trial division. Given an integer N, we first try dividing N by 2, then by 3 and so on. If n is actually divisible by a number such as 3, then we must repeatedly attempt to divide by that same number. since the same prime factor may appear multiple times in the factorization.
Here is a first program for prime factorization:
n = int(input('Enter n: '))
i = 2
s = ''
while i < n:
if n % i == 0:
s = s + str(i) + ' '
n = n // i
else:
i += 1
s = s + str(n)
print(s)Here's the program in action:
Enter n: 60 2 2 3 5
Study the program to understand how it works. The program divides out all values that are less than n. Once they are all gone, the remaining value of n is the last prime factor, so we print it at the end.
Note that the program will attempt to divide by non-prime factors, such as when i = 4. But n will never be divisible by such values. That's because any non-prime factor i is itself the product of two (or more) smaller primes, and we have already divided those primes out of n, so n cannot possibly be divisible by i.
The program works, but is inefficient because it potentially tests all values from 1 to n. Just as in our primarily testing algorithm, we can make the program much more efficient by stopping our loop once i reaches sqrt(n).
We can see that this is valid using the same argument as in the last lecture. Once again, if ab = n for integers a and b, then we must have either a ≤ sqrt(n) or b ≤ sqrt(n). Proof: Suppose that a > sqrt(n) and b > sqrt(n). Then ab > sqrt(n) ⋅ sqrt(n) = n, a contradiction. So either a ≤ sqrt(n) or b ≤ sqrt(n).
It follows that if we have tested all the values from 2 through sqrt(n) and none of them divide n, then n must be prime. Therefore we can end the loop and simply print n itself, which must be the last prime factor.
Here is the updated program:
n = int(input('Enter n: '))
i = 2
s = ''
while i * i <= n:
if n % i == 0:
s = s + str(i) + ' '
n = n // i
else:
i += 1
s = s + str(n)
print(s)You should all have seen the concept of limits in your high school mathematics courses. In computer science we are often concerned with limits as a variable goes to infinity. For example:
limx => ∞ (1 / x) = 0
limx => ∞ 3x / (x + 1) = 3
limx => ∞ (x + 2) / (sqrt(x) + 3) = ∞
We will say that a function f(n) grows faster than a function g(n) if the limit of f(n) / g(n) as n approaches ∞ is ∞. In other words, the ratio f(n) / g(n) becomes arbitrarily large as n increases. For example, n2 grows faster than 10n, since
limn => ∞ (n2 / 10n) = limn => ∞ (n / 10) = ∞
Here is a list of some common functions of n in increasing order. Each function in the list grows faster than all of its predecessors:
1
log (log n)
log n
sqrt(n)
n
n log n
n sqrt(n)
n2
n3
n4
2n
3n
n!
nn
We will define big-O notation as follows:
f(n) = O(g(n)) if limn => ∞ f(n) / g(n) = C for some 0 < C < ∞
(This definition is actually a bit of a simplification since it does not cover cases in which the limit does not exist. However, it is adequate for our purposes for the moment. You may see a more general definition later in this class, or in ADS 1 next semester.)
For example:
3 n2 + 7 n + 16 = O(n2) because limn => ∞ (3n2 + 7n + 16) / n2 = 3
5 n3 log n + n log n + 10 = O(n3 log n)
nn + n! + 2n = O(nn)
100 = O(1)
We see that we can write a function in big-O notation by discarding all lower-order terms, plus any constant factor.
We would like to consider how long our programs take to run as a function of their input size.
As a first example, consider this code, which computes a certain function of an input number n:
n = int(input('Enter n: '))
s = 0
for i in range(n):
s += i
for j in range(n):
s += i * j
print('The answer is', s)Here the input size is n. This code will run quickly when n is small, but as n increases the running time will certainly increase. How can we quantify this?
Here is the same code again, with comments indicating how many times each line will run:
n = int(input('Enter n: '))
s = 0 # 1 time
for i in range(n):
s += i # n times
for j in range(n):
s += i * j # n * n times
print('The answer is', s)Suppose that the first commented statement above runs in time a, the second runs in time b and the third runs in time c, where a, b, and c are constants representing numbers of nanoseconds. Then the total running time is a + b n + c n2.
In theoretical computer science we don't care at all about the constant factors in an expression such as this. They are dependent on the particular machine on which a program runs, and may not even be so easy to determine. So we write:
running time = a + b n + c n2 = O(n2)
Coming back to our previous example, let's make a slight change to the code:
n = int(input('Enter n: '))
s = 0
for i in range(n):
s += i
for j in range(i): # previously this was range(n)
s += i * j
print('The answer is', s)What is its running time now?
Once again the statement "s = 0" will execute once, and "s += i" will execute n times. The statement "s += i * j" will execute 0 times on the first iteration of the outer loop (when i = 0), 1 time on the next iteration (when i = 1) and so on. The total number of times it will execute is
0 + 1 + 2 + … + (n – 1) = n (n – 1) / 2 = n2 / 2 – n / 2 = O(n2)
Thus, the code's asymptotic running time is still O(n2).
This example illustrates a principle that we will see many times in the course, namely that
1 + 2 + 3 + … + N = O(N2)
Let's consider the running time of the algorithms on integers that we have already studied in this course.
First let's look at primality testing using trial division, which we saw in the last lecture:
n = int(input('Enter n: '))
prime = True
i = 2
while i * i <= n: # i.e. while i <= sqrt(n)
if n % i == 0:
prime = False
break
i += 1
if prime:
print('n is prime')
else:
print('not prime')Notice that for this algorithm some values of n are harder than others. For example, if n is even then the function will exit immediately, even if n is very large. On the other hand, if n is prime then the function will need to iterate all the way up to sqrt(n) in order to determine its primality.
In general we are interested in determining an algorithm's best-case and worst-case running times as the input size n increases. For some algorithms these will be the same, and for others they will be different. The best-case running time is a lower bound on how long it might take the algorithm to run, and the worst-case running time is an upper bound.
Let's annotate the code above with the number of times each statement might run:
n = int(input('Enter n: '))
prime = True
i = 2
while i * i <= n:
if n % i == 0: # min = 1, max = sqrt(n) - 1
prime = False
break
i += 1 # min = 0, max = sqrt(n) - 1
if prime:
print('n is prime')
else:
print('not prime')
The statement "if
n % i == 0"
might run only once, or as many as sqrt(n) – 1 times. The number of
executions of "i += 1" is similar. This
function runs in O(1) in the best case, and in O(sqrt(n)) in the
worst case.
Next, let's revisit our integer factorization algorithm:
n = int(input('Enter n: '))
i = 2
while i * i <= n:
if n % i == 0:
print(i, end = ' ')
n //= i
else:
i += 1
print(n)This is a bit trickier to analyze. In the worst case, n is prime, and the test 'if n % i == 0' will execute sqrt(n) – 1 times. Thus the worst-case running time is O(sqrt(n)).
Now suppose that n is a power of 2. Then the test 'if n % i == 0' will run log2 n – 1 times, and the program will run in O(log2 n) = O(log n). In fact this is probably the best-case running time, though we will not prove that formally here.
For integers a and b, the greatest common divisor of a and b, also written as gcd(a, b), is the largest integer that evenly divides both a and b. For example:
gcd(20, 6) = 2
gcd(12, 18) = 6
gcd(35, 15) = 5
The greatest common divisor is useful in various situations, such as simplifying fractions. For example, suppose that we want to simplify 35/15. gcd(35, 15) = 5, so we can divide both the numerator and denominator by 5, yielding 7/3.
Naively, we can find the gcd of two values by trial division:
a = int(input('Enter a: '))
b = int(input('Enter b: '))
for i in range(min(a, b), 0, -1):
if a % i == 0 and b % i == 0:
break
print('The gcd is', i)But this may take time O(N), where N = min(a, b). We'd like to be more efficient, especially since in some situations (e.g. cryptography) we may want to take the gcd of numbers that are very large.
Another way to find the greatest common divisor of two numbers is to factor the numbers into primes. For every prime p, if the prime factorization of a includes pi, and the prime factorization of b includes pj, then the prime factorization of gcd(a, b) will include pmin(i, j). (In other words, the prime factorization of gcd(a, b) is the intersection of the prime factorizations of a and b.)
For example consider finding gcd(252, 90). 252 = 9 * 28 = 22 * 32 * 71. And 90 = 9 * 10 = 21 * 32 * 51. So the common primes are 21 * 32 = 18, and we have gcd(252, 90) = 18.
Euclid’s algorithm is a much more efficient way to find the gcd of two numbers than prime factorization. It is based on the fact that for all positive integers a and b, gcd(a, b) = gcd(b, a mod b). (We will not prove this here, though the proof is not difficult.) So, for example,
gcd(252, 90)
=
gcd(90, 72)
= gcd(72, 18)
= gcd(18, 0)
= 18
We can implement Euclid's algorithm in Python like this:
a = int(input('Enter a: '))
b = int(input('Enter b: '))
while b > 0:
a, b = b, a % b
print('The gcd is', a)In all the examples of gcd(a, b) above we had a > b. But note that this function works even when a < b ! For example, if we call gcd(5, 15), then on the first iteration of the while loop we have a % b = 5 mod 15 = 5. So then we assign a = 15 and b = 5, and further iterations continue as if we had called gcd(15, 5).
We have seen how to compute the greated common divisor of two integers A related concept is the least common multiple of two integers p and q, which is the smallest integer that is divisible by both p and q. For example,
lcm(60, 90) = 180
As with the gcd, we can find the least common multiple of two numbers by computing their prime factorizations. For every prime p, if the prime factorization of a includes pi, and the prime factorization of b includes pj, then the prime factorization of gcd(a, b) will include pmax(i, j). (In other words, the prime factorization of gcd(a, b) is the union of the prime factorizations of a and b.)
How can we compute a least common multiple efficiently? Here is a useful fact: for all integers and b,
gcd(a, b) · lcm(a, b) = a · b
Why is this true? Here is an informal argument. First observe that for any integers i and j, trivially
min(i, j) + max(i, j) = i + j
Now, given integers a and b, consider any prime p. Suppose that the prime factorization of a contains pi, and the prime factorization of b contains pj. Then we have already seen that the prime factorization of gcd(a, b) contains pmin(i, j), and the prime factorization of lcm(a, b) contains pmax(i, j). Therefore the prime factorization of the product gcd(a, b) · lcm(a, b) will contains pmin(i, j) + max(i, j) = pi + j, which is also present in the prime factorization of the product a · b. Since this is true for all primes in the factorizations, we have gcd(a, b) · lcm(a, b) = a · b.
Now we can rearrange the formula above to give us lcm(a, b):
lcm(a, b) = a · b / gcd(a, b)
We can compute the gcd efficiently using Euclid's algorithm, so this formula gives us an efficient way to compute the lcm.