Today: modules, how the internet works

Explain Glossed Over Lines in
wordcount.py / pylibs.py

There are lines of Python code we have glossed over. Piece by piece, we will fill these in.

Look at Top of File

Today we'll explain the import sys

#!/usr/bin/env python3

"""
Stanford CS106A Pylibs Example
Nick Parlante
"""

import sys
import random

def read_terms(filename):
...

`import sys`

Import a module named "sys"
Later lines can call the functions defined in sys
Let's talk about modules...

Modules and Modern Coding

Modules hold code for common problems, ready for your code to use. We say that you build your code "on top of" the libraries. Modern coding is part custom, and part building on top of module code.

alt:your code built on top of modules like sys

We ♥ Modules

Question while coding:
Is part of this solved in a module already?
Using module code you didn't have to write is very attractive
This is kind of a no-brainer case to make!
Somebody else wrote it, you can just use it
It's well tested
It has real documentation - aka docs
Your teammates may already be familiar with it
CS106A we see this "module" theme a little
CS106A needs to cover fundamentals
loops, strs, dicts, files ..
Courses beyond CS106A, likely use modules more

Module = Name + Code + Docs

A module has a name, e.g. "math"
A module contains functions, solving common problems
A module also has documentation "docs" explaining the use of its functions
e.g. "math" module contains math functions
e.g. "random" module contains functions for pseudo-random numbers
e.g. "urllib" module contains functions for urls and web requests

Step 1: `import math`

To use a module, include a import math line
Import the module by its name
Typically these are grouped near the top of your file

Step 2: `math.sqrt(2)`

On later lines, refer to functions in the module with a dot
e.g. math.sqrt(2)
e.g. random.randrange(10)
Readable: in this way, it's clear when calling a function..
that it is coming from that module
There are other ways of doing import
This form is simplest: import math .. math.sqrt()

>>> import math
>>> math.sqrt(2)  # call sqrt() fn
1.4142135623730951
>>> math.sqrt

>>> 
>>> math.log(10)
2.302585092994046
>>> math.pi       # constants in module too
3.141592653589793

Quit and restart the interpreter without the import, see common error:

>>> # quit and restart interpreter
>>> math.sqrt(2)  # OOPS forgot the import
Traceback (most recent call last):
NameError: name 'math' is not defined
>>>
>>> import math
>>> math.sqrt(2)  # now it works
1.4142135623730951

Module = Dependency

When you write code using a module
Your code now depends on that module's existence
If that module disappeared, your code would stop working

1. "Standard" Modules — Fine

Standard = included/maintained as part of Python3 install
These are the best modules to use
Can rely on this module now and in the future
Very rare for a module to be dropped
No separate install is required
The standard module is installed when python is installed

Many Standard Modules

Do not: memorize whole list of modules
Do: check the list for help when starting a project
Standard Python Modules List
A few examples...
csv module to read CSV files, such as produced by Excel
datetime module of calendar functions
zipfile module for reading/creating .zip files
urllib module for making web http requests from Python

2. Non-Standard "pip" Modules — Depends

Other modules are valuable but they are not a standard part of Python. For code using non-standard module to work, the module must be installed on that computer via the "pip" Python tool. e.g. for homeworks we had you pip-install the "Pillow" module with this command:

$ python3 -m pip install Pillow
..prints stuff...
Successfully installed Pillow-5.4.1

A non-standard module can be great, although the risk is harder to measure. The history thus far is that popular modules continue to be maintained. Sometimes the maintenance is picked up by a different group than the original module author. A little used module is more risky.

Aside: Security of Pip Module

When you install a module on your machine from somewhere - you are trusting that code to run on your machine. In very rare cases, bad guys have tampered with modules to include malware in the module. Be more careful if installing a little used module. In contrast, python itself and its standard modules like urllib are very safe as so many people use them.

Module Docs

Every module has formal "documentation" - "docs"
Explain what its functions do
The "abstraction" of each function
What it does
How to call it
Demo web search: "python math module"
python.org - the official home of python docs
Official python.org math docs
Watch out for SEO
Search Engine Optimization
Some possibly lame site tries to get a better search ranking

Hacker: Use dir() and help() (optional)

Feel like a hacker, use dir() and help() on module
In the interpreter >>>
dir(module) - shows a list of all the defs in the module
help(module.fn) - shows some help text for that function
The """Pydoc""" we write to describe each function
That Pydoc is what help() returns (demo later)
Names starting with double-underbar = internal thing you probably do not need

>>> import math
>>> dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'remainder', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'tau', 'trunc']
>>>
>>> help(math.sqrt)
Help on built-in function sqrt in module math:

sqrt(x, /)
    Return the square root of x.
>>>
>>> help(math.cos)
Help on built-in function cos in module math:

cos(x, /)
    Return the cosine of x (measured in radians).

How to Create Your Own Module?

You already have! A regular old foo.py file is a module.

wordcount.py Is a Module

How hard is it to write a module? Not hard at all. A regular Python file we have written works as a module too with whatever defs the foo.py file has.

alt: wordcount.py is a module named wordcount

Consider the file wordcount.py in wordcount.zip

Forms a module named wordcount

Suppose you have built some useful functions
Someone else in your lab wants to use them....
Them pasting in their own copy is not ideal
What does a module contain?
We have wordcount.py
python3 wordcount.py - runs main()
wordcount.py is also a module named just "wordcount"
Think of all the defs in wordcount: read_counts(), clean(), print_counts(),
import works on wordcount (in the same directory)
Access functions as module.xxx just like usual
Run python interpreter in wordcount directory to try this
Try importing wordcount, calling the read_counts() function
Call wordcount.clean()

Try this demo in the wordcount directory

>>> # Run interpreter in wordcount directory
>>> import wordcount
>>>
>>> wordcount.read_counts('test1.txt')
{'a': 2, 'b': 2}

A module/file contains many defs
Can import a module/file, call its defs:
module.fn_name()
Style: for a function to be usable from another module...
it should take in data as parameters and return a value
i.e. black box style
we've done this all along, see now the bigger picture
Babygraphics project:
treats babynames.py as a module
import babynames
calls babynames.read_files()

dir() and help() work on wordcount Too

Look at wordcount.py, look at the functions
dir() and help() work here too
See where the """Pydoc""" goes!

>>> dir(wordcount)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'clean', 'main', 'print_counts', 'print_top', 'read_counts', 'sys']
>>> 
>>> help(wordcount.read_counts)

read_counts(filename)
    Given filename, reads its text, splits it into words.
    Returns a "counts" dict where each word
    ...

How babygraphics.py Used babynames.py

# 1. In the babygraphics.py file
# import the babynames.py file in same directory
import babynames

...

    # 2. Call the read_files() function                                                                  
    names = babynames.read_files(FILENAMES)

Module Example: urllib

Look at how web pages work
Then play with urllib

How Does The Web Work?

alt: web client makes request to server, gets back HTML response, browser draws the HTML

The web browser app has url ("client" side)
Web server is running on some machine ("server" side)
Browser sends GET request to server
Server gets request, sends back HTML response data
HTML is a text code
Browser "renders" the HTML on screen
Common request/response data types: HTML JPEG PNG GIF SVG
Video formats MPEG4 h.265 (crazy patent problems)
New open format "AV1" .. you heard it here first!
Demo:
Visit python.org or sfgate.com or whatever
Right-click on page, "View Source" to see the HTML text that makes a web page
Think of all the surfing you have done .. HTML code defines each page

HTML

Here is the HTML code for is plain text with a bolded word in it, tags like <b> mark up the text.

This <b>bolded</b> text

HTML Experiment - View Source

Go to python.org. Try view-source command on this page (right click on page). Search for a word in the page text, such as "whether" .. to find that text in the HTML code.

Thing of how many web pages you have looked at - this is the code behind those pages. It's a text format! Lines of unicode chars!

Web Page - HTML - Python

Every web page you've ever seen is defined by this HTML text behind the scenes. Hmm. Python is good at working with text.

urllib Demo

Python code to request an HTML page by url
urllib - making requests to a server, getting back data
An example of using a standard module
docs - urllib.request docs on python.org
Makes a URL look like a local file mostly
Read the text of the web page
Use s.find() to display a fragment of it

(See copy of these lines below suitable for copy/paste yourself.)

>>> import urllib.request
>>> f = urllib.request.urlopen('http://www.python.org/')
>>> text = f.read().decode('utf-8')
>>> text.find('Whether')
26997
>>> text[26997:27100]
"Whether you're new to programming or an experienced developer, it's easy to learn and use Python"

Here is the above Python lines, suitable for copy paste:

import urllib.request
f = urllib.request.urlopen('http://www.python.org/')
text = f.read().decode('utf-8')

f.read() - reads all the bytes in one call
decode('utf-8') - decode raw bytes, returning unicode string
Does not always work, they may be blocking python
Also some web sites convey their HTML in a different way
Can try http: or https:

Data From the Web vs. Files

Very often data is drawn from file, into your program
New picture: data is drawn from a URL
urllib packages up a few functions for this
Handling the details
A nice example of a module

How Does The Internet work?

Now does the Internet work? I pulled these slides together I had laying around from another class. Neat to see how something you use every day works.

Internet - TCP/IP Standards

Internet - world-wide network built on open standards
Open standards: TCP, IP, HTML, JPEG, PNG
These are vendor neutral - unbelievably successful pattern
-No license required
-No patents required
-Brings in a lot of participation
Vendor-proprietary "internet" attempts pre-TCP/IP failed
One of the most valuable inventions of recent history
-Important lesson about open standards

1. Packets

Say we have 50KB image.jpg
50,000 bytes
How to send the image.jpg on the wire?
Use packets
Divide bytes of image.jpg into packets
Say each packet is 1500 bytes (a common size)
Then image.jpg divides into about 32 packets
Network transmits one packet at a time

packets transmitted on the wire

Sending One Packet

An over simplification
Look at transmission of one 1500 byte packet
Each byte is 8 bits, e.g. 0 1 1 0 1 0 1 0
-Go through all the bits, from left to right
-For each 1, put 3 volts on the wire
-For each 0, put 0 volts on the wire
Modern ethernet is runs at at least 100 million bits per second
-aka 100 mbps
In reality, there is more complex scheme than 3-volts/0-volts
Wifi sends the 0/1 data on a radio channel
-The encoding is more complex
-But you get the idea

IP Address

Every computer on the internet has an IP address
Here looking at IP v4 addresses, v6 addrs some day in the future
e.g. 171.64.64.166
IP addr is exactly 4 bytes (4 1-byte numbers)
Left part encodes "neighborhood" on internet
Just like phone, 650-725-0000
e.g. 171.64.xxx.xxx generally Stanford campus
e.g. 171.64.64.xxx my floor of Gates building
Cannot just make up an IP addr, depends on location

Sandra Bullock Blooper

TCP/IP blooper in this video of "The Net"...

video

Blooper: in the video the IP address is shown as 75.748.86.91 - not a valid IP address! Each number should be 1 byte, 0..255

Router

A computer typically connects to a router for service
-We'll say the router is "upstream"
The router provides the internet service, handling the computer's packets
Typically when you connect to wi-fi, you get a router too
The router will assign a temporary IP address to the computer
The router typically has the IP address ending with ".1"

The most common way for a computer to be "on the internet" is to establish a connection with a "router" which is already on the internet. The computer establishes a connection via, say, wifi to communicate packets with the router. The router is "upstream" of the computer, connecting the computer to the whole internet.

IP Packet - Hopping

For a computer to send a packet:
Label it with the IP address of the source and destination
Send it to local router
Router in turn sends it on to bigger router
The packet proceeds .. hop hop hop .. over to final destination

Lots of Packet Hops

The packet is passed from router to router - called a "hop". There might be 10 or 20 hops in a typical internet connection.

packet proceeds by multiple hops

Routers - Decentralized and Cooperative

Each router knows enough to figure the next hop, not the whole route
There is no "center" of the internet that knows everything
The initiating computer does not typically know anything, delegating to its router
"Core" routers, towards the middle, bigger, fancier, more connections
Routers measure connection functionality/breakage all the time
-Choose alternative routes in real time from breakage, congestion
-e.g. a backhoe breaks an important link, routers re-route in realtime
Routers are a distributed, collaborative system
-Each does its part, cooperatively solving the whole thing
-VS. centralized/top-down system

The routing of a packet from your computer is like a capillary/artery system .. your computer is down at the capillary level, your packet gets forwarded up to larger and larger arteries, makes its way over to the right area, and then down to smaller and smaller capillaries again, finally arriving at its destination.

Note: Special "Local" IP Addresses

Note that 10.x.x.x and 192.168.x.x addresses are special "local" IP addresses
These addresses are not valid out on the internet at large
They are used within an organization but not outside it
These are translated to a real IP addr as a packet makes its way
Frequently given out by Wi-Fi routers .. why I mention them

What Does it Mean to Be On the Internet?

On the internet - e.g. connect to a Wi-Fi router
1. Computer connects to an upstream router to handle traffic. Most Wi-Fi access points combine Wi-Fi radios and a router.
2. The router typically gives the computer an IP address to use
The computer cannot pick an arbitrary IP address, since the left part of the address depends on the location on the internet ... details known by the router
Also, you don't want to pick an IP address in use by someone else, so the router gives you a known good one
3. DHCP "Dynamic Host Configuration Protocol" - automatically configure network settings to work locally. Computers very often use this feature to get needed network configuration from the router automatically.

So what does it mean for a computer to be on the internet? Typically it means the computer has established a connection with a router. The commonly used DHCP standard (Dynamic Host Configuration Protocol), facilitates connecting to a router; establishing a temporary connection, and the router gives your computer an IP address to use temporarily. Typically DHCP is used when you connect to a Wi-Fi access point.

Demo: Your Internet

Bring up the networking control panel of your computer. It should show what IP address you are currently using and the IP address of your router. You will probably see some text mentioning that DHCP is being used. Your computer will likely have a local IP address, just used while you're in this room.

Demo: Ping

"Ping" is an old and very simple internet utility. Your computer sends a "ping" packet to any computer on the internet, and the computer responds with a "ping" reply (not all computers respond to ping). In this way, you can check if the other computer is functioning and if the network path between you and it works. As a verb, "ping" is also used in regular English this way .. not sure if that's from the internet or the other way around.

Experiment: Most computers have a ping utility, or you can try "ping" on the command line (works on the Mac, Windows, and Linux). Try pinging www.google.com or pippy.stanford.edu. Not all computers respond to ping. Type ctrl-c to terminate ping.

Milliseconds fraction of a second used for the packet to go and come back. 1 ms = 1/1000 of a second. Different from bandwidth, this "round trip delay".

Here I run the "ping" program for a few addresses, see what it reports

$ ping www.google.com  # I type in a command here
PING www.l.google.com (74.125.224.144): 56 data bytes
64 bytes from 74.125.224.144: icmp_seq=0 ttl=53 time=8.219 ms
64 bytes from 74.125.224.144: icmp_seq=1 ttl=53 time=5.657 ms
64 bytes from 74.125.224.144: icmp_seq=2 ttl=53 time=5.825 ms
^C                            # Type ctrl-C to exit
--- www.l.google.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.657/6.567/8.219/1.170 ms
$ ping pippy.stanford.edu
PING pippy.stanford.edu (171.64.64.28): 56 data bytes
64 bytes from 171.64.64.28: icmp_seq=0 ttl=64 time=0.686 ms
64 bytes from 171.64.64.28: icmp_seq=1 ttl=64 time=0.640 ms
64 bytes from 171.64.64.28: icmp_seq=2 ttl=64 time=0.445 ms
64 bytes from 171.64.64.28: icmp_seq=3 ttl=64 time=0.498 ms
^C
--- pippy.stanford.edu ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.445/0.567/0.686/0.099 ms

Traceroute

On the mac, just open a terminal as shown below
How to do: traceroute on windows
See series of hops
First few hops in your IP neighborhood
Farther away hops .. more milliseconds
Note New York, London .. big jump in milliseconds
Hops does not go up linearly with distance
East bay - 30 miles from Stanford - 11 hops
Russia - 10000 miles from Stanford - 19 hops

Traceroute is a program that will attempt to identify all the routers in between you and some other computer out on the internet - demonstrating the hop-hop-hop quality of the internet. Most computers have some sort of "traceroute" utility available if you want to try it yourself (not required). On Windows it's called "tracert" in Windows Power Shell, and it does not suppor the "-q 1" option below, but otherwise works fine.

Some routers are visible to traceroute and some not, so it does not provide completely reliable output. However, it is a neat reflection of the hop-hop-hop quality of the internet.

Traceroute codingbat.com

codingbat.com is housed in the east bay - 13 hops we see here. The milliseconds listed is the round-trip delay.

$ traceroute -q 1 codingbat.com
traceroute to codingbat.com (173.255.219.70), 64 hops max, 52 byte packets
 1  rt-ac68u-b3f0 (192.168.1.1)  7.152 ms
 2  96.120.89.177 (96.120.89.177)  9.316 ms
 3  24.124.159.189 (24.124.159.189)  9.638 ms
 4  be-232-rar01.santaclara.ca.sfba.comcast.net (162.151.78.253)  9.775 ms
 5  be-39931-cs03.sunnyvale.ca.ibone.comcast.net (96.110.41.121)  31.753 ms
 6  be-3202-pe02.529bryant.ca.ibone.comcast.net (96.110.41.214)  10.273 ms
 7  ix-xe-0-1-1-0.tcore1.pdi-paloalto.as6453.net (66.198.127.33)  10.570 ms
 8  if-ae-2-2.tcore2.pdi-paloalto.as6453.net (66.198.127.2)  11.344 ms
 9  if-ae-5-2.tcore2.sqn-sanjose.as6453.net (64.86.21.1)  13.555 ms
10  if-ae-1-2.tcore1.sqn-sanjose.as6453.net (63.243.205.1)  11.583 ms
11  216.6.33.114 (216.6.33.114)  11.938 ms
12  if-2-4.csw6-fnc1.linode.com (173.230.159.87)  14.833 ms
13  li229-70.members.linode.com (173.255.219.70)  11.549 ms

Traceroute Serbia

A random Serbian address - 31 hops - the farthest thing I could fine. See the extra delay where the packets go across the Atlantic - I'm guessing hop 16. The names there may refer to Amsterdam and France. Note that the packets are going at a fraction of the speed of light here - a fundamental limit of how quickly you can get a packet across the earth.

Or try: www.fu-berlin.de

May hang at the end, but we can at least see the early hops.

$ traceroute -q 1 yujor.fon.bg.ac.rs
traceroute to hostweb.fon.bg.ac.rs (147.91.128.13), 64 hops max, 52 byte packets
 1  rt-ac68u-b3f0 (192.168.1.1)  9.136 ms
 2  96.120.89.177 (96.120.89.177)  9.608 ms
 3  24.124.159.189 (24.124.159.189)  20.184 ms
 4  be-232-rar01.santaclara.ca.sfba.comcast.net (162.151.78.253)  15.058 ms
 5  be-39911-cs01.sunnyvale.ca.ibone.comcast.net (96.110.41.113)  11.050 ms
 6  be-3411-pe11.529bryant.ca.ibone.comcast.net (96.110.33.94)  11.294 ms
 7  be3111.ccr31.sjc04.atlas.cogentco.com (154.54.11.5)  10.420 ms
 8  be2379.ccr21.sfo01.atlas.cogentco.com (154.54.42.157)  20.021 ms
 9  be3110.ccr32.slc01.atlas.cogentco.com (154.54.44.142)  37.200 ms
10  be3037.ccr21.den01.atlas.cogentco.com (154.54.41.146)  36.318 ms
11  be3035.ccr21.mci01.atlas.cogentco.com (154.54.5.90)  49.991 ms
12  be2831.ccr41.ord01.atlas.cogentco.com (154.54.42.166)  66.591 ms
13  be2718.ccr22.cle04.atlas.cogentco.com (154.54.7.130)  67.178 ms
14  be2993.ccr31.yyz02.atlas.cogentco.com (154.54.31.226)  77.369 ms
15  be3260.ccr22.ymq01.atlas.cogentco.com (154.54.42.90)  86.026 ms
16  be3042.ccr21.lpl01.atlas.cogentco.com (154.54.44.161)  152.559 ms
17  be2183.ccr42.ams03.atlas.cogentco.com (154.54.58.70)  161.324 ms
18  be2813.ccr41.fra03.atlas.cogentco.com (130.117.0.122)  164.945 ms
19  be2960.ccr22.muc03.atlas.cogentco.com (154.54.36.254)  172.507 ms
20  be2974.ccr51.vie01.atlas.cogentco.com (154.54.58.6)  197.670 ms
21  be3463.ccr22.bts01.atlas.cogentco.com (154.54.59.186)  181.075 ms
22  be3261.ccr31.bud01.atlas.cogentco.com (130.117.3.138)  184.336 ms
23  be2246.rcr51.b020664-1.bud01.atlas.cogentco.com (130.117.1.14)  189.231 ms
24  149.6.182.114 (149.6.182.114)  182.364 ms
25  amres-ias-amres-gw.bud.hu.geant.net (83.97.88.6)  191.607 ms
26  amres-mpls-core----amres-ip-core-amres-ip.amres.ac.rs (147.91.5.144)  187.181 ms
27  *
28  stanica-134-241.fon.bg.ac.rs (147.91.134.241)  204.945 ms
29  stanica-134-250.fon.bg.ac.rs (147.91.134.250)  192.673 ms
30  stanica-134-250.fon.bg.ac.rs (147.91.134.250)  193.978 ms
31  stanica-134-250.fon.bg.ac.rs (147.91.134.250)  193.032 ms !Z

TCP/IP Summary Picture

packet hopping across many routers

Explain Glossed Over Lines inwordcount.py / pylibs.py