Today: realistic challenging parse/loop examples, while-True if-break

Iowa Caucus App Debacle

Ripped from the headlines: Iowa App Debacle. Lots of press about a failed software project. What is the lesson here? Have more time to finish? That is not the real insight.

Iron Triangle of Project Planning

There are 3 dimensions trying to complete a project. You specify 2, but then the third is determined by the iron triangle. Or in short "pick 2". There are various phrasings, but here is a version with terms for a software project:

Time - how much time you have
Features - how many things it does (aka "scope")
Quality - how well it's going to work

Suppose you have a long, neat list of features. And you have not enough time. What happens? Quality comes in terrible.

Same scenario, and you are the decision make, and it's vital that quality is high. What do you do? Cut down the feature list. Make the list of features more lame, but with a chance of good quality. Put another way: there is a tradeoff between features and quality when time is fixed.

In the Iowa case, I've read that it was supposed to OCR pictures of the voting sheets. Well, if the schedule slips and there's only a month left, drop that feature and just do the critical pieces. Process the pictures by hand or something later, or maybe there's no pictures at all. Triage down what needs to get done with a clear eye on the schedule.

Leadership is owning the decision to make a less-featured, more lame app in order to hit the time/quality benchmarks. It's a trap to keep the feature list up and pray that the quality will break you way this time. That way lays the rocky shoals of software project disaster!

Strings and Return - Q1

We have written many black-box functions: input = parameters, output = return. Return is how all functions work, even built-in ones like str.upper(). How to use the return value? Consider these lines of code. What is s after this runs?

s = 'Hello'
s.upper()
# what is s now?

Solution

s is still 'Hello', it was never changed

# s.upper() **returns** the new value, but the code does not use it
# so it is dropped. That return values are dropped if not used is an
# unintuitive aspect of code

Change s - Q2

The function call, e.g. s.upper(), returns a result. The calling code needs to catch or use that result somehow.

Q2: How to write the above code to change s to be the upper case form?

Solution

s = s.upper()

# or could use a different variable to hold the changed form, leaving
# s with its original value
up = s.upper()

Last Time: all_brackets

From last-time. Try all_brackets() function.

> list/parse problems

Let's work out the code for all_brackets

Input case: 'xx[abc]xx[42]xx' -> ['abc', '42']

What is the code for all_brackets, starting with the following code? Standard CS106A steps - diagram an input case, introduce vars left + right on the diagram to work out the details. Run it.

def all_brackets(s):
    search = 0
    result = []
    while search < len(s):
        left = s.find('[', search)
        if left == -1:
            break

        # Your code here





    return result

all_brackets() Solution

def all_brackets(s):
    search = 0
    result = []
    while search < len(s):
        left = s.find('[', search)
        if left == -1:
            break
        # Your code here
        pass
        right = s.find(']', left)  # or left+1
        if right == -1:
            break
        result.append(s[left + 1:right])
        # Key: set up var at end of loop
        search = right  # or right+1
    return result

Suppose `while search < len(s):` Was Broken

What does test do: while search < len(s):
Turns out: nothing
Try changing it to while True:
Now the if/break logic takes care of exiting the loop always
Think about the s = 'xxxx' - no '[' at all, trace the code
Pattern:
Use while True: at the top
Use if/break logic inside the loop to detect "done" condition
When to use while-True form:
If the "done" checking requires a couple steps
e.g. calling s.find() and saving it in a var, then do the check
For simple cases: while test:
For complex cases, common pattern: while-True / if-break

Edge Cases - Two Skills

First think of a regular case
e.g. xx[abc]xx[42]xx
Also check "edge" cases
Empty string input: ''
Brackets with nothing between them: 'xx[]xx'
String with zero brackets: 'xxx'
Mentally trace through the code for those cases
Two skills - regular cases, but also checking off edge cases

search = right + 1 ?

(Maybe discuss this later)
Update search var at bottom of loop
What value works there?
The simplest is fine:
search = right
Slight optimization:
We *know* that index right is not a left bracket
So we could begin the search 1 char later
search = right + 1
Both of these are fine
Maybe KISS is better, so omit the + 1

Data and Parsing

Here's some fun looking data...

$GPGGA,005328.000,3726.1389,N,12210.2515,W,2,07,1.3,22.5,M,-25.7,M,2.0,0000*70
$GPGSA,M,3,09,23,07,16,30,03,27,,,,,,2.3,1.3,1.9*38
$GPRMC,005328.000,A,3726.1389,N,12210.2515,W,0.00,256.18,221217,,,D*78
$GPGGA,005329.000,3726.1389,N,12210.2515,W,2,07,1.3,22.5,M,-25.7,M,2.0,0000*71
$GPGSA,M,3,09,23,07,16,30,03,27,,,,,,2.3,1.3,1.9*38
$GPRMC,005329.000,A,3726.1389,N,12210.2515,W,0.00,256.18,221217,,,D*79
$GPGGA,005330.000,3726.1389,N,12210.2515,W,2,07,1.3,22.5,M,-25.7,M,3.0,0000*78
$GPGSA,M,3,09,23,07,16,30,03,27,,,,,,2.3,1.3,1.9*38
...

The above is what a GPS chip outputs
-buried deep in your phone, this is going on
A standard: NMEA
- NMEA_018 (wikipedia)
Things to notice: it's just text
-a series of text lines ending with \n
-each line is made of chars
Text is a super common exchange format between systems
"Parsing"
-Have "raw" text like this
-Find and pull out the data you want
Big Data
Analytics
The first step every time...
Parsing!
Only then can we do analytics
People want to work on the colors in their great graph
Look at that GPS data .. solve that first, then you can play with the colors

Advancing Through String With `var += 1`

Framing for today's example
Imagine a string on paper
Moving your index finger right looking for something
Python: have some var index into string
var += 1 .. like moving one to the right

at_words() Example Functions

Series of very real-world string algorithms

> parse examples

We've reached the critical mass of features
nested loops, break, list/append
Today's functions are not easy
Lab style - I'll demo some, you try coding as we go
Make drawing, think about code
Then there's other problems for more practice
Plus section

at_words(s)

Demonstrate several patterns on this one
Find all the @aaaa words in a string, return in a list
Where 'a' is any alphabetic char
'xx @abc @xyz xx' -> ['abc', 'xyz']
Not just the first word, all the words
Establishes code patterns we'll re-use
We'll work through this one carefully
Points about this code:
Keep "search" index - how far we are through s
Use while True + if/break to detect done
Use str.find() to locate each @
Use a nested while to skip over alpha chars to find end
Use < len(s) to protect use of s[xxx]
Remember to update "search" at loop bottom
Var names: search, at, end - try to keep things straight

at_words(s): For each '@' in s, parse out the "word" substring of 1 or more alphabetic chars which immediately follow the '@', so '@abc @ @xyz' returns ['abc', 'xyz'].

at_words() #1 Find End of Alpha Chars

AKA skip over the alpha chars
Consider the while loop to find the end of the alpha
The following code is very close
Deal with bug below

    end = at + 1
    while s[end].isalpha():
        end += 1

at_words() Bottom of loop

Use slice to pull out word
Then advance search for next iteration

    word = s[at + 1:end]
    result.append(word)
    search = end

at_words() Bug #1

Bug: run end off the end of s, testing non-existent s[end]
e.g. this happens if input is s = '@abc'
Think through how the loop works for that case
Solution:
Add guard < like this:
while end < len(s) and s[end].isalpha():
Q: How to test if index i is valid in s?
A: i < len(s)
Only check s[end] after checking that end is valid
Boolean Short Circuit
Evaluate expression left-right
As soon as boolean value determined, stops trying
A False in the midst of an and stops
So the < guards the s[end].isalpha()
Common guard pattern:
Check i < len(s) before trying s[i]

at_words() Case #2 - Zero Chars

What about 'xx@abc @ @xyz'
Consider slice of middle @ above
`s[at + 1:end]
Turns out to be s[8:8]
Which is the empty string ''
Add logic to screen out empty string:
if len(word) > 0: result.append(word)

at_words() Observations - Want vs. Don't-Want

Remember to update search= at end of loop! Infinite loop city!
Here skipping over what we want, looking for first don't-want
For later problems we'll see the opposite strategy:
Skipping over what we don't want, looking for the first want

at_words() Solution

def at_words(s):
    search = 0
    words = []
    while True:
        at = s.find('@', search)
        if at == -1:
            break
            
        # Pass over alpha chars to find end
        end = at + 1
        while end < len(s) and s[end].isalpha():
            end += 1
        
        word = s[at + 1:end]
        # Screen out len-0 word
        if len(word) > 0:
            words.append(word)
        
        # Set up next iteration
        search = end
    return words

exclaim_words()

Like at_words, but right-left
'x hey!@ho! -> ['hey!', 'ho!']

exclaim_words(s): For each '!' in s, parse out the "word" substring of one or more alphabetic chars which are immediately to the left of the '!'. Return a list of all such words including the '!', so 'x hey!@ho! returns ['hey!', 'ho!']. (Like at_words, but right-to-left)

parse_words()

Work this one as demo/exercise if we have time
The '@' to mark start is not realistic!
e.g. we should be able to parse words without @
'^^^abc xyz$' -> ['abc', 'xyz']
Sketch out what the loops do, search/begin/end vars
Have the outside while True: loop
Now - 2 forms of while/skip inside
Inner-loop-1 to find the first alphabetic char
Inner-loop-2 to find the end of the word
Sometimes skip over chars you want
Sometimes skip over chars you don't want
Lines which use s[xxx] have a xxx < len(s) guard

parse_words(s): Given a string s, parse out and return all "words", where a word is made of 1 or more adjacent alphabetic chars, so '^ abc xyz$' returns ['abc', 'xyz'].

parse_words() Solution

def parse_words(s):
    search = 0
    words = []
    while True:
        # Your code here
        pass
        # Find a first alpha char (note: not)
        begin = search
        while begin < len(s) and not s[begin].isalpha():
            begin += 1
        
        # No alphas found -> done
        if begin >= len(s):
            break
        # True here: s[begin] is first alpha
        
        # Move end past the group of alphas
        end = begin + 1
        while end < len(s) and s[end].isalpha():
            end += 1
        
        # Now we know where it is
        word = s[begin:end]
        words.append(word)
        search = end
        # or end + 1
    return words