- Final Report
- The Blackout History Project
[ Overview | Findings & Evaluation | Future Activities | Budget | Supporting Materials ]
[ NOTE: Because of delayed project deployment an important phase of recruitment is not yet concluded, and so this report lacks a comprehensive analysis of recruitment and interactivity. Since these are two main goals of the Blackout project, this report should be considered an interim draft of the final report to Sloan. The definitive version will be filed by March 15, 2000, when the results of on-line recruiting and interactivity have been fully analyzed.]
In the three years since its inception in January of 1997 the Blackout History Project has evolved from a highly ambitious methodological scheme for web-based research into a more modest but effective web-based environment for primary work in the recent history of science and technology. Due to the technical complexity of its original plan the project was significantly delayed and thus has not yet implemented the full range of research tools initially proposed. Those tools which have been developed are nonetheless quite valuable to this new approach to historical research, and have been designed so that they may in the future be scaled up to accommodate a more complex research project. The software tools at hand have already facilitated a type of historical work that would not otherwise have been possible, eliciting valuable preliminary historical materials from the two targeted subject populations (power engineers and New York "survivors" of the 1965 and 1977 power failures). While much remains to be done due to the initial delay, the essential machinery is in place and will only become more effective as time goes on.
The evolution of the project is reflected in its institutional history. Four major institutions have supported this project: the Alfred P. Sloan Foundation's Recent History of Science and Engineering on the Web grant program provided major funding throughout the entire period; Stanford University Libraries provided essential technological direction and support for all first wave projects, collectively known as Science and Technology in the Making (STIM), offering digital archive planning and Oracle database programming and serving from spring 1997 to late summer 1999; the Scholarly Technology Group at Brown University provided central web hosting for the Blackout site from the fall of 1997 until the fall of 1998; and since the fall of 1998 the Center for History and New Media (CHNM) at George Mason University has provided project support, database programming and serving, central web hosting and a valuable link from their widely-visited web portal. Rather than retrace this history, which has been documented in year-end reports for 1997 and 1998, I will simply observe that the effectiveness of the Blackout project has been greatly improved by its move to the Center for History and New Media in the fall of 1998. Of the five STIM projects mine was the most dependent on complex database-driven technology that required a long time to develop, and until the move to CHNM it was the least able to draw on essential local resources needed to implement the software and catalog development supported by the central STIM team at Stanford once it was developed. While this improvement benefited from timingócoming at the end of a long phase of start-up developmentóit also reflects CHNM's unparalleled environment, in which web-based historical research is fostered across a wide range of topical domains. If there is one lesson to take from this project, it is that concentrated technological development such as that provided by the STIM team at Stanford is extremely difficult to implement without a complementary support structure in local institutions where projects are housed. Web-based research, much like politics, is in some inescapable sense "local."
This observation applies as much to methodology as it does to institutional support. CHNM's numerous ties to historical and related organizations in the New York metropolitan area have provided a wide-ranging network within which to identify contacts for interviewing and recruitment to the web site. The strategic advantage of having access to such a network cannot be overstated, although it should be emphasized that previous work to build substantial content for the site was also important in keeping visitors there once they arrived (see the discussion of research outcomes below for this and other generalizations). The web as it is currently used is such a passive, information-gathering medium for most people that it requires supplemental social support to drive significant interactivity. This may be less true for engineers and IT workers whose primary social networks are not locally basedóthe Mousesite and PCR projects can speak more directly to that questionóbut for my main target population (New Yorkers and retired New York area electric utility employees), it was an essential reality.
If the rapid-cycle pace of recent scientific and technological development demands new historical habits, then those must be grounded in the social and organizational realities of the historical subjects in question. Contributing historical materials to a web site may be relatively easy, but it is not a priority for most people, even for those highly motivated people who exhibit a strong interest in their own past. Being able to draw on strong social networks among subject populations is an important precondition for engaging them in the process of making their own history, whether that happens on the web or elsewhere. The evidence from the Blackout project indicates that many people are indeed strongly motivated to participate in the making of their own history, but are unable to sustain more than modest interaction with a web site they find interesting because it falls outside the run of their daily lives. We have been able to elicit substantial interactivity from within established social networks and even, significantly, from individuals not tied to recruitment networks. But those interactions have not generally produced sustained second-order interactivity that would make the site "go of itself" and produce independent momentum. [This may change in the future, perhaps even by the time the definitive version of this report is filed.] Thus, it is vital that any future research along these lines must include a systematic evaluation of how subjects may be encouraged to connect their lives (jobs, relevant social experiences) to new technology-mediated habits of historical participation. The current crop of engineering and scientific societies supported by the second and third waves of Sloan funding might be in a position to study this question more closely, especially if they augment current efforts at community-building with targeted incentives for habit-building among individuals in especially promising groups of engineers and scientists.
The Blackout project has adjusted its design over the last 18 months to draw more effectively on the social networks and interests of study populations as they became evident. Building on the successful example of EVOnline, whose study population most resembled that of the Blackout project, we adapted an initial site structure that was heavily recursive and content-heavy into a modified structure which aims to build a critical mass of interactivity through a less dense web of content "highlights" tied more directly and insistently to appeals for visitor participation. We shifted away from in-depth interviewsóonce a modest number were available on the site to attract visitor interestótoward recruitment that prodded potential respondents to complete the on-line survey immediately. We combined print and web-based advertising with word-of-mouth and "guerilla marketing" in numerous on-line forums relevant to study populations. The response to this strategy, described in the remainder of this report, will not be completely clear until recruitment is complete and I have had sufficient time to analyze the results (hence the interim status of this report). While there are some unquestionable benefits to the sequence in which the project has evolvedóe.g., many visitors came to the site or returned to it to view the rich collection of archival contentóit is already clear that the more recursive strategy adopted early on was not only costly in terms of time and money, but was also more likely to be effective after a critical mass of interactivity had been established. We will be able to make use of the object-oriented recursive system developed in the first phase, and it may turn out to be the most valuable by-product of the entire research effort, but the timing of its development was problematic for spurring interactivity.
The remainder of this report describes and evaluates the outcomes of the new research strategy employed in the last 18 months of the Blackout project. Because the project has been delayed and is not finished with recruitment, these sections are provisional and will be updated in a later, definitive Final Report. For the same reason the section on future activities has great bearing on any attempt to measure the "outcomes" of this study, as many planned activities will not take place until after the term of the grant is up (i.e. after 12/31/99).
Findings and Evaluation
The main outcome of this project is the Blackout History web site and its supporting databases. I designed and implemented much of the main web site myself, including digitization of documents. These efforts would have been largely in vain without the support of STIM and CHNM, who provided invaluable guidance and assistance in two critical redesignsóthe first with STIM team members Jim Coleman, David Soergel and Sigrid Mueller in July 1998, and the second with CHNM members John Cheng and TuVinh Vuong in July 1999. STIM also carried out a mass conversion of my pre-scanned documents into Adobe Acrobat files, while CHNM student assistants Troy LaChance and John Summers conducted, transcribed and digitized interviews. Databases were developed according to my rough specifications and were considerably refined in a WebObjects-Oracle system by STIM Team members Coleman and Soergel, and in a Tango-Filemaker system by CHNM affiliates Elena Razlogova and Jessica Finnefrock.
A number of historical materials and software tools make up the Blackout site:
- Survey: As of this writing 44 respondents have completed the on-line "blackout experience" survey, an open-ended questionnaire which parallels interviews questions about respondents' perceptions of the 1965 and 1977 blackouts. [If current response rates remain constant over the next few months, it is foreseeable that we may have more than 50 responses by the end of this month and more than 100 by the summer. This will be clearer by the time the definitive Final Report is submitted in mid-March.] Most the responses have some historical value when viewed from a social or cultural angle. If recruitment continues to go well a subset of "blackout experience" respondents will be targeted to complete a more precise on-line instrument that will compare blackout survivors' expectations against those of the broader population of electricity consumers.
A second survey, of electric utility industry employees, will go on-line in March in tandem with an announcement in the IEEE Power Review.
- Comments: Some site visitors (23 total) have used the "lightbulb" feature of the site to comment on a specific web page or file. These vary greatly in historical value, and are less numerous than I had anticipated. Some anecdotal contact with respondents suggests that the periodically slow performance of the system on the STIM machine in California (due to net traffic) may have been a factor. [This performance issue should be eliminated when the Web Objects system is ported to an NT server at GMU; to be addressed in the definitive Final Report.] Clearly this feature, available on every page of the site, was underutilized.
- Discussion Forum: A handful of visitors (9) found their way into two of the three discussion forums: 6 discussed their experiences of the 1965 or 1977 blackout, and 3 discussed the question of whether the blackouts were "acts of God." As with the comments, these also vary in historical value, although some thoughtful responses stand out. [This part of the forum may suffer from the same performance issue that may have affected the comments.]
- Interviews: To stimulate the interest and spark the memories of potential respondents we conducted 13 interviews and 1 focus group. These were transcribed and in some cases digitized, although two interviews and the focus group had to be withheld from the site due to respondents' fears of publicity (despite their anonymity). The time and cost involved in interviewing diminished the cost-effectiveness of this activity beyond the initial cohort. Providing interview sound files by Real streaming media proved to be cost-competitive with transcription, at least for a modest number of files, but had limitations (sound quality, audience interest) which made digitization a lower priority than other activities still being implemented.
- Contributed materials: Over the course of the project individuals have contacted me via the Blackout site and donated materials pertinent to the blackouts. Bruce Wollenberg, professor of power engineering at the University of Minnesota, forwarded the entire text of a power engineering listserv discussion on the blackouts of 1965 that he moderated in 1995-6. Robert Thomas, professor of electrical engineering at Cornell, sent an electronic case study (written in Hypercard) of the 1965 blackout via email attachment. A lawyer sent by email attachment the text of the 1984 New York State Court of Appeals decision concerning the issue of Con Edison's "gross negligence" in the 1977 blackout; while this could have been obtained by Lexis / Nexus, the legal perspective provided by the lawyer was valuable. Some survey respondents sent me materials that captured their experiences. One recent survey respondent has promised to send me a tape recording of a news broadcast from the 1977 blackout. Another respondent sent a tape recording of the song she was playing at a performance on Broadway when the lights went out in 1977. A third respondent sent a photo of the t-shirt he purchased after the 1977 blackout (stating "Thanks CON ED / Black-out No. 2 / New York City / July 13th1977 / 9:34 P.M."). A fourth respondent emailed the lyrics of an enigmatic song that I believe to be part of the folklore of the 1977 power failure.
- Archive: Because the blackouts have received little scholarly treatment or public attention since their occurrence, I digitized 27 major primary documents that are directly relevant to the 1965 or 1977 power failure. Although the archive was not the primary goal of the project, it has proven of great interest to visitors and is often mentioned when visitors send email about why they found the site valuable. One prominent Y2K site abstracted the 1977 portion of the archive because of its shock value, and that link drove a large proportion of all visitor traffic onto the site (ca. 5-8% of recent hits, according to the most recent server report done on 2/7/2000).
- Metadata system: The comments and discussion forums described above are captured as "web objects," digital items stored in a relational Oracle database and connected to each other by metadata. (The survey was originally based in this system, but was shifted over to a simpler system based in Filemaker for short-term practical considerations. It will eventually move back into the Web Object system.) This extraordinarily powerful system took a long time develop, and will not be fully implemented until it is moved to a GMU server. In its current incarnation the metadata system allows new comments and discussion contributions to automatically "inherit" the metadata properties (classifications) of the items to which they are responses. When a visitor searches for the item, it is easy to find others with the same classification, including the item to which it refers. Thus, a comment on a specific document (say, the 1965 Life Magazine coverage) would record both a link to that document and a link to all other items sharing the same metadata classification as the magazine article. An abridged view of this system can be glimpsed by visiting the search engine in the help section of the site. The metadata system could be applied much more intensively than currently is the case, but the level of interactivity on the Blackout site does not yet warrant the effort. Sloan would be wise to consider sharing the metadata system code with other projects, especially those which anticipate large amounts of highly recursive interactivity. It is probably not worth investing time or money to adapt it to smaller-scale projects in which interactions are relatively "shallow" or decontextualized.
- Scholarly work: (See supporting materials for conference presentations.) Two papers will be prepared during 2000: one reporting on the Blackout project as an experiment in historical research, and another on the blackouts as historical events whose social, cultural and technological meanings reveal a shift in popular acceptance of technocracy during the 1960s and 1970s. Two other papers have been partially developed but their completion will depend on the project's success in coming months: the first concerns stories and urban legends of the blackouts as collective biography, and the second examines the role of failure in determining the path dependence of mature, large-scale technological systems such as the Northeast power grid.
The debates section of the site was intended to provide a stage on which informed commentators and experts could advance our knowledge of the blackouts. As of this writing, only one out of the five scholars who committed to writing brief essays for this section has actually done so. Richard Hirsh, historian of technology at Virginia Tech and expert on the electric utility industry in this period, has completed a very useful brief analysis of the blackouts' significance to the industry. I am still moderately hopeful about the others, although their schedules leave little time for pro bono scholarly work. In the meantime I will modify the debates section so that Hirsh's essay can appear in a meaningful context; perhaps as a highlight, rather than as a solitary contribution to a non-existent "debate."
Despite the relatively modest number of contributions, these materials were accessed by a large number of visitors in the past year or so. More than 17,500 visits (not just hits) have been logged on the site during the last 13.5 months; more than 4,800 of those occurred within the last 60 days. This traffic was not generated by a few sites (e.g. robots, STIM project participants, etc.): it arrived from more than 8,500 hosts, the largest of which (a robot) accounted for less than 1.5% of all hits; the largest number of visits from a single host was 251. Visitors reached an average of 2.7 pages per visit, lingering an average of 9 minutes and 54 seconds before leaving. (See CHNM server report for Blackout project, 2/7/2000.)
An evaluation of the historical value of the site, especially as it relates to on-line recruitment and interactivity, is difficult to provide at the writing of this interim draft. We are still in the midst of an important phase of recruitment derived from on-line advertising, so a thorough assessment of recruitment and interactivity will have to wait until mid-March 2000. However, three tentative observations may be made.
A first lesson has already been mentioned but deserves further elaboration: strategies to promote online recruitment and interactivity must be based in social networks, although other strategies may amplify the impact of the resulting traffic once it arrives on the site. Respondents with some personal connection to the study or its participants (e.g. relatives of contacts, Con Ed employees) were highly motivated to interact with the site and complete an on-line survey or comment. Respondents who received a personal phone call or an email forwarded from a friend were also motivated, though less than in the first case. Finally, people responding to an advertisement or a link produced by a search engine were driven to the site by their curiosity and often indicated so in email or brief comments, but rarely went beyond such expressions of curiosity. There were some important exceptionsóthe unsolicited text of Bruce Wollenberg's power engineering listserv thread is the best exampleóbut these were almost by definition arbitrary, based on a serendipitous intersection of personal motivation and web browsing. "Guerilla marketing" seems to offer an acceptable middle way between intensive social networking and more automated approaches. Placing carefully targeted messages on forums in New York area websites (including unlikely candidates such as fan sites devoted to the Mets and the Yankees) seemed to produce interactivity on the site, as did registering the site in every link exchange and newsgroup conceivable, but the yield of such activity will not be clear until a final analysis of interactivity is possible.
As an general rule, contributions rose when we did the electronic equivalent of "pressing the flesh." But once visitors made it to the site other strategies were necessary to keep them there. Stocking the archive with a relatively deep set of primary documents seemed to hold great interest for the most common sort of traffic. According to the most recent server report, the 1977 archive home page was the third most frequently requested page in the site: although it required a relatively large number of clicks to be reached (5.6 on average, reflecting sustained depth of interest) this single page managed to hold visitors for a relatively long period of time (1 minute and 44 seconds on average, longer than more than 40% of all visits within the entire site). Providing snippets of content in a highlights section was also successful. The 1977 highlight section was the most frequently requested page on the site, getting 35 hits each day, encouraging viewers to linger there for relatively long periods of time (1:35), and attracting attention almost immediately upon a visitor's arrival at the site (within 2.3 clicks). So, content does matter for site interactivity. The question remains: how much and which content?
A second lesson reflects an old design principle: the function and the form of a web-based research site must be mutually reinforcing. Our on-line survey received a total of 226 hits which occurred after an average of 7.3 tedious clicksóseen in this light, a yield of 44 completed surveys is remarkably high! A larger number of visitors (just under 800 hits) looked at existing surveys without going on to the survey form, but even if all of these were potential respondents the response rate would still be impressive. What is most puzzling is how a site with over 17,500 visits generating more than 47,000 hits could attract fewer than a thousand hits to the on-line survey when it was linked directly from the home page, which generated the highest proportion of recent hits (8.14%) for a single page within the site. Clearly the structure of the site did not force visitors to confront the survey in a sufficiently aggressive manner. Perhaps a more effective approach might have mimicked the EVOnline's splash page, which channels relevant visitors directly into the on-line survey before showing any other part of the site. The even lower number of visitors who submitted a comment or participated in an on-line forum may have resulted from a similar structural problem. Perhaps the lightbulb did not cry out to be clicked as I had intended, or the discussion forums were too rigidly defined and tucked away in slow-loading sections to draw large amounts of interactivity. While the jury is still out on this score, it does seem that interactive features must be clearly, perhaps blatantly foregrounded for the average visitorónothing is "too obvious." The number of interactive features may also be confusing to the average visitor.
A third and final lesson has to do with timing, which is everything. The NT server on which part of the Blackout site ran at the beginning of this 18 month period was knocked out of commission by a power surge during a thunder storm in the early fall of 1999, thus illustrating all too effectively the ripple effects of technological failure in interdependent systems. This server was about to run the Web Objects code transplanted from the STIM site. When it was lost, the transfer of technology was halted (only recently has the project been able to make preparations for the transfer to resume). That pushed back other objectives, such as implementing a second, more tightly programmed on-line survey of power engineers. While the Blackout site was up and running on another server within a relatively short time, thanks to prudent backup procedures, an essential part of its technological growth was disrupted and has yet to be put into place. I have responded by making discrete tasks and goals, such as the second on-line survey, more loosely coupled. This has probably facilitated direct activity on the site, which is needed most right now, but it has also delayed further the implementation of tools which had already absorbed a large amount of project time. Moreover, if I had pursued a simpler, more loosely coupled web design and recruitment strategy earlier on in the project I could have taken greater advantage of the free publicity and media attention generated by Y2K fears and the anniversary of the 1977 blackout.
The Blackout History Project web site will continue into the indefinite future, residing on a server at CHNM until interactivity and historical value are exhausted or the project is terminated for extrinsic reasons. In addition to the efforts described above to stimulate recruitment and interactivity, most of which I will continue to pursue throughout the spring of 2000, some longer term goals will soon be within reach. I expect to implement the second on-line survey, of power engineers, by the time the definitive version of this report is complete in mid-March. Within a month of its launch an announcement should reach the tens of thousands of potential respondents who read the IEEE Power Engineering Review. During the spring I will also conduct a limited number of interviews with power engineers and contacts in relevant organizations (e.g. the New York Power Pool). Finally, I will press the remaining four contributors who have committed to writing a brief essay for the site in hopes that a genuine, if modest, scholarly forum can be created. By the end of the year I expect to prepare two papers on the results of the Blackout study (described above). I would also like to use the Web Objects metadata system to create a test model of recursive scholarly interaction with a major collection of primary sources (perhaps a dataset), but that goal is external to the Blackout project and probably will not be feasible outside of a separate grant devoted to that objective.
The Office of Sponsored Programs at GMU will provide a final accounting budget for the funds which were spent directly on the Blackout project.
Overall the Blackout budget was reasonably well-matched to the development needs of the project. It seems fair to say that the full budget was required to get the project to where it stands today: that is, prepared with the necessary tools, electronic materials and expertise to conduct the significant portion of the study that remains, and sufficiently experienced to continue recruitment and interactivity independently of the grant. But as noted above, this could not have happened without considerable institutional support, nor can it continue without the ongoing support of CHNM. In retrospect much of what has been accomplished could have been done on a smaller budget. Still, some of the most exciting developments (e.g. the metadata system), though yet to be fully implemented, required costly programming and development time. We can pursue the refinement and further development of these tools with greater foresight and informed caution, but the initial process of developing them seems to have been unavoidably messy and expensive -as befits a genuine experiment in research methods.
- Baltimore SHOT (October 1998) electronic poster presentation (with David Kirsch), http://sloan.stanford.edu/EVSlides/SHOT_98_Final/ and original proposal, http://sloan.stanford.edu/SHOTTalk1.htm
- GMU workshop (April 1999), http://chnm.gmu.edu/sloan/ and
Presentation "slides," http://chnm.gmu.edu/sloan/talk.html
- Stanford Conference (August 1999) web-based presentation with links to "slides," http://chnm.gmu.edu/blackout/sloan/stanford_talk.html