Final Report
The Blackout History Project
http://blackout.gmu.edu/

Principal Investigator: James T. Sparrow
Center for History and New Media
George Mason University
(Blackout@gmu.edu)


[ Overview | Findings & Evaluation | Future Activities | Budget | Supporting Materials ]


[ NOTE: Because of delayed project deployment an important phase of recruitment is not yet concluded, and so this report lacks a comprehensive analysis of recruitment and interactivity. Since these are two main goals of the Blackout project, this report should be considered an interim draft of the final report to Sloan. The definitive version will be filed by March 15, 2000, when the results of on-line recruiting and interactivity have been fully analyzed.]

Overview

In the three years since its inception in January of 1997 the Blackout History Project has evolved from a highly ambitious methodological scheme for web-based research into a more modest but effective web-based environment for primary work in the recent history of science and technology. Due to the technical complexity of its original plan the project was significantly delayed and thus has not yet implemented the full range of research tools initially proposed. Those tools which have been developed are nonetheless quite valuable to this new approach to historical research, and have been designed so that they may in the future be scaled up to accommodate a more complex research project. The software tools at hand have already facilitated a type of historical work that would not otherwise have been possible, eliciting valuable preliminary historical materials from the two targeted subject populations (power engineers and New York "survivors" of the 1965 and 1977 power failures). While much remains to be done due to the initial delay, the essential machinery is in place and will only become more effective as time goes on.

The evolution of the project is reflected in its institutional history. Four major institutions have supported this project: the Alfred P. Sloan Foundation's Recent History of Science and Engineering on the Web grant program provided major funding throughout the entire period; Stanford University Libraries provided essential technological direction and support for all first wave projects, collectively known as Science and Technology in the Making (STIM), offering digital archive planning and Oracle database programming and serving from spring 1997 to late summer 1999; the Scholarly Technology Group at Brown University provided central web hosting for the Blackout site from the fall of 1997 until the fall of 1998; and since the fall of 1998 the Center for History and New Media (CHNM) at George Mason University has provided project support, database programming and serving, central web hosting and a valuable link from their widely-visited web portal. Rather than retrace this history, which has been documented in year-end reports for 1997 and 1998, I will simply observe that the effectiveness of the Blackout project has been greatly improved by its move to the Center for History and New Media in the fall of 1998. Of the five STIM projects mine was the most dependent on complex database-driven technology that required a long time to develop, and until the move to CHNM it was the least able to draw on essential local resources needed to implement the software and catalog development supported by the central STIM team at Stanford once it was developed. While this improvement benefited from timingócoming at the end of a long phase of start-up developmentóit also reflects CHNM's unparalleled environment, in which web-based historical research is fostered across a wide range of topical domains. If there is one lesson to take from this project, it is that concentrated technological development such as that provided by the STIM team at Stanford is extremely difficult to implement without a complementary support structure in local institutions where projects are housed. Web-based research, much like politics, is in some inescapable sense "local."

This observation applies as much to methodology as it does to institutional support. CHNM's numerous ties to historical and related organizations in the New York metropolitan area have provided a wide-ranging network within which to identify contacts for interviewing and recruitment to the web site. The strategic advantage of having access to such a network cannot be overstated, although it should be emphasized that previous work to build substantial content for the site was also important in keeping visitors there once they arrived (see the discussion of research outcomes below for this and other generalizations). The web as it is currently used is such a passive, information-gathering medium for most people that it requires supplemental social support to drive significant interactivity. This may be less true for engineers and IT workers whose primary social networks are not locally basedóthe Mousesite and PCR projects can speak more directly to that questionóbut for my main target population (New Yorkers and retired New York area electric utility employees), it was an essential reality.

If the rapid-cycle pace of recent scientific and technological development demands new historical habits, then those must be grounded in the social and organizational realities of the historical subjects in question. Contributing historical materials to a web site may be relatively easy, but it is not a priority for most people, even for those highly motivated people who exhibit a strong interest in their own past. Being able to draw on strong social networks among subject populations is an important precondition for engaging them in the process of making their own history, whether that happens on the web or elsewhere. The evidence from the Blackout project indicates that many people are indeed strongly motivated to participate in the making of their own history, but are unable to sustain more than modest interaction with a web site they find interesting because it falls outside the run of their daily lives. We have been able to elicit substantial interactivity from within established social networks and even, significantly, from individuals not tied to recruitment networks. But those interactions have not generally produced sustained second-order interactivity that would make the site "go of itself" and produce independent momentum. [This may change in the future, perhaps even by the time the definitive version of this report is filed.] Thus, it is vital that any future research along these lines must include a systematic evaluation of how subjects may be encouraged to connect their lives (jobs, relevant social experiences) to new technology-mediated habits of historical participation. The current crop of engineering and scientific societies supported by the second and third waves of Sloan funding might be in a position to study this question more closely, especially if they augment current efforts at community-building with targeted incentives for habit-building among individuals in especially promising groups of engineers and scientists.

The Blackout project has adjusted its design over the last 18 months to draw more effectively on the social networks and interests of study populations as they became evident. Building on the successful example of EVOnline, whose study population most resembled that of the Blackout project, we adapted an initial site structure that was heavily recursive and content-heavy into a modified structure which aims to build a critical mass of interactivity through a less dense web of content "highlights" tied more directly and insistently to appeals for visitor participation. We shifted away from in-depth interviewsóonce a modest number were available on the site to attract visitor interestótoward recruitment that prodded potential respondents to complete the on-line survey immediately. We combined print and web-based advertising with word-of-mouth and "guerilla marketing" in numerous on-line forums relevant to study populations. The response to this strategy, described in the remainder of this report, will not be completely clear until recruitment is complete and I have had sufficient time to analyze the results (hence the interim status of this report). While there are some unquestionable benefits to the sequence in which the project has evolvedóe.g., many visitors came to the site or returned to it to view the rich collection of archival contentóit is already clear that the more recursive strategy adopted early on was not only costly in terms of time and money, but was also more likely to be effective after a critical mass of interactivity had been established. We will be able to make use of the object-oriented recursive system developed in the first phase, and it may turn out to be the most valuable by-product of the entire research effort, but the timing of its development was problematic for spurring interactivity.

The remainder of this report describes and evaluates the outcomes of the new research strategy employed in the last 18 months of the Blackout project. Because the project has been delayed and is not finished with recruitment, these sections are provisional and will be updated in a later, definitive Final Report. For the same reason the section on future activities has great bearing on any attempt to measure the "outcomes" of this study, as many planned activities will not take place until after the term of the grant is up (i.e. after 12/31/99).

Findings and Evaluation

The main outcome of this project is the Blackout History web site and its supporting databases. I designed and implemented much of the main web site myself, including digitization of documents. These efforts would have been largely in vain without the support of STIM and CHNM, who provided invaluable guidance and assistance in two critical redesignsóthe first with STIM team members Jim Coleman, David Soergel and Sigrid Mueller in July 1998, and the second with CHNM members John Cheng and TuVinh Vuong in July 1999. STIM also carried out a mass conversion of my pre-scanned documents into Adobe Acrobat files, while CHNM student assistants Troy LaChance and John Summers conducted, transcribed and digitized interviews. Databases were developed according to my rough specifications and were considerably refined in a WebObjects-Oracle system by STIM Team members Coleman and Soergel, and in a Tango-Filemaker system by CHNM affiliates Elena Razlogova and Jessica Finnefrock.

A number of historical materials and software tools make up the Blackout site:


The debates section of the site was intended to provide a stage on which informed commentators and experts could advance our knowledge of the blackouts. As of this writing, only one out of the five scholars who committed to writing brief essays for this section has actually done so. Richard Hirsh, historian of technology at Virginia Tech and expert on the electric utility industry in this period, has completed a very useful brief analysis of the blackouts' significance to the industry. I am still moderately hopeful about the others, although their schedules leave little time for pro bono scholarly work. In the meantime I will modify the debates section so that Hirsh's essay can appear in a meaningful context; perhaps as a highlight, rather than as a solitary contribution to a non-existent "debate."

Despite the relatively modest number of contributions, these materials were accessed by a large number of visitors in the past year or so. More than 17,500 visits (not just hits) have been logged on the site during the last 13.5 months; more than 4,800 of those occurred within the last 60 days. This traffic was not generated by a few sites (e.g. robots, STIM project participants, etc.): it arrived from more than 8,500 hosts, the largest of which (a robot) accounted for less than 1.5% of all hits; the largest number of visits from a single host was 251. Visitors reached an average of 2.7 pages per visit, lingering an average of 9 minutes and 54 seconds before leaving. (See CHNM server report for Blackout project, 2/7/2000.)

An evaluation of the historical value of the site, especially as it relates to on-line recruitment and interactivity, is difficult to provide at the writing of this interim draft. We are still in the midst of an important phase of recruitment derived from on-line advertising, so a thorough assessment of recruitment and interactivity will have to wait until mid-March 2000. However, three tentative observations may be made.

A first lesson has already been mentioned but deserves further elaboration: strategies to promote online recruitment and interactivity must be based in social networks, although other strategies may amplify the impact of the resulting traffic once it arrives on the site. Respondents with some personal connection to the study or its participants (e.g. relatives of contacts, Con Ed employees) were highly motivated to interact with the site and complete an on-line survey or comment. Respondents who received a personal phone call or an email forwarded from a friend were also motivated, though less than in the first case. Finally, people responding to an advertisement or a link produced by a search engine were driven to the site by their curiosity and often indicated so in email or brief comments, but rarely went beyond such expressions of curiosity. There were some important exceptionsóthe unsolicited text of Bruce Wollenberg's power engineering listserv thread is the best exampleóbut these were almost by definition arbitrary, based on a serendipitous intersection of personal motivation and web browsing. "Guerilla marketing" seems to offer an acceptable middle way between intensive social networking and more automated approaches. Placing carefully targeted messages on forums in New York area websites (including unlikely candidates such as fan sites devoted to the Mets and the Yankees) seemed to produce interactivity on the site, as did registering the site in every link exchange and newsgroup conceivable, but the yield of such activity will not be clear until a final analysis of interactivity is possible.

As an general rule, contributions rose when we did the electronic equivalent of "pressing the flesh." But once visitors made it to the site other strategies were necessary to keep them there. Stocking the archive with a relatively deep set of primary documents seemed to hold great interest for the most common sort of traffic. According to the most recent server report, the 1977 archive home page was the third most frequently requested page in the site: although it required a relatively large number of clicks to be reached (5.6 on average, reflecting sustained depth of interest) this single page managed to hold visitors for a relatively long period of time (1 minute and 44 seconds on average, longer than more than 40% of all visits within the entire site). Providing snippets of content in a highlights section was also successful. The 1977 highlight section was the most frequently requested page on the site, getting 35 hits each day, encouraging viewers to linger there for relatively long periods of time (1:35), and attracting attention almost immediately upon a visitor's arrival at the site (within 2.3 clicks). So, content does matter for site interactivity. The question remains: how much and which content?

A second lesson reflects an old design principle: the function and the form of a web-based research site must be mutually reinforcing. Our on-line survey received a total of 226 hits which occurred after an average of 7.3 tedious clicksóseen in this light, a yield of 44 completed surveys is remarkably high! A larger number of visitors (just under 800 hits) looked at existing surveys without going on to the survey form, but even if all of these were potential respondents the response rate would still be impressive. What is most puzzling is how a site with over 17,500 visits generating more than 47,000 hits could attract fewer than a thousand hits to the on-line survey when it was linked directly from the home page, which generated the highest proportion of recent hits (8.14%) for a single page within the site. Clearly the structure of the site did not force visitors to confront the survey in a sufficiently aggressive manner. Perhaps a more effective approach might have mimicked the EVOnline's splash page, which channels relevant visitors directly into the on-line survey before showing any other part of the site. The even lower number of visitors who submitted a comment or participated in an on-line forum may have resulted from a similar structural problem. Perhaps the lightbulb did not cry out to be clicked as I had intended, or the discussion forums were too rigidly defined and tucked away in slow-loading sections to draw large amounts of interactivity. While the jury is still out on this score, it does seem that interactive features must be clearly, perhaps blatantly foregrounded for the average visitorónothing is "too obvious." The number of interactive features may also be confusing to the average visitor.

A third and final lesson has to do with timing, which is everything. The NT server on which part of the Blackout site ran at the beginning of this 18 month period was knocked out of commission by a power surge during a thunder storm in the early fall of 1999, thus illustrating all too effectively the ripple effects of technological failure in interdependent systems. This server was about to run the Web Objects code transplanted from the STIM site. When it was lost, the transfer of technology was halted (only recently has the project been able to make preparations for the transfer to resume). That pushed back other objectives, such as implementing a second, more tightly programmed on-line survey of power engineers. While the Blackout site was up and running on another server within a relatively short time, thanks to prudent backup procedures, an essential part of its technological growth was disrupted and has yet to be put into place. I have responded by making discrete tasks and goals, such as the second on-line survey, more loosely coupled. This has probably facilitated direct activity on the site, which is needed most right now, but it has also delayed further the implementation of tools which had already absorbed a large amount of project time. Moreover, if I had pursued a simpler, more loosely coupled web design and recruitment strategy earlier on in the project I could have taken greater advantage of the free publicity and media attention generated by Y2K fears and the anniversary of the 1977 blackout.

Future Activities

The Blackout History Project web site will continue into the indefinite future, residing on a server at CHNM until interactivity and historical value are exhausted or the project is terminated for extrinsic reasons. In addition to the efforts described above to stimulate recruitment and interactivity, most of which I will continue to pursue throughout the spring of 2000, some longer term goals will soon be within reach. I expect to implement the second on-line survey, of power engineers, by the time the definitive version of this report is complete in mid-March. Within a month of its launch an announcement should reach the tens of thousands of potential respondents who read the IEEE Power Engineering Review. During the spring I will also conduct a limited number of interviews with power engineers and contacts in relevant organizations (e.g. the New York Power Pool). Finally, I will press the remaining four contributors who have committed to writing a brief essay for the site in hopes that a genuine, if modest, scholarly forum can be created. By the end of the year I expect to prepare two papers on the results of the Blackout study (described above). I would also like to use the Web Objects metadata system to create a test model of recursive scholarly interaction with a major collection of primary sources (perhaps a dataset), but that goal is external to the Blackout project and probably will not be feasible outside of a separate grant devoted to that objective.

Budget

The Office of Sponsored Programs at GMU will provide a final accounting budget for the funds which were spent directly on the Blackout project.

Overall the Blackout budget was reasonably well-matched to the development needs of the project. It seems fair to say that the full budget was required to get the project to where it stands today: that is, prepared with the necessary tools, electronic materials and expertise to conduct the significant portion of the study that remains, and sufficiently experienced to continue recruitment and interactivity independently of the grant. But as noted above, this could not have happened without considerable institutional support, nor can it continue without the ongoing support of CHNM. In retrospect much of what has been accomplished could have been done on a smaller budget. Still, some of the most exciting developments (e.g. the metadata system), though yet to be fully implemented, required costly programming and development time. We can pursue the refinement and further development of these tools with greater foresight and informed caution, but the initial process of developing them seems to have been unavoidably messy and expensive -as befits a genuine experiment in research methods.

Supporting Materials