A Study of State-Funded Community Networks in Indiana:

Final Report

Methodology

The study used three methods to gather data :

Content analysis

Content analysis was used because it is an effective method for discerning patterns and themes in textual and graphic data. There were two objectives for this phase of the project, the evaluation of each site against a standard and the evaluation of the structure, depth, quality, design (text, graphics, CGI/Javascript/Java) and utility of each site's existing content. To develop the coding scheme, an AI document outlining the required features of CN web sites was used as a starting point. Next, three randomly selected web sites were examined and exhaustively described in terms of their content and structures. This produced a list of features and structural elements used as the basis for the first version of the coding scheme.

A trained coder then began to examine systematically each CN's web site, modifying the coding scheme as needed. By the third iteration, the scheme had stabilized into the final version (see Appendix A: Web Site Coding Scheme). The coding scheme was checked for reliability by one of the researchers, who recoded three web sites and compared the results to the work of the coder; the result of this exercise was intercoder reliability of 94%.

To get a sense of the structure of these CNs, site maps were drawn for each network.

Surveys

Based on the research questions and a set of interviews with a sample of CN Board members, a survey instrument was developed. The instrument went through several iterations during a pretesting phase, where it was examined by two CN Board members and two researchers not connected with this project. Based on their observations and criticisms, the instrument was modified into the final version (see Appendix B: Interview and Survey Instrument). The survey was administered to every CN board and technical staff member at all CNs that were not visited by the researchers. Its purpose was to gather data about the demographics of CNs, including the types of hardware and software used to develop content and maintain the network. It was also be used to develop an inventory of the resources and services made available through the networks and to collect management information, including statements of local goals, planning, and methods of evaluation. Finally, the surveys were used to collect information about the successes, failures and challenges faced by these networks.

During the first week of September, 1997, 194 surveys were mailed to CN Board members and technical staff. By the end of November, 1997, 43 surveys had been returned, of which 38 contained usable data, for a response rate of about 22%.

Case studies

In order to gather more detailed information about CNs, seven sites were selected for case study analyses and visited by the researchers. These case study sites reflected geographic distribution (they were located across the state), and population size (large and small) and type (rural and urban). During these site visits, more extensive data were collected through interviews, observation, and document collection (budget documents, operations and training manuals, policy and mission statements). Questions were asked about the range of information resources and services which were being offered to the public, how they were being used and how they could be improved. Also, data were gathered about the management and daily operation of these networks.

In addition, the researchers set up an unmoderated CN listserv in August, 1997, with membership restricted to CN board members and technical staff to provide a "private forum" for them to discuss issues raised during the research. The hope was that this communication channel would be used by the CN members and would provide additional longitudinal data; unfortunately, after a flurry of introductions, list traffic dropped off and eventually ceased.

Utility of this methodology:

One goal of this research was to develop a portable methodology for the evaluation of CNs that can be used in other settings and the data collection strategies described below can serve as a basis for such a methodology. The combination of data collection techniques proved to be very useful in gathering a wide range of information about CNs. Content analysis is a powerful and non-intrusive technique for discerning patterns in a sample of data instances, particularly when they are textual in nature. The coding scheme allowed both the description of the "typical" CN web site and a means by which the individual web sites could be compared and contrasted. This scheme can easily be used as a template for analyses of other CN sites, although some modifications are to be expected to adapt the instrument to local conditions.

Surveys and interviews are useful techniques for gathering data from a large number and wide range of people. The interview schedule was an effective tool for opening and carrying out in depth conversations with CN Board members; the questions were on target and the language was easily understood. The survey instrument developed for this study generated rich data from those who took the time to fill it out. However, there was a low response rate. This may indicate that the instrument requires further modification; it may also indicate that the researchers did not handle the promotion and distribution of the instrument in as effective a way as could have been done.

The site visits were very useful because they allowed the researchers to observe the routine workings of CN Boards; most of the visits coincided with Board meetings. Conversations with Board members before and after these meetings proved to be rich sources of data, allowing the researchers to follow up on matters raised in the meetings and probe further about issues covered in the interviews.