Investigating the Design and Evaluation of Research Web Sites

Table of Contents

Since its inception at CERN, the World Wide Web has been intended as a means for real-time collaboration. The desire to collaborate with other around the world formed the backbone of this project, and so the primary level of the WWW has been to share information with others. However, the methods in sharing information have never been standardized, and most likely never will be. It is only with a mix between research and trial-by-error that one can provide an effective web site that can functionally share its information with people all over the world.

It was natural for a group that concentrated on collaboration to use a service that was also based on sharing information. The Collaborative Software Development Laboratory (CSDL) initially published their page during the first generation of web site design. In the early days of the World Wide Web, most information was simply listed in one large page. These sites did not concentrate too much on the structure of the site, or the design of the pages contained within, as the major point of the WWW was to share one's information. The CSDL page had a simple listing of research projects accomplished by the group, as well as reference points to the many publications written by the group. Due to the lower modem speeds and bandwidth issues, a minimum of pictures were used, including only the organization's logo and a group photo.

Unfortunately, as web site design evolved over the years, the CSDL web site did not change along with it. Its layout, which was effective in the early-nineties, was greatly outdated, and the overall feel of the site was monotone and uninteresting. The content of the site was also outdated, as many new projects were not included in the site, and there was no mention of the group losing and adding members during the years. As the web page began to take a new role over the years--that of exhibiting an organization's identity, it was apparent that a change had to be made to the site. The neglect of the web site now had the ability to cause a negative impression of CSDL to be formed by visitors, and this needed to be resolved.

The Aziza design group (formally 691 Web Development Team) was commissioned by CSDL to implement a new web site. The group was assigned not only to update the entire site, but also to research and investigate the process and life cycle of World Wide Web site development. This examination into the various methods involved would then become one of CSDL's own research projects.

The updating of the CSDL web site would be used as the basis of this research document. Issues such as the balance between providing information and providing an image of the group would be discussed and examined, and ways to share research information over the World Wide Web would be studied. To back the data researched, evaluations by the various users of the site would be conducted and examined. This document records our web site design processes, what insights we had about those processes, our findings, and finally, our conclusions.

The Collaborative Software Development Laboratory is a highly prolific organization. In the years that it has existed, members have literally turned out thousands of pages of HTML and documentation. Despite this, though, the design group's first urge was still to sit down and immediately create a web page. It was quickly realized, however, that to be able to complete the web site in a relatively short amount of time (one semester) some organization was required. Before the thought of designing pages could be considered, an overview of the project would be needed. To this end, a Software Requirements Specification was created.

Software Requirements Specification

The Software Requirements Specification (SRS) gave a point to start from and outlined the group's goals; it directed all actions through the entire design and construction of the site. The SRS was comprised of several informational parts: an overview of the CSDL site as an organization, an evaluation of the then-current CSDL site for its strengths and weaknesses, an evaluation of similar research and development sites for their strength and weaknesses, a determination of the uses and users of the new site, a determination of the goals that the new site should meet, and finally, guidelines for designing the new site. This process, although several weeks long, was well worth the effort.

By collecting information on the CSDL organization and evaluating the then-current site, the Aziza design group familiarized ourselves with the structure of CSDL and the nature of the information that would need to be organized. The organization of the current site gave an idea, at least in part, of how the updated site might be presented.

Doing evaluations on both CSDL's site and other similar sites focused attention on what good ideas other web sites were employing and also provided first-hand experience with some ideas and formats not to use. Information that was just as valuable to know as the former.

To determine the uses and users of the new site, the group interviewed each member of CSDL. These interviews clarified what the customer (the CSDL members) expected out of the site - who they thought would use it, the information that they wanted available through the site, and a basic idea of how that information was to be presented.

With all of those things in mind, the group developed the web site's functional requirements and design guidelines. The purpose of these were to give the group a direction to go in with the development of the new site. Spending time early in the project to get a good idea of what the new web site was required to provide eliminated doubling back and extra work due to forgetting any needed functionality.

After completing the SRS, the group had a fairly good idea of what information the site was to contain, but it was still amorphous. The lines along which CSDL's information would be divided and presented were not yet articulated. To formalize those decisions, the group created several schemas related to the various sections of the site.

Schemas

For this project's purposes, schemas were detailed listings of the required information for each element of the site. Determining these lists moved the project along on several levels. First, it necessitated the decision of what sections the site would have, i.e., a section on Members, a section for research projects, a section for software tools, etc. This called for a careful consideration of the categories in order for the CSDL site to have a firm semblance of structure without being too rigid. It would have been extremely bad design if adding something to the site called for its restructuring. At the same time, it was not desirable to create a miscellaneous section just to have a place to put everything that would not conveniently fit into the available categories. Secondly, the decision of what information each section would contain provided a template for each section, meaning that each instance of a section would have a formalized structure and an organized, uniform feel.

Overall, the schemas provided an overview of the logical structure of the site. It allowed a visualization of the interlinkings of the different sections and gave a short listing of all information which would be provided. Once the schemas were completed, individual Aziza members were assigned to put together different informational pages. For instance, Jennifer was assigned to complete the member page of Carleton Moore and the research pages for his projects. Although the layout of the site was not yet complete, the schemas told us what information was needed and how it would be organized on the page.

The development stage of the project concentrated on formalizing two things - navigation and message. While a bad navigational system is never desired, an appropriate one can add an extra dimension to the represented organization. Consider how appropriate it would be for an art dealer to organize their site as an art museum; each wing dedicated to a different style, each room dedicated to an individual artist. From this illustration one can see how closely layout and theme are tied and also how a fitting or complimentary theme can strengthen the overall impact of a site.

Because the layout of the site was such an important and key element, the best way to decide on a system was to prototype different ideas, and present these to the group for feedback. These prototypes gave the CSDL members something concrete to look at and think about. In this manner, the group was able to show the look of various options, such as frames versus no frames, and different types of navigational schemes.

Pre-coding Prototypes

The first navigational scheme ideas were limited to paper. Each member of the Aziza group brainstormed extreme ideas of web page design without any regard to the work involved. The main idea was to obtain as many ideas as possible, from classical and mainstream, to avant-garde and wacky. It was possible that the group would decide against using any of the first few prototypes, but the real purpose was for everyone to present their best ideas, running the gamut from one extreme to the other. As the group examined the different navigational paradigms, it was agreed that the more outlandish ones, while being greatly liked, would prove to be too difficult to upkeep, and so initially a traditional "side-link" design was chosen.

The first request by Dr. Philip Johnson, the "manager", was for the incorportation of a large photo as the background of the main page. This photo would have clear separations of sky, mountain, sand, and water; each element/section of the picture would correspond to a different section of the web site. This particular idea did not survive, as the group decided that there was a need of more than four sections in the site, and creating a viable image map would be nearly impossible, due to the different video resolutions available.

Alpha Prototypes

Instead, The Aziza design group decided to use half of the suggestion: using a large photo as the background. The first prototype consisted of multi-colored buttons, based on standard colors defined by the W3 organization. The initial idea was to have a side bar of buttons, where the background of the side bar corresponded to the button selected. This would enable the user to know which section they were in, giving both functionality and color to the site. However, this scheme proved to be too gaudy, and the group rejected it almost immediately after prototype completion.

The next prototype used the same idea of multi-colored buttons, but in a frame format. The idea of having the entire background as a large picture was dismissed, and a tiled background was used instead. More things were added to this prototype, such as a logo and more wording, but all still felt that there was great room for improvement.

The feeling at this point was that thinking together as a group was leading nowhere, and so the group decided to split up and come back with four different ideas from which we might choose from. One idea included using pictures as the back of the buttons, and having a neutral background for the rest of the site. Another idea used an applet that would provide pop-up menus corresponding to the button that the mouse pointer was over. This applet was well received by both the group and Dr. Johnson, so this idea would be expanded in future prototypes.

Beta / Delta Prototypes

The group experimented on different ways of using the navigation buttons as a location indicator, but none did as well as the original applet method. Another avenue of design came forth at this point - the idea of using an antique aloha shirt design as the background of the buttons. The design wrapped across the buttons, and an antique-looking script was used as the font. Despite the fact that this design issue had nothing to do with navigation, all agreed that it looked better than the original button set, and so the decision to expand on these buttons was made. The group used this new site layout along with the java applet, and all felt that the development was coming along nicely.

A new point, however, was brought up during the presentation of this layout - that of screen real estate. When looking at the main page from a browser running 640x480, the buttons filled over 1/3 of the screen, which was unacceptable. To solve this problem, it decided to greatly shrink the buttons and put them on the top of the screen rather than down the side. The aloha-shirt design was used again, but use of a different font was needed in order to fit on the smaller buttons.

The group felt that they were close to finalizing the site design, but Dr. Johnson introduced a new design issue--that of having a site map on the screen for navigation. The group implemented this, using the Delta1/Top button prototype, and adding a list of links along the side of the screen. The prototype was presented to Dr. Johnson and the group waited on his approval.

Gamma Prototypes

An email was sent out by Dr. Johnson to the group, discussing his new vision for the site. Instead of choosing between top navigational buttons and a side site map, the two were to be blended together into one. The CSDL banner was to be removed, and replaced by a logo. The site map was to appear on the main page, giving it a functionality other than that of a welcoming screen, while the navigational buttons would remain at the top of all other child pages in the site. Furthermore, the Comic Sans font was to be used throughout the site, matching the many posters which were made by CSDL. The group agreed on these ideas, and implemented it in time for a presentation to the CSDL group.

During this meeting, some design aspects were brought out by the group members. Perhaps the most controversial aspect of the site was the large picture background, the only design element that had been retained from the original alpha prototypes. It was agreed upon that the picture was too "tourist bureau"-like, and gave the wrong first impression. This picture also made it very hard to read the text, and so the task of finding a new background picture was made definate. The buttons were also scrutinized, and a request for better buttons was made. Because of the changes to the site, the aloha-shirt design did not fit in anymore, and so these were disgarded.

Final Touches

The new CSDL site design was almost finalized, with only some minor graphical enhancements to be changed. The logo was redone, first using CorelDraw to edit the original .cdr file and include the name of the laboratory. This was then exported into Photoshop and shrunken, keeping both the clarity of the logo, and the compression of the JPEG format together. The buttons were also changed, discarding the original design of the aloha-shirt background. The group saw the problems of having only selected and unselected buttons, so a "focus" indicator was added to our buttons. This indicated if a user was in a section, but not necessarily at the main page of the section. Several prototypes of these buttons were made, and an oval design was finally chosen. The group also decided to obtain better background pictures, and these were chosen from the collection of Douglas Peebles, a professional photographer. His permission to use his pictures on the CSDL site was approved, and a script was run to rotate these pictures daily. Finally, the decision to retire the Comic Sans font from the web site was made. While being popular among some group members, and Dr. Johnson, the font took up too much on-screen real estate, and was not very readable in large blocks of text. It was agreed that the font specifications be left up to the user's browser.

As of this writing, the CSDL web site design consists of oval buttons at the top, with a rotating picture background. This site layout has evolved greatly from our original vision, yet the first idea for the site is still in place: the photo background. Although some of the more innovative ideas thought up in the beginning were not used, it was learned that design of a navigation tool for a web site involves not only the look of the graphics, but also the functionality and ease of upkeep for the site administrator. In an organization where no one is able to devote their time to upkeeping a site, as is the case with CSDL, the graphics should be simple and easy to edit.

Once the final layout for the site was developed, the group was able to get down to the actual building of the new site. Each member of the group had individual assignments of which members, projects, and areas of the site they were to work on. This allowed everyone to work on our own. What this meant, however, was that communication was crucial. Miscommunication could have created inconsistencies, leading to double work fixing mistakes. The time spent working on the SRS, schemas, and etc. would have been for naught.

In an effort to avoid this, our group posted everything it could think of. In this way, all information was always available. The time spent was worth it - especially if a member missed a meeting. With everything available on-line, the member could easily and without hassle, catch up to the rest of the group. Following are the types of documents that we utilized to keep group communication flowing:

Meeting Notes

Each week the Aziza group had anywhere from one to four meetings. One of the weekly meetings was a class meeting in which the group would meet with Dr. Johnson. The rest of the meetings were group meetings, where the group would get together and iron out project details. Because a great deal of information was discussed during each one of these meetings, it was invaluable to have the meeting notes posted on the web. It allowed us to go back and review decisions at any time of the day or night.

"To Do" Lists/Deadlines

While each meeting was described in detail, it was useful to have a location to keep a "laundry list" of things to do. There was a To Do list for each meeting in which assignments were handed out. The To Do lists had areas for members to insert a "Yes" once a given task was completed. These provided an overall view of both what had been done and what was left to do. It also served as a reminder for who was doing what assignment - occasionally there were times when we would become confused over who was responsible for what task and looked to the To Do lists for clarification.

General To Do List

The General To Do List was put together as the project neared its close and the things left to do could be listed. Its content was added to as things were found that needed to be done and updated as they were completed. Having this list and watching as its items were checked off gave the group a feeling of accomplishment. Each item that was completed was one step closer to a finished web site.

Monthly Calendars

As the project came to a close, deadlines came due quicker and more frequently. Monthly calendars gave us an overview of when things were due in relation to each other. Seeing all of the deadlines on a monthly calendar helped to reinforce how crucial time management would be in the last few months.

Hours

Each member of our group kept an accounting of how his or her hours were spent on the project. During the project, it was a useful gauge for how much or how little was accomplished for the week. These summaries also showed in black and white how efficient the group or individual was. If a lot of hours were recorded, but there was little work to show, then something needed to be changed. Conversely, if relatively few hours were being recorded, but a lot was accomplished, then there was an ego boost for the week.

At the end of the semester, keeping the hours allowed the group to view how time was divided between the different types of work and how the work level had its varying high and low points. Keeping this accounting brought home how the type of work changed throughout the project's life time. About half way through the semester we needed to re-define the time categories that had previously been designated.

Standards

The following documents were put together for the purpose of coordinating the work methods of four different people. As mentioned earlier, one of the toughest parts of team projects is communication, especially communication of standards. Without standards, it is far too easy to spend time re-doing work that could have easily been done in the correct format the first time. It was for this reason that standards for our work were created.

Standards - Site Standards

These refer to general HTML writing standards that the group tried to adhere to while writing the individual pages. This not only included the nitty-gritty of HTML writing, such as using relative links instead of hard-coded links, but also examples of standard headers and footers to be used.

Standards - Templates

As noted before, each section of the CSDL site had a schema written up for it. Since each instance of a section category, such as each member's page, was to contain generally the same information, it was natural to have produced template pages that contained the basic structure of each type of page, pre-formatted and only waiting for content to be filled in. This cut down considerably the amount of writing that would otherwise have had been done and also on any content inconsistencies.

Trying to maintain a site (even a small one) is a never-ending task. The site adminstrator is forever fixing broken links or adding and changing the contents of the site. Because of this, the Aziza group dedicated a section to site maintenance.

Man Pages

In the FAQ section of the CSDL site, there is a question/link titled "How do I update the site?" Under this link is an index of Manual Pages. This is a listing of pages that give instruction on how to maintain the site: from moving a member to the Ohana section to restarting the server, they provide instruction and information on how to do it all.

This section was created because the process of maintaining a site can often be involved and sometimes confusing. By providing a checklist on how to do things, whomever is the maintainer does not need to have intimate knowledge of the inner working and inter-linkings of the entire CSDL site. They just need to have a fair knowledge of HTML and UNIX commands and can quickly update the site with confidence that they are not missing or forgetting something.

WebTester

There is also an installed a utility, WebTester 1.05, that checks for broken links and orphaned files on a site-wide basis. It runs nightly and automatically posts its' error results under the FAQ page's question, "What are the results of the most recent link check on the CSDL site?" It is an easy way to keep the site running error-free and smoothly.

Usually, testing and evaluation are combined issues. In respect to the CSDL web site, however, the two have been seperated. In this case, testing refers to internal evaluations such as goal achievement and a few technical concerns.

Previous Site Versus Updated Site

One method of testing the new site's qualities is to compare it to the previous site. Following is a brief evaluation of the old site and a listing of the improvements made by the new site.

Previous Site Versus Updated Site - Evaluation of Old Site

1. Strengths. The major strengths of the previous CSDL web site were its speed and abundance of information. The loading of the main page was very quick because the ratio between text and graphics highly favored text. Information on the major projects were quickly accessible through links near the top of the page. Navigating through these links provided a great amount of information regarding not only the projects, but also the general ideas behind CSDL.
2. Currency. The previous site had not been revised for some time - since August 28th, 1997 - there were many broken links, missing graphics, and orphaned pages. Also, the bottom group picture contained many members who were no longer an active part of CSDL.
3. Unimpressive appearance. The site did not take advantage of newer technologies that the web offered. It appeared that colors had been abandoned and design was almost non-existent leading to an unfinished feeling; as if plain text had been converted into HTML to get the pages posted.
4. Readablilty. Most pages consisted of plain text across the screen, or lists of documents. The use of headers, at a minimum, could improve the portrayal of information. Further changes, such as fonts, sizes, and columns, could help to improve page layout.
5. The site as a recruiting agent. The site may have been considered "boring" to some members of CSDL's audience. Since one of the primary goals of the web site is to attract new members of CSDL, the low tone of the site may have been detrimental to accomplishing this task. There was nothing in the site that indicated a desire to sign up new students.
6. Feeling of non-collaboration. The site contained an immediate branching into the projects. This created a feeling of disjointedness been the pages. Each seem to have been written and organized differently from the other, giving an ironic twist to the goal of "collaborative" software development.
7. Lack of organization. The last link in the Projects section included a primer which was comprised of a lot of informative documents on CSDL and information on how the group operates. However, this inclusion in the Projects section seemed out of place and may have been hard to find when being sought for by a viewer.
8. Publication lists. The previous site expedited finding the technical documents by dividing the documents up into different categories, such as by author or by subject. However, these links were not complete and the Master List was very long, making it hard to search through, and tedious to load.
9. Search engine. There was no search engine or means to search the site.
10. Lack of member visibility. The way that the site was organized, members were second in order of importance to the projects - information about them were sub-linked into the project sections.

Previous Site Versus Updated Site - Improvements

1. Top page. The original site had a top page that contained two major "sections": one with links to seven projects and one with links to publication indexes. Besides being out of date, this organization was superficial and incomplete. The current top page provides six major sections with over 50 links. The page is up to date with double the number of research projects. It also provides "first class" status to members, affiliates, and software tools produced by the site, providing a much fuller representation of the mission products, and relationships of CSDL with other groups. Furthermore, the top-page is organized such that it prints out in one page: i.e. it occupies about the same amount of space as the old one!
2. Navigation. The old site had only a single navigation link from interior links back to the top page. The current button-based navigation bar can take you to any of the six major sections.
3. Search. The old site had no search mechanism. The current one has a nice search facility with pre-built keywords.
4. Page formats. The old site had no common representation for information. The new site provides template structures for each type of information (research project, software tool, member page), that help ensure the completeness and consistency of the information. In addition, the research template is consistent with the ISERN format.
5. Addition of java-based tools. The new site is designed to enable CSDL java-based tools to be used from a browser-based interface, using the servlet mechanism.
6. Section summary pages. The old site jumped from the top level into the details. The new site enables people to obtain:
   1. an overview of CSDL by reading just the top-level page. (1 page)
   2. a more detailed overview by reading just the top-level page and the single page section summaries (7 pages)
   3. even more details by looking at the research, software, member, etc. pages.
   4. the most detailed view by following links from (c) to reading actual technical reports, design pages, etc.
7. Project and tool distinction. The new site makes a critical structural and semantic distinction between the tools produced by CSDL and the research produced by CSDL that may involve these tools. The old site mixed these separate issues together.
8. Sponsors/Affiliates. The old site made no mention of sponsors/affiliates. The new site makes them first class members.
9. Site evaluation. The new site contains an evaluation of its design and effectiveness, thus making itself a research project. The old site didn't.

Meeting Functional Requirements

Another way to internally test the success of the new site is to look at the functional requirements that were listed in the SRS and the ways in which their achievements were approached.

This is a listing of the main point of each of the functional requirements and an accompanying discussion on how they addressed:

1. Present CSDL as an organization that has produced and is producing high-quality research and software and is composed of interesting, friendly people.

   This was indirectly addressed by creating a Research section, a Software Tools and Tools section, and a Members section. The Research section brings attention to the fact that CSDL is a Research and Development group. Having a distinction between Research and Software Tools and Tools emphasizes that CSDL is not just a bunch of students who got together to write programs. They are a collection of graduates, undergraduates, faculty, etc. who perform valuable research and who produce marketable products. The members section brings personality into the site and provides a formal area where the group members can represent themselves.

2. Provide up-to-date information about all group members.

   Each member has their own page in the site. Once they are no longer members of the group, they can be moved to the Ohana section. In effect this takes them off the active research list, but does not cut them off from the group.

3. Organize information in such a way that a user is not overloaded by too many choices or too much information on a page, yet is not required to follow one given path or ten links to find a certain page.

   The site was designed with the intention of having an even layering of information. All of the navigational pages (the main page and the summary pages) not only provide links, but also provide a lot of summarized information. If more detail is needed, the user has to merely click on a desired subject.

   There is also a strict navigational system implemented throughout the site. Each of the pages in the first three levels (main page, summary pages, pages linked to from the summary pages) has a top-situated navigation bar. The buttons on this bar provide direct links to the main page and the summary pages. In all pages throughout the site, except for the main page, there is also a bottom-situated text navigation bar. It provides the same links to the main page and summary pages. Linking only to the main and summary pages provides a low-cost, controlled, yet still flexible, way to navigate throughout the site. It also sends the user to a position in the site where they can quickly re-orient themselves if they happen to get lost or come to the CSDL site via an inner-page.

4. Make all existing CSDL web pages and technical reports accessible from a single home page. At present there are many HTML pages in the CSDL directory that are orphans or grand-orphans. These need to be discovered, organized, and linked in.

   All pages in the CSDL site are accessible in some way or form through the main page. The WebTester's service provides a daily listing of of broken links and orphaned pages that can be used as a reference when maintaining the site.

5. Present all information on attractive pages that are visually unified and easily navigable, without the use of large or distracting graphics. Every graphic should have a purpose that is clearly explained and/or easily understood. No one feature should overpower any other and every thing must lend itself to heightening the readability of the information.

   As of this writing, the CSDL site uses one relatively large graphic for the background of the navigational pages. This does, however, have a purpose. For one, it shows off one of the advantages of joining CSDL - the possibility of working in beautiful Hawaii. Secondly, the graphic is faded enough so that it is not overly distracting and provides that "something" that distinguishes the CSDL site from others. Moreover, the graphic disappears from anything deeper than the navigational pages, replaced by a background of a solid, complimentary color.

6. Provide applet-based interfaces to some CSDL systems such as JWiz, JavaLOC, JavaDiff, Mako, and Honu to provide "community services".

   A mechanism has been provided where community services, such as JavaWizard, can be run. A java web server has been installed on Natasha to safely run services without the security hassles previously faced on www.ics. Within the site, adding a service function to an existing project was designed to be a simple process.

7. Provide a flexible search mechanism for locating technical reports and web pages by author, topic, keyword, word in title, and year.

   Searching the web site has been given its own section. However, it does more than just search for technical reports. It does site-wide searches on any word or words that the user chooses and also provides several sets of keywords that the user can choose from to help direct their search.

8. Design pages in such a way that they are search engine friendly. This will involve using the META HTML commands for each page that we would like a search engine to be able to find. We also need to submit the main pages to search engines.

   We have not met this requirement.

9. Present CSDL projects in such a way that the ISERN relationship is explicit and in a format similar to projects presented on-line by other ISERN organizations.

   The general outline of each research page follows the ISERN standard. Also, if a project has an ISERN affiliation, it is noted in the Participants section of the research or tools page.

10. Promote CSDL's activities within ISERN by providing ISERN-related links.

    Each research page contains a section where participants are listed. If the project has an ISERN affiliation, it is noted there. Also, ISERN is represented in the site's Affiliates section. There is a link to the ISERN home page provided in that area.

11. Show with logos, CSDL's affiliation with other organizations such as DEC, ISERN, Tektronics, NSF, Worldpoint, PICHTR, and Makai Ocean Engineering.

    All industry partners and affiliates are given their own section called Affiliates. This section contains their logos and a short description of the nature of their affiliation with CSDL.

12. Produce instructions for maintenance of this site after the 691 Web Design Group has moved on.

    There is a question under the FAQ section, called "How do I update the site?". It links into the manual pages index - a listing of help pages that gives a breakdown of how to update and maintain the site.

13. Incorporate active information about the process of developing the web site.

    This document is one of the many that relate to the design and development of this web site. The Aziza research project is dedicated to documenting a case study of WWW Research and Development site development.

14. Provide a link to the "How to Become a Hacker" page.

    This page has not yet been written.

Site evaluation of the updated CSDL site was handled in two different ways. One way was to hold in-depth interviews, or interviews in which members of the Aziza development group physically sat down with an evaluator and led them through the site. The second way was through a questionnaire sent out via e-mail asking for 10 minutes of the reviewer's time - 5 minutes to be spent looking around the site, and 5 minutes to send back the reviewer's first impressions.

In-Depth Interviews

Each interview followed this general outline:

1. Find out how much they know about CSDL.
2. Their first impression of the page.
3. Have they ever visited the old site?
4. Ask them to find the link to their first section of interest as shown in the table above. Once there, ask them for their impressions:
   1. enough info?
   2. too much info?
   3. is it clear?
   4. is it interesting?
   5. any other comments
5. Ask them to find the link to their second section of interest as shown in the table above. Once there, ask them for their impressions:
   1. enough info?
   2. too much info?
   3. is it clear?
   4. is it interesting?
   5. any other comments
6. If applicable, do the third section of interest in the same manner as the first two.
7. Let them play around and get whatever comments they give us.

In-Depth Interviews - Response Summaries

Comments from the in-depth interviews were consolidated and grouped according to subject. The responses/comments were:

Background Image

• Background is good, attractive, but a little hard to read.
• Background is nice to keep the eyes from getting tired.
• The background is OK.
• Likes the background. The photo is nice, not distracting. Very grabbing.
• Likes the background picture, says it makes the text stand out. He doesn't like pages with plain backgrounds. Says it draws attention down and makes you want to scroll down. Feels that it is not too intrusive, but is interesting and soft.
• Likes the rotating picture idea.
• Likes the fixed background (demo done using IE).
• "I love the background picture. It adds an extra dimension - I just love it".
• Didn't notice the background picture until she started scrolling and it didn't move.
• "That's nice, now that I'm looking at it. Very pretty. Very soothing."
• "The background is muted enough."
• The small font size does not compete successfully with the background image.
• Might consider using background images as a navigational aid. A different background image used throughout each section of the site.
• On graphics...might be nice if the background on some of the secondary pages was different.

The response on the background image of the CSDL web site was overwhelmingly positive. Although we expected this issue to be more controversial among the reviewers, there were no outright negative comments about including a large picture as the background of the site. A few of the interviewees did not notice the background picture initially, showing that the pictures were not overwhelming. Most of the others agreed that the background picture added another dimension to the site, either just being something nice to look at, or even as a way to get the viewer to scroll down the page. Another interesting point gained from these interviews were that some interviewees had similar ideas to concepts that were suggested during the design phase, notably that of using background images as a navigational aid.

FAQ Page

• He thought the FAQ was really nice. "I loved the first bullet and the logical outline. So many FAQs list questions in the order in which they are received or added to the page instead of logically grouping them".
• "This is nice. You have a lot of stuff in here. Now what is this about Cherry Garcia? Now this I *have* to look at". The mention of Asian Cuisine at the top of this page also caught her eye as she was moving to another page, causing her to go back for a second look.
• "Hey, this page is cool...even tells you how to configure your UNIX environment."
• "Usually, FAQ questions lack the key words that get the user to know whether that is the link that they should follow...these [questions] are really good."
• Liked that page was organized by topics.
• New member orientation is good.

A lot of praise on the FAQ page was given for its organization and content. Not only did the page provide useful information on CSDL and related topics, it also piqued interest because of the clarity and tone of the writing contained within. It seems that most interviewees had been disappointed by other FAQ pages encountered upon the web, either because of their disorderliness or their boring tone. The extra time spent on organizing the CSDL FAQ page seems to have paid off in providing a positive image of the organization.

Member Pages

• The term "ohana" was not clear to him - says if we use Hawaiian terms that we should put English explanations in parentheses.
• Danu's picture is bad. Should retake the picture ASAP.
• Pictures on the personal homepages look blurry around the person's head except for Jay's picture.
• Likes the pictures with names very much. Feels that he kind of knows these people now, and that if he were to call CSDL up he would have "a better feeling - I would hold back less and be more inclined to communicate freely".
• There was enough information, there was not too much information, it was clear, it was interesting.
• When asked for any other comments, she said that it seemed as though the order of the sections was wrong, and that it would be more appropriate to have: Professional Experience, Publications, Presentations, Software, Skills, Education.
• He says that the depth of information provided made CSDL members "exquisitely recruitable" and that there was definitely "enough [information] to target". (comment from a recruiter)
• He said the Members page with the sunglasses was "offputting" at first, until he went to the individual pages and saw pictures without the sunglasses "It's probably a good idea, actually, you should keep it".
• Members need to have address, phone number and e-mail link on the individual pages. If the page is downloaded as a resume page, it does no good to have the information only on the main page.
• As far as "fit and finish...the member pages get right up there with a picture and the important information"."
• She was concerned that the picture could make it difficult for someone trying to comply with EEOC regulations since it divulged race and gender. She especially liked the LEAP description "This is nice. I like this section. This is the kind of thing I would be interested in [as an employer]. If he wrote this himself it shows that he is at least literate. You wouldn't believe the mistakes you see on some resumes and the terrible English".
• "EEEK....what is wrong with Danu's picture?"
• "What the heck is with the shades? hehehe...so is this like how engineers try to act cool even though the whole world knows that engineers know way too much calculus to be cool...?"
• "I like the history...i.e. the short info on when each member came to CSDL and their contact information."
• "Maybe you need to play with the color table on Danu's picture."

Several comments referred to the pictures taken of the CSDL members. Using pictures of the members in sunglasses on the Member Summary Page seemed to provide a comedic break in amongst the technical information provided throughout the site, somewhat relaxing the user. One thing to consider is that since EEOC regulations require the disregard of race and gender in employment, the pictures may make a recruiter to skip over these pages, in an effort to avoid a potential problem. However, one interviewee stated that the pictures made him more comfortable in communicating with the members, which could be a positive point when recruiting.

A particular comment that appeared several times referred to Danu Tjahjono's picture, in the "ohana members" section. All of the pictures in the ohana section were sent in by the member themselves, and Danu's picture was not of the same color depth and resolution than the others. It seems that most users expect consistency across the web sites, so when one object is different from the others, it draws a lot of attention. This can be used positively, as in drawing the user's attention to a particular point in the page, but if this result is not purposely done, it can seem out of place.

Most interviewees felt that the amount of information provided on the individual pages was of the right length; it gave enough information to target them as potential employees without being too long to read. Some suggested small changes to the individual members pages, such as a different ordering of categories, or address / phone number information.

Navigational Bar

• "The menu buttons have a nice effect on showing which [section] you are in...
• If you had the navigation bar as a frame on the side then you wouldn't need one at the top and the bottom, and you would never need to scroll to get to it.
• Good for telling you where you are [in the site].

Of the three comments received on the navigational bar, two were positive statements about the use of "focus indicators" in the site. This seems to indicate that users not only want a quick method to accessing areas of the site, but they also wish to know where in the site they are at that time.

On-line Service

• Likes the star (for JWiz) because it draws attention, but feels that it is then unclear what the rest of software tools are - what are they if they aren't on-line services?
• "That's very helpful!"
• "What's an on-line interface?"
• It is good to show some demo programs.

Half of the comments received on the on-line services were of confusion on what an on-line service actually was. This could be due to either poor word choice of the Aziza group, or due to the lack of similar services on the WWW. Taken positively, this would mean that the CSDL site could be a forerunner of service-based web sites, something that is only starting to be used today.

Research Pages

• There was enough information - told exactly what the research was, gave good examples.
• There was not too much information, were not overloaded.
• On the AutoFAQ research page: It was clear except that more clarification was needed that the research had not actually been done yet.
• On the BRIE page: "What does that stand for?" There was nothing on the BRIE research page that clearly explained the abbreviation.
• On the Research summary page: "Good for a person who knew what they were looking for."

The interviewees who viewed the research pages seemed to be split up into two groups--those who knew previously about the research projects, and those who had no prior knowledge. This brings up a point in research web site development: providing content for more than one audience. It is sometimes hard to find the correct balance between explaining your information to new visitors to your site and not insulting current members or affiliates of the group. Future work should be done to the research pages to give a little more background information for those who have no prior erudition of the topics involved.

Scrolling

• Opening page should fit on one screen, especially the pages with rich graphics. Scrolling [on those pages] is not attractive to users with slow computers and modems.
• Wish that everything was visible on the navigational pages without scrolling.
• On the publications page: "Seems as though it could all fit on the same page." This was referring to her dislike of scrolling navigational pages.
• On publications split by research project: You have room, so you should spread out the publications table out more.
• His first impression was that the main page was very long. He said he was browsing on a laptop with a 12 inch screen and for him, a "friendly page means one that is complete on the screen".
• The first page was so long that "Members" did not appear on the first screen at all. He explained how as a headhunter he spends two or three hours a day surfing the net for prospects and that despite the presence of the individual pages he would have skipped over our site altogether in his normal work mode. For his purposes, the names should be first in the site map.

Of all the issues regarding the look and feel of the site, the scrolling of the pages was received most negatively. Although not everyone had something to say about the scrolling, those who did stated that they disliked it. Unfortunately, the amount of information included within the CSDL site made it very difficult to prevent scrolling. One way around scrolling is to have the page split up into multiple one-screen pages, but this would take even longer to view than if the page scrolled. Some sections will continually grow larger, such as the publications section, and so reformatting of a page in order to have it fit on one screen is unreasonable. Another point to keep in mind is that the web is not the same as a desktop publishing mechanism. One cannot insure what the visitor to the site will be using as a browser, and therefore no one will know how the page will look on the visitor's screen. Because of the nature of HTML, this issue will have to be unresolved until new standards are made regarding the formatting of pages.

Search Engine

• Should have two search engines: One detail and one quick search. Have the quick search located on the first page.
• When he goes to a page, he does not wander around. He likes to go straight to the documents containing his interests. The search engine is very helpful in this case.
• Search engine is very helpful in locating information.
• Visited the search page and liked it very much. "You guys are really thinking out there. Nice. Very nice!". Liked the keyword lists. "Cool!"
• When searching for information on Danu: "I expected to see his picture come up next to his page information."

The search engine seemed to be a good addition to the CSDL web site. Because of the large amount of information located in the site, a search mechanism was needed to allow users to find specific pages related to their interests. One interviewee stated that the page should have another search engine on the main page, an idea that was removed from the final layout. It was assumed that since the search engine was one link away from the main page, that a second search mechanism would be redundant and costly for on-screen real estate. This is not necessarily the case with all users--those looking for information quickly would appreciate the time saved by having the search mechanism on the main page.

• "Academic sounding. It's a little fuzzy just exactly what you do." There are over 40 hot spots on the first screen plus text, that is too much information.
• He prefers the map rather than links. (Changing color of links once the link has been visited).
• He likes the index on the left (good for people with slow connections and provides the information that he would want to look at to find out more about CSDL).
• He likes the menu on one side separated from the general text.
• "That yellow star is really hard to see against that background [in the Software Tools section]."
• "I almost understood the star as a sign that something was new - how do you know what is recent?"
• It would be nice if you had a "new!" label so you could tell when something has been added.

One interviewee seemed to want the differentiation of the main page from the site map in order to make it less overwhelming. The original purpose of including the site map on the main page was to give it some importance rather than it being just a prosaic, introductory page. It was hoped that those who were interested in the introduction of the site would disregard the site map, and vice-versa, but this was not the case for all. However, others appreciated the site map being on the first page, allowing for a fast way of navigating around the site. One interviewee even mentioned specifically the separation of the site map from the rest of the main page, showing that most users view the page as being made up of two distinct items.

The star used in the Software Tools section was distracting for some, either by being hard to see, or by being similar to icons used on other pages to represent new items in the site. When designing new sites, it is always wise to consider current standards used in other sites. Using something such as an icon in a different context can be confusing for the user, and in this case, also bring something to mind which the user may feel is lacking, which was a "new!" label in the map. Future revisions of this site should find a better replacement for the yellow star, as well as considering adding a "new" label to the site map section.

Software

• She felt that we almost needed to "advertise" the various software products. She personally would have liked to see the number of people who were using or who had downloaded each product and would have like to read quotes from users of the software, rather than just our descriptions.
• "There is absolutely nothing missing from [the Egret research page]. It is excellent...at first I thought, so when are they going to mention that source code is included in the distribution and then they did, and then i wanted to see what the interface looks like and it was there!"

Although one interviewee felt that the research pages were fully complete, another wanted the group to advertise the software available. This may be very helpful when looking for new affiliates, or new users, and a nice way of showing feedback from users. Future revisions of the site should include user comments on the software packages as a minimum.

Software Download Form

• Shouldn't have the mechanism ask for an email address before the download. Make that an option after download. People will give a fake address anyway if they don't want to disclose their email.

The practice of requiring a pre-registration before a download seems to be annoying to most users. It may be better to request registration rather than to force it, providing download information and not irritating users of the site.

Third-level Graphics

• There is no need to have graphics in the 3rd level and below. People going down to that depth would not care about the layout, just interest more in the information [that] it provides. People should be getting tired of pictures by then.
• 3rd level pages are ok. Graphics are not important at that level. 3rd level and down are fine. Page should only contain information. Graphics would be distracting at those levels, clustering the viewer's attention.

Some interviewees felt that graphics on the third level of the site should be removed. While it is true that those who would actually go to this level would not be demanding of the images, the Aziza design group felt that the inclusion

Overall

• Organized.
• Better layout. Easier to find information than on the old site.
• More professional [than the old site].
• Attractive.
• [Would be] nice if you had a guest book.
• It was clear.
• It was interesting.
• [He] felt that it could increase his knowledge.
• When asked what his impression of CSDL was after looking over the site for about 20 minutes: "If I joined, it would be a lot of tough work - but that's a Good Thing. People there seem very free about giving information and help." All-in-all, he thought the site was well thought out. "The people that did the work on this took the time to do it right and to do it well."
• On a powerpoint presentation "That was really cool. Really slick. Nice colors, that's very nice." She thought having on-line PowerPoint presentations made our site stand out.
• "Well organized". "It's easy to get around".
• States that this site would make a difference to her as an employer. "A great tool for a student. It would be a great thing to put on a resume, 'See www.blah.blah.blah'".
• He suggests that we provide our pages to such services as headhunter.net, Career Mosaic, and ESPAN. These are where he spends a lot of his net time, and would have found our member pages only if he was in "highly precise targeting mode". (comment from a recruiter)
• "You need to try to find ways to relate the research and software projects to what is real and familiar to people who are slightly outside your specific areas of research. Don't use general terms. You need more specific examples of people using various things."
• "Overall, I'd rate this [site] a 100 on user ease of navigation."
• "If just surfing, I want to quickly jump in, get something, and jump out. If motivated, then I would sit down and read. If I am already mentally committed [to really looking at the site], then I am motivated [to read.] Chunking your data can't hurt."

Overall, the comments were very positive regarding the new CSDL web site. Of those who had seen the previous site, all agreed that the new site was a major improvement. The site was commended on various aspects, including the look, design, and content. Some offered some suggestions in improving the site, such as submitting it to career service web sites, and making the content more user-friendly. Altogether, however, the site portrayed a good image of CSDL as well as the Aziza design group.

It seems that the overall sentiment is that the site is pretty good, but that there is still room for improvement. Depending on who the reviewer was, the type of improvement differed, however most of the suggestions were things that could easily be implemented and would improve the quality of the CSDL site for its users - for example submitting the site/the member pages to head-hunter services.

There was an area though, that while not a great point of contention, did greatly bother the people that it bothered (i.e. either they didn't care at all or they really cared). This was scrolling navigational pages or even scrolling pages in general. Many reviewers did not comment on the scrolling, but those who did were quite firm in voicing their preference for non-scrolling pages.

First Impression Interviews

As mentioned earlier, there were two means of evaluation utilized: (1) in-depth interviews where the interview was person-to-person, then (2) first-impression interviews where all communication is by e-mail. The following e-mail message (or a form very close to it) was sent out to approximately 300 potential CSDL site users:

Date: Mon, 11 May 1998 11:10:45 -1000
From: johnson@hawaii.edu
To: uhm-ics-students@hawaii.edu
Subject: 10 Minute Research Help Request


Greetings, colleagues,

We are concluding a one-semester project on research web site design,
and would like to request 10 minutes of your time to help evaluate
our efforts. As part of evaluation activities, we are interested
in soliciting "first impression" data.  Could you please:

(a) Spend five minutes surfing the new Collaborative Software
    Development Laboratory web site at:

       http://csdl.ics.hawaii.edu:/

(b) Spend five minutes replying to this email with short
    answers to the following questions:

   1. What is your first impression of the site's layout/presentation?

   2. What is your first impression of the site's content?

   3. What is your first impression of the CSDL research group?

   4. What do you feel we should change or improve in this site?

   5. Would you like to be emailed a pointer to the technical report
      on the results of this project when it is completed?

Thanks very much for your participation. If you could reply within the
next 48 hours, that would be most helpful.

____________________________________________________________________

First Impression Interviews - Response Summaries

About 50 responses were returned, roughly 1/6 requests. The responses are summarized as follows, grouped by common subject matter:

• Nice, attractive, grabbing
• It looks pretty good. Must spend some time in designing it.
• Much better than the old one.
• Exciting images, yet make the text somewhat hard to read
• Well organized, clean
• Very easy to find things and jump directly to the items
• Does not have the "cluttered" feel that many web pages have nowadays
• About the group members' picture wearing shades:
  • Loved the 'company uniform' with the shades
  • The pictures of everyone wearing shades are pretty funny and sneaky. Yet a few others dislike this idea.
  • Not very helpful to recognize people wearing sun glasses.
    
• Close-up pictures have an eerie feel to them
• Nice colors settings for the page.
• Nice sitemap, very helpful.
• The site is user friendly due to the easy accessing of pages
• Pretty positive.
• Very easy to navigate, and could tell what was going on at a glance
• "When not loading the images, the box lines cut off the text in the banner so you can't read it. I know it's hard to do but I hate it when I can't read it. Then I tried to load the images and only half of them came through. I don't know if that's you or me, but it's never happened before"
• "I find that web page designers waste way too much effort trying to get the layout "just right", when it really doesn't seem to matter much (except for perhaps about 2% of web sites)"
• Nice, yet the images size needs to be smaller. It took 3 minutes to load the page on 33.6 kps modem.
• "The background, by the way, is very subtle on my computer {PowerMac running Netscape} and not essential if it delays load.)"
• Prefer the buttons on the side if possible.
• Buttons are nice.
• The font color of the text as well as the unvisited links of it does not show too much contrast to the BG color - blue text against blue background, a bit irritating to the eye.
• "The light blue makes me a bit sleepy though :)"
• Background photos seem to be tiled - not good - they took too long to load
• Like the layout of "Research" page. Not only do I have the project name but I immediately have a description, contact, and a date without moving to a "text only" document. This is a very nice feature and quickly allows me to navigate to the site that best suits my needs.
• The layout is very clean and readable
• It is hard to understand the abbreviations in the menu to the left.
• Mountain background isn't tall enough and I get this weird underworld mountain sticking up beneath it. Brings to mind ideas of Hades.
• Redundancy within the buttons in the top and the options in the left.
• Like the "pushbutton" images and the way that various sections have a uniform layout/style
• Prolific
• Good job. Now if only the ICS web site was half as good.
• The refresh rate when you scroll down the home page. It might be my machine but it doesn't do that at other sites. It’s real jerky.
• From the home page, in the blue box that has the member summaries and the individual members when I selected your name, the transition to your site took an inordinately long time with a pink colored background inching down my screen.
• The first time I did this I ended up re-booting my computer because I thought the time it was taking was a function of my computer seizing up. The second time I did it, it did the same thing. I don't think it’s a memory thing with my computer. The site was not terribly complex with lots of graphics - just a small photo of you and the rest was text.

The suggestions are focused on the site's design and layout. They are very constructive and helpful comments for future web designers to consider and reformat the site accordingly during the optimization of the site.More than 90% of the surveys indicates the site's overall layout, design and graphics are well-defined and attractive. The site is organized, clear making the site is very easy to navigate. However, the graphics on the first page is intensive and it takes approximately 3 minutes for loading on a 33.6 kps modem. This is an issue that we are all aware of, yet could not find the best solution. The main load is derived from the background picture, yet reducing or fading colors meaning reducing the picture quality. This concern among other comments is an example of an item in our "To Do List" for the optimization of the site.

First impression on site’s content

• Very informational
• Compact, dense
• Adequate for most purposes.
• Affiliate section is great
• "The "research" and "software" sections seemed to be rather repetitive (AEN/LEAP, etc.). I realize that there were some things that weren't, but that was still the impression I was left with."
• "Member section still a little thin; for example, from the summaries page I expected, when clicking on names to see more info ("I like to surf," "my hobbies are," whatever)"
• Seems pretty comprehensive and has some value to the interested browser, i.e. software that can be downloaded plus papers that can be downloaded.
• "Is a ‘browser-based interface’ any different from a browser? Does the word ‘interface’ really add anything? It seems to be used throughout the page as a noise-buzzword.
• Too many acronyms
• I particularly liked:
  
  • the format of the research summaries.
  • the fact that the "Research" page listed the active period of various projects
  • the FAQ for new group members

• "I was a little disappointed to find that many publications weren't available from the web site"
• The names of the software are not suggestive.
• More information on personal web pages.
• Provide downloads for J-wiz and the others
• Hi-Time very interesting
• Too much content is in the first page
• "I did read through some of the FAQ. I got through the first three levels of the CSDL Maturity .... model (I forgot the exact name). The first two levels made sense to me. I started reading a description of the different languages (?) and what you should be able to do with them in order to demonstrate competency in them and I got lost.
• the understanding of the first page contents is not visually immediate. "Textual comments on this page" window encapsulating the textual part could probably help to have a more immediate communication.
• The summaries at the beginning of each sub-section give a further overview. I really liked this multi-layer approach
• The FAQ idea is brilliant and would make it so much easier for a new person coming into the group.
  

Regarding the aspect of site's content, our surveys shows that almost everyone agreed that the site is very informational. This is the goal and also requirement that CSDL members and web designers have been trying to push during the design and construction of the new site. Some of the sections in the site such as FAQ, Affiliate gave a major good impact on viewers. There were also some issues that the site web designers should notice. The first concern was about the acronyms and abbreviation used for projects' name. Viewers felt uncomfortable because some of the abbreviation do not have explanations of what they stand for. One approach to correct this is to dedicate a section in every research page explaining about the meaning of its name. The second issue was about the length of the first page. More than a few of viewers felt that the first page should be simple and visible without scrolling. Is it a personal preference or a web standard? This is the question that requires web designers to do more researching and changing accordingly.

Impression on the site as a whole

• Good for luring prospective CSDL'ers
• Simple, yet elegant
• The introduction makes viewers want to look at all those neat acronyms and names on the left side of the main page.
• It is good to have a search engine. Very helpful.
• Up to date (according to the last-modified line) which is a good look.
• A nice research web page -- not too much flashy presentation, big lists of software and papers.
• This is clearly a busy group. There are a lot of projects going on, but most projects seem to have only on person working on them. The group has a clear sense of humor, but obviously (as the content of the web site indicates) takes research – particularly the _process_ of research - very seriously
• Page come up after initiation of search is not consistent with other web pages.
• Does a good job of providing an overview of the group and contains most everything one would want to get at the details of what you are working on.
• Last updated, contact information are helpful

The comments in this section are key points that we are looking to determine the success or failure of our semester project of web design. The comments have proven the fact that our design of the site met the functional requirements we had set out at the planning stage earlier in the semester and our efforts putting into this project have been award. "luring prospective CSDL'ers", "providing a good overview of the group" . . . are the goals that the site web designers have been trying to achieve.

First impression of CSDL Research group after visiting the site

• It seems like there are interesting research going on.
• There's quite a bit of research projects happening there
• Small, but doing a lot.
• Sounds like CSDL does a lot of cutting-edge development
• Knowledgeable of subject(s)
• "The CSDL faculty is strong, at least Johnson, Jay and Russ are there."
• "I cannot say - in my mind it depends on more than a smartly presented set of web pages - I am not a particularly strong proponent of the 'perception is reality' school of thinking!"
• The researches are impressive.
• A software engineering group.
• "As a fellow faculty member at UH and someone who is interested in software development methods (from the Business Mgmt perspective), I'm happy to know you guys exist. During my 2 years at UH, I've been holed up in my office and really knew nothing about your center or activities. Since there might be some overlap / mutual interests, I think the site is valuable as a way of making yourselves known. It seems like you're doing and have done some interesting work."
• Looks like a bunch of smart individuals trying to find solutions for useful things (rather than high-end stuff that doesn't apply to the average surfing Joe).
• closely-knitted but very active, knowledgeable and well-connected research group
• Viable organization with good outside support. Not too big, about the right size to get something done.
• Well focused.
• Very productive and energetic group.
• Group achieved a right balance between working on products and processes
• "A handsome, learned, widely published, brilliant associate professor, P.M. Johnson"
• "I didn't realize you'd been published so much and done all those presentations and lectures and everything. I am totally impressed! And here I thought you just lounged on the beach all day!

The comments in this section shows the site's ability to organize and convey the essential information about CSDL researches to viewers. From browsing, viewers could get a good overview of what has been done for the last 7 years and most of viewers were surprised of how much researches happening in the group. The site has demonstrated a very good image of the group to the world as being a very productive, energetic and knowledgeable group. The comments in this section could have a major influence on the CSDL's future expansion.  

Suggestion on site improvement

• The word "Ohana" on the Members page is OK in Hawaii, however for people coming to the site from other places, they would probably not get the point/meaning.
• If there is a director of the center, put it on the main page.
• Replace the abbreviation menu with more descriptive titles. Since this will occupy more space, I suggest that you organize the information in a hierarchy.
• Link to "Research" and "Software Tools and Services". A short description of the contents could
  be displayed, rather than displaying cryptic initials in the main menu.
• Make more specific statement about related research and what position you are in now, even better, draw a comparison with any currently available commercial package (e.g.,MS scr_save?) if existing.
• In today's world, there are so many software products, if you can emphasize the unique, cool feature, that will fish in net surfers.
• More descriptions about the researches at the first levels. For example: There is not an explanation of what AEN stands for.
• More on-line demo
• Everything should be visible on the first page without scrolling
• The order of the sections on personal home page is somewhat incorrect. How about

1. Professional Experience
2. Publications
3. Presentations
4. Software
5. Skills
6. Education

• Sitemap button should have an ALT tag of "Main/Sitemap".
• On the B/W Mac, one viewer complained for not being able read the text on the Member profiles pages after the background loaded
• Make the links with more contrast in the site map.
• A bit clustered?
• Logo is nice but viewer expected to see the text name of the group as a title on the home page, not in small text below the button area
• "the bold headings in the right frame ("Research", etc) might be links to the summary pages directly, instead of making me mentally sort the detailed menu on the left"
• "I thought the text description on the summary pages were about a sentence too brief - I wanted to know more, but not *that* much more, not so much that I would really go through the 15 waits to see the detailed pages. If I already knew what the tools did (i.e. jWiz) it would be ok. Example: "CSRS: a tool for collaborative software review" - is it stand-alone? Is it suitable for big or small projects/groups? Does it require special infrastructure? I know I can go dig it up, but if I am just dropping in on a search, I don't have time to dig on everything."
• "The words ‘Open Source arrangement’ should be a link to more detailed information about what an open source arrangement is."
• The third paragraph on the right could have been removed or integrated with the content table. This would put your address, etc on the first page.
• To improve, however, the menu could have been framed with an enabled frameborder, preferably at the bottom. The one here takes up a bit too much space at the top, plus that the space between the top and the menu itself is bit too wide.
• Making a frame, and making it enabled border to facilitate web browser’s search with more flexibility.
• "I saw that the background is supposed to change everyday, but the light blue seemed a bit sterile and dull. Maybe it adds to readability for not being distracting, but I think that it's a bit too boring. Other than that, it looked nice for the 5 minutes I spent clicking around."
• Use the first photo as you have it and match the blue background for a solid back since you don't need the extra space the second photo takes up. Also, a GIF file at 72 dpi can look good and loads quicker than a JPEG
• Jwiz: ... Please enter in the URL of your .java file in the text field below, and press the Submit button to proceed.---> How about: Please enter the URL...
• ... as well as empirical approaches to improving ---> How about: either improving or to improve.
• Make the navigation banner on top MUCH smaller, you loose too much screen real-estate.
• Check to see if browser is IE4 and if not redirect them to where they can download it. Add lots of ActiveX and DHTML stuff. Also Microsoft Agents are nice. Add in a few LabCams, voice recognition, collaboration support, and real-time DNA computing demo.
• Use fonts and color to make it attractive.
• GIF buttons are 34 colors, could be reduced to 16 saving 33 1/3 % bandwidth
• Nice if there were links to papers etc. on Philip Johnson's page at least!
• Integrate the buttons to the site map
• The left bottom is empty. need put something there
• The color can be a little bit brighter.
• Overlap between research and tools are too large. I was expecting research description under research, and maybe tools to download on the tool button.
• More software screen shots and more downloadable or web-based software
• Pointers to other (non-CSDL) related research, papers, would be nice.
• At first blush the research topics and tools and services look like so much alphabet soup. Something like a tool tip on rollover (or status message) would help if it provided one-line definitions. Even the pages for these sections don't always explain the acronyms.
• The Doug Peebles backgrounds are nice but the small text is too hard to read in the dark areas. They should be bleached more (de-saturate and push towards white)
• The extra "selected section" cursor with the "selected button" mostly looks distracting (the 4 corner fillets). I don't think average viewers will understand the metaphor. (also it's not extensible in depth) Look for another way to locate the user in the site hierarchy. I understand it because I've struggled with the same problem. See http://www.kaiserhawaii.com for one way that I've dealt with it. Note the use of tabs and outline expansion. Subsection localization is tricky.
• I would like to also be able to click on the bold keyword in the content overview to access that area, seeing that is where I am
  reading about it, rather than having to find somewhere else to go to this site. (A very picky point though! :-)
• One usability problem occurs if you use a link in the FAQ part.There is no way back except you use the browsers back button and
  no chance to change to the homepage if the page was found using a search engine. You should back-link all pages at least using the text links of the homepage.

The long list above are suggestions collected from viewers during the survey time. Many comments, if researched properly, will make major improvements to the current site. Some other suggestions are focused more on the cosmetic change of the site. Some are likely to be personal references. Yet, together they portrait what viewers expect to see and what are the most protruding points when browsing the site.  Being able to understand web viewers' behaviors and expectation will help the web designers to bring the site closer and more pleasurable for viewers.  

Site evaluators whom would like to be emailed a pointer to the technical report on the results of this project when it is completed.

1. "Wei Qian" <
2. "GiebinkTribe" <
3. "Brandon S. Higa" <
4. "Justin-Michael Okamura" < >
5. "joel litao jiao" <
6. "Mark Waterson" <
7. "Sharon Distasio" <
8. "Elaine Yakura" <
9. "Edoardo S. Biagioni" <
10. "Nguyen D Dao" <
11. "Elizabeth J. Davidson" <
12. "David Arnold" <
13. "Linwang Lu" <
14. "Janet Rowell" <
15. "Laurel King" <
16. "Monir Hodges" <
17. "Tim Mansfield" <
18. "Yong-Sheng Zhu"<
19. "Kristian Sandahl Z/TZ" <
20. "Doug Theine" <
21. "David C. Brauer" <
22. "Ted Phelps" <
23. "lkjj" <
24. "Rod Ruggiero" <
25. "Charles Herring" <
26. "Blanca J Flores" <
27. "Jintae Lee" <
28. "Jian Ma" <
29. "Xiaowei Sun" <
30. "Pentland,B" <
31. "Jenifer S Winter" <
32. "Claes Wohlin" <
33. "Filippo Lanubile" <
34. "Gillian Sloan" <
35. "Jonah Ungacta" <
36. "Todd Blume" <
37. "Cantone" <
38. "Geraldine Fitzpatrick" <
39. "Richard Oed" <
40. "Kameo" <

Evaluation Analysis

Between the four members of the group, a total of 664.5 hours was spent on the CSDL web site during the semester. A breakdown of the categories is shown below:

Total Hour analysis

Despite all the time taken in planning out the site, HTML coding took up the largest amount of time during the semester. This was partly due to the many changes which had to be made after new requests from Dr. Johnson were given, resulting in a time-consuming search through the site for the correct replacements. Although a macro was written to crawl through the site and do global replacements, this still took time to do.

Group Meetings took the second largest allotment of time. These meetings proved to be very helpful for us, as it helped us stay in synch with each other. It was at these group meetings that we learned who would do what, who didn't do their assignment, and what was still needed. These group meetings also proved to be encouraging to each other as the semester wore down.

Writing was a major part of both the SRS and the web page itself, taking the third largest allotment of time during the semester. Writing was evenly split up among the SRS, this Tech Report, and the sections within the site itself. However, these 96.6 hours do not reflect the many hours Dr. Johnson spent on the site, revising and writing sections that he had greatest knowledge of.

Organizational work, and Other work made up about 10% each of the total work done. Orgazational work included transcribing meetings, posting notes, and various other tasks needed to keep the group running smoothly. Considering the amount of work accomplished by the group, the 10% spent on organizational work seems small in comparison. Other work included tasks such as graphics, interviews, and items which did not fit in with any other category, yet did not deserve its own category.

Web Browsing and Research took about 9% of the total hours spent. Various things were researched during the design of this site. For instance, the search engine used was compared to others, and determined to be the best for CSDL's tasks. Information on how to create servlets also had to be researched, as no books covered this beta software in detail.

Class meetings took 46 hours out of the semester. These meetings were also helpful in determining if the Aziza group was on the right track according to Dr. Johnson's vision of the web site. However, despite taking up only 7% of the hours spent, class meetings proved to be the most time consuming in the sense that most of the new changes and implementations were assigned at this time.

The programming was split up into coding forms for the web search mechanism, and the java servlet for JWiz. As this was not the main focus of the group, a small time was dedicated to doing these tasks. Finally, time accounting took up 2% of the total time spent during the semester. This tracking of hours helped with seeing the productivity of the group, as well as who specialized in what as the semester went on.

As our group found out, there are several new things that you can run into when working as a group rather than as an individual.

Developing Roles

We found that as the project wore on, each of us fell into job roles. It may sound bad, but it was actually a good thing. For one, we could usually tell who would be doing what job, and that person knew what areas were their responsibility. As an example of how work may end up being grouped, this is how our group work was divided:

Anne Disney - Anne, aside from being in the Aziza group, was also a CSDL member. Because she was familiar with CSDL she was in charge of going through the CSDL documents not related with specific projects and grouping them for inclusion into the new site. She was also our liaison with management. Many times she was able to discuss things at greater length with Dr. Johnson then we were able to cover in class meetings. This aspect of group-manager interaction was very useful and our group was fortunate that Anne was able to fill this role.

Jarrett Lee - Jarrett became the graphics guy. He would do most of the site work related to graphics. Most of the members' photographs have been touched up and all of the buttons are courtesy of him. He was also responsible for setting up the servlett to run the community services. When it was time for a formal presentation of this project, he assumed the role of group spokesperson.

Tuan Huynh - Tuan was responsible for the web server and the search engine. He was in charge of everything in these areas - from installation and setup to maintenance and documentation.

Jennifer Saito - Jenn was the group leader. She was responsible for most of the organizational work. She posted most of the meeting notes, to do lists, and standards. She also headed the class meetings with Dr. Johnson.

Decisions

As you might guess, making a decision as an individual is much easier than making a decision as a group, you only have yourself to consult. When it comes to making a group decision, there are many new things that come into play. For one thing, you need to remember that you must consult the group - while you can try to make unilateral decisions, this method is not the best for morale. After that you must be tactful and be considerate of the other members. Compromise and a willingness to adhere to decisions is also key. You are going to need to work closely with the rest of the group and open hostilities among members has never speeded up any process.

All of the niceties, though, require time and patience. When time is running out on the project, spending that valuable resource in this manner can become frustrating. The benefits of having a group all going in the same direction, though, are worth the effort.

Coordinating Communication

The same difficulty that occurs when making decisions applies to Coordination. As an individual, you tell yourself, "Okay, this is what I'm going to do" and do it. As a group, you come to the decision and then need to make sure you all know about the decision and follow it. This is why posting things like meeting notes, to do lists, and standards are important. This is also why compromise is important - you will all have to live with and abide by the decisions that the group has made.

As much as we would like to say that our research project went smoothly and there is nothing that we could have done differently, that is not the case. There were several areas in which there was room for improvement.

File Locking

If you ask anyone in the Aziza group they will probably tell you that the next time, there should be a file locking mechanism. The reason for this is that because there were so many people making modifications to the files: us, Dr. Johnson, the CSDL members, there was always the danger that one of us would overwrite someone else's hard work.

What would happen is that we would make a copy of the file and work on it off-site, then replace the now-outdated file with a new one. If two people were doing this at the same time, the first person's work would be overwritten by the second person who copied their file back into the site.

The best that we could do was to send out e-mail letting everyone else know when we would be making changes to which documents. Making back-ups of the entire site gave us some peace of mind, but it would have been a lot easier if there were an alternate method of preventing file errors.

As we look back on the process of the project, some things became apparent to us. The most striking of these has been that the outlook of CSDL itself has somewhat changed due to the update of the site. With the addition of the services section, a question that members are now asking themselves is "what can we make that is a service to the community?" So in a way, the types of projects CSDL may pursue in the future has been affected.

Regarding the two different types of evaluation, we found that first-impression, e-mailed surveys were more critical than those done in-person. In fact, the in-depth interviews proved to be slightly less helpful than those mass-mailed to people. Looking over the in-depth interviews, it seemed that either the group of people selected were less demanding, or they were more reserved in their comments. A greater majority of the comments by in-depth interviewees was positive, compared with an even ratio of negative to positive comments by the first-impression interviewees.

This data could mean a number of things. One point to consider is that a member of the Aziza design group controlled the in-depth interviews. This may have inadvertently taken the interviewee along a positive tour of the site, causing their reaction to follow suit. Another item of consideration is that some people tend to be less negative to others in-person. This is either due to not wanting to offend, or being afraid to offend. People may have held their true feelings about the site within themselves, and this may have affected the in-depth interviews.

To contrast, those communicating through e-mail traversed through the site on their own, as in a real-life situation. There was no prepared route to take them through, instead, the user looked at sections that interested them, and which they would go to otherwise. Furthermore, these interviewees volunteered their time to look through the site and comment on it. Instead of merely accepting an interview, this group of people decided to spend some of their time to help out with this research, and so their comments may be more truthful. If one is to decide to spend time on something, it most likely not going to be wasted by giving open praises. The interviewees were also able to think longer on their impressions to the site. Instead of being on the spot with another person, this group was able to spend as long as wanted to answer a question, or even refuse to answer a question at all. Finally, the in-person intimidation was not present during email. This may have relaxed the interviewee, allowing them to think more clearly, and giving more insightful information than those who were interviewed in-person.

This is just a single case study, but it has proven to be very interesting. It may be a possibility that controlled interviews do not reflect real-life browsing of web sites, and so should not be taken into consideration. However, they can be useful in finding out specific reactions about a web site, such as a certain page in a site. Some reactions to a site may not be remembered or noticed by a user, and so only an interviewer would be able to record this type of data. Nonetheless, this case study has received more subjective comments from first-impression interviews. Due to a lack of a time constraint, less pressure, self-volunteering, and the ability to spend longer on the answers, these comments have been shown to find more about what the general public thinks on a web site.

It seems that the best method for obtaining data on a web site is to perform both in-depth interviews, and an informal first-impression interview. Both can obtain data that the other cannot, and correlation between the two can be stronger than opinions from one group.

This is the list of things that needs to be done after the Spring 98 semester ended. These are the things that Aziza group could not achieve during the semester due to some difficulties beyond their reach or time constraint.

1. Detail analysis of site evaluation result.
   Several points from the initial evaluations were made, but due to the ending of the semester, unable to be researched further. Hopefully future research may be made to confirm or disgard these new points. These include:
   • Will rearranging the sections in the Member Pages prove to be better in attracting recruiters?
   • Is there a better way to arrange items to reduce scrolling?
   • Should the site map and the main page be split?
   • Should the software packages be "advertised", including positive quotes from end-users?
   • Should third-level graphics be completely removed, or added in?
   • Are there any misleading items in the site that have been standardized as something else?
2. A mechanism for users to enter their email addresses, when downloading softwares from CSDL, for future update notification.
3. Publicize the site.
   • Move the new site from port 8080 to 80
   • Back up and disable the old site
   • Add an entry to the rc script to automatically start the web server when server reboots.
   
   Due to members not having root access, the web site is unable to switch from port 8080 to 80. When a new system adminstrator for the ICS department is hired, the site will be transferred to the default port number, and the old site will be disabled.
4. Get the JWiz servlet to work.
   Currently, the servlet is functional, but because of account restrictions, and having no one with root access, the classpath of the web server cannot be set to include JWiz. When the classpath is able to be set, the servlet will be available to the public.
5. Make sure there is no more orphan pages in the site.
6. Add "~csdl/private_html/Search/bin_allsite_search/htdig -v &", a script to update the search database, to a non-Aziza member's crontab.
7. Miscellaneous items
   • Find a better icon for the "On-Line services available" programs
   • Add "New" icons for newly updated items
   • Possibly replace Danu's picture with one that matches the others
   • Possibly find better phrasing for "on-line interface"
   • Provide a less-technical explanation for less-technical users

• Louis Rosenfeld & Peter Morville, Information Architecture for the World Wide Web, 1st Edition. O'reilly 1998.
• Chuck Musciano & Bill Kennedy, HTML: The Definitive Guide, 2nd Edition. O'reilly 1997.
• Stephen Spainhour & Valerie Quercia, Webmaster in a Nutshell. O'reilly 1996.

  ---

  site map / main | research | tools | members | affiliates | publications | faq | search