Navigation – Plan du site

AccueilRubriquesCartographie, Imagerie, SIG2009Usability evaluation for a web-ba...


Usability evaluation for a web-based public participatory GIS: A case study in Canmore, Alberta

Évaluation de rentabilité pour GIS participatoires publics basés sur le WEB: Une étude de cas dans Canmore, Alberta
Yunliang Meng et Jacek Malczewski


Cet article traite de l’évaluation de l’utilisation et de l’ergonomie (concept de « usability », renvoyant autant à l’utilisation qu’à la notion de « user friendly ») des programmes de SIG participatifs et notamment des SIG participatifs en ligne. Précisément, nous discutons ici d’un programme testé « en réel » dans la ville de Canmore (Alberta, Canada) où les habitants étaient invités à participer en ligne au processus de décision devant mener au choix d’un emplacement pour un nouveau parking dans le centre-ville de Canmore. Le programme utilisé était ArgooMap et les données sur l’ergonomie et l’utilisation du programme ont été collectées automatiquement par le programme UsaProxy. Les résultats montrent que l’ergonomie du programme est assez différente selon le profil des utilisateurs. Elle est supérieure pour ceux qui avaient une expérience préalable des SIG, un niveau d’éducation plus élevé que la moyenne et davantage d’expérience du web. Nous pensons que ces résultats pourront être utiles à ceux qui utilisent les SIG participatifs en ligne et peuvent aider à l’amélioration de ce type de programmes.

Haut de page

Texte intégral

This research was supported by the GEOIDE Network (Project: HSS-DSS-17) of the Networks of Centers of Excellence. The authors would like to thank Gary Buxton, senior manager of the Planning and Engineering Department at the Town of Canmore, for his ongoing support in data preparation, criteria selection, and preliminary test for ArgooMap application. Also, we would like to thank Carsten Keßler, Claus Rinner, and Soheil Boroushaki who contribute greatly to ArgooMap development and customization. We acknowledge all participants in the case study for their contribution to this research project. We wish to thank anonymous reviewers for their constructive comments on an earlier version of this paper.


1Urban planning has traditionally been recognized as a centralized, bureaucratic activity carried out by planning authorities, executed top-down and led by the planning professionals (Krek, 2005). This top-down approach has been amplified by the use of computer technologies for urban planning such as Geographic Information Systems (GIS) and Spatial Decision Support Systems (SDSS). These systems have typically been restricted to planning professionals. However, diverse interests, values and objectives are inherent among relevant stakeholders in urban planning. It is thus necessary to involve all stakeholders including both planners and the general public in the crucial decision-making processes of urban planning. As a result, public participation has increasingly become an important element of urban planning.

2In the 1990s, GIS-based group decision support software has been used to assist in collaborative decision-making (e.g.: Nyerges, 1995; Jankowski et al., 1997; Craig and Elwood, 1998; Talen, 1999). One can distinguish between group decision-making by experts and group decision-making involving public participation. The latter approach has been developed into a broad research area generally referred to as Public Participatory GIS (PPGIS). The concept of PPGIS includes a variety of approaches, which aim at making GIS, relevant data, and other spatial decision-making tools available and accessible to all those with a stake in planning and decision-making (Schroeder, 1997). Nevertheless, GIS used in PPGIS projects has often been criticized as a centralized, exclusionary, expensive, and technocratic technology that needs expert users for effective and efficient operations (Dragićević, 2004).

3The rise of the Internet and the WWW technologies has created many opportunities for those involved in Geographic Information Sciences (GISci) and stimulated the integration of PPGIS into the Internet technology (e.g.: Evans et al., 1999; Kingston et al., 2000; Keßler, 2004; Zhao and Coleman, 2006). This type of systems is often referred as Web-based PPGIS (Web-PPGIS). The systems provide the local stakeholders (the general public and planners) with an opportunity to be informed and involved in the decision-making process, build local spatial knowledge, exchange ideas, and attain consensual decision-making. While the new technologies provide new opportunities for GISci researchers and planning professionals, they also create a number of challenges. The rise of Web-PPGIS makes an increasing number of lay people get access to spatial information and Web-PPGIS to visualize the information and to participate in the decision-making process. The lay people are usually composed of heterogeneous groups with different levels of knowledge and skill (Healey, 1997). Therefore, the “easiness” of using such systems for a wider range of users becomes an important consideration. This is related to a concept of “usability”. This concept involves examining the extent to which a computer technology provides support to users to achieve specified goals in an effective, efficient, and satisfactory manner (Nielsen, 1993). Even though the usability of GIS has improved immensely, users still need to have considerable technical knowledge to operate the systems (Traynor and Williams, 1997). There has been relatively little research on usability evaluation in the area of PPGIS or Web-PPGIS (Haklay and Tobón, 2003; Sidlar and Rinner, 2007; Haklay and Zafiri, 2008). One of the drawbacks of the research is the lack of a comparable and quantitative approach to PPGIS usability evaluation. Jordan (1998, p. 8) argues that “a product which is usable for one person will not necessary be usable for another”. There are a number of user characteristics (age, gender, education, similar products’ experience, etc.), which could affect how usable a system is. However, the relationships between user characteristics and system usability have not been examined in the Web-PPGIS studies.

4We suggest that empirical studies are needed to evaluate the usability of a Web-PPGIS quantitatively and to explore user characteristics as the main determinants of the system usability. This paper focuses on two objectives: (i) to demonstrate the usability of a Web-PPGIS for different users based upon a public participatory planning case study, and (ii) to examine the relationships between user characteristics defined by age, education, gender, GIS experience, web surfing experience and public participatory planning experience, and the system usability. These two research objectives will be investigated by using a Web-PPGIS: ArgooMap (Rinner, 2001; Keßler, 2004; Rinner et al., 2008; Boroushaki and Malczewski, 2009) to tackle a multicriteria site selection problem, which involves determining the “best” site for a new parkade in the downtown of Canmore, Alberta. ArgooMap is used for supporting Canmore residents to collect and manipulate GIS data and other planning related information, give their preferences to determine the best location for a proposed parkade, and exchange ideas with others. It is expected that the general public, especially those marginalized groups, are empowered by having a voice in the decision-making process.

5The paper is organized as follows. Section 2 provides a brief review of Web-PPGIS and usability evaluation research. Section 3 demonstrates how to evaluate the system usability for different users in a real-world public participatory planning situation. The results and analysis are given in Section 4. The quantitative relationships between user characteristics and the system usability are determined. Finally, discussion and conclusions are presented.

Web-PPGIS and Usability Evaluation

The Rise of Web-PPGIS

6In 1990s, GIS and group decision support software have been used to support public participatory decision-making (e.g.: Craig and Elwood, 1998; Talen, 1999). However, the GIS technology used in PPGIS projects has often been referred to as an elitist technology due to its complex user interface and high start-up costs (Curry, 1995; Pickles, 1995; Kingston et al., 2000). As a result, many groups are poorly represented in GIS (Aitken and Michel, 1995; Curry, 1995; Harris et al., 1995; Pickles, 1995; Rundstrom, 1995). Chua and Wong (2002) suggest that GIS is limited to small groups of sophisticated users. Individuals and citizens without GIS/IT capacity, access to GIS and its spatial analysis capabilities “may find it difficult to challenge official decisions convincingly” (Obermeyer, 1998, p. 3). Also, the public preferences, ideas or values are collected using traditional public participation methods (e.g.: public meetings) in some PPGIS projects (e.g.: Craig and Elwood, 1998; Hopkins et al., 2004). The methods can discourage public participation, because such meetings have particular temporal and spatial constraints so that many people can not attend. Also, many people are unwilling to speak up in front of other people in a public meeting. The result is that public meetings are often dominated by vocal individuals who may have extreme views (Kingston et al., 2000).

7The Internet and WWW technologies have great potentials to disseminate GIS and promote public participation in planning processes. In the 1990s, a small number of the PPGIS projects were available online (e.g: Sawicki and Peterman, 1998). Today these types of projects are almost routinely set up on the Internet (e.g.: Evans et al., 2004; Zhao and Coleman, 2006; Boroushaki and Malczewski, 2009; Simão et al., 2009). One of the main advantages of using a Web-PPGIS is its low cost. At the user end, the cost of the Internet service has become affordable while the access speed has greatly increased. This means an increasing number of people can use the Internet to access a Web-PPGIS at workplace, home, libraries, community centers, Internet cafés or other places at their convenience. Web-PPGIS is also extremely efficient for GIS data provision and maintenance (Chua and Wong, 2002). Once the GIS data are on the server, multiple users can view and retrieve them. Data can be easily transferred, changed and updated in a short time. The data are available 24/7 on the Internet, so users can obtain information at their convenience. Another advantage of Web-PPGIS is that it encompasses a variety of interactive technologies that enable users to conduct query, select data for retrieval, analyze data, create graphs and maps, plot routes, express their views, etc. Users can customize their data analysis and presentation that were once only performed by experts and collaborate with others (Chua and Wong, 2002). However, users still need basic technical knowledge in computing, statistics, GIS and mapping etc. to use the technologies. Web-PPGIS also overcomes many other problems brought by the traditional GIS and public participation methods (Kingston et al., 2000; Kebler, 2004). For example, the complexity of GIS and spatial analysis is hidden from the user. Consequently, the requirement for users’ technical capacity is minimized. A Web-PPGIS offers a high degree of flexibility; therefore a planning project can be easily altered or updated with more relevant information. It also enables people to make comments and express their views in a relatively anonymous and non-confrontational manner.

Usability Evaluation for Web-PPGIS

8A considerable progress has been made in developing Web-PPGIS over the last decade or so (e.g., Evans et al., 1999; Kingston et al., 2000; Keßler, 2004; Tang and Coleman, 2005). However, rapid technical advances in the area of Web-PPGIS have raised many questions regarding the evaluation perspective on Web-PPGIS. Jankowski and Nyerges (2001) argue that more conceptual and empirical research on the use of the GIS technologies is needed. Haklay and Tobón (2003) suggest that usability evaluation methods provide a sound base for the PPGIS evaluation. Usability is a study area that deals with issues such as: understanding how people use computer systems in order to develop or improve their design and matching users’ requirements so that they can carry out their tasks safely, effectively and enjoyably (Preece, 1993). Usability evaluation refers to the process of systematically collecting data on how users make use of the system for a particular task in a particular environment (Preece et al., 2002). The objectives of usability evaluation are to make computer technology accessible and easy to use for a wider range of users (Zhao and Coleman, 2007).

9There are two main types of usability evaluation methods: (i) the inspection methods, and (ii) the user testing methods (Banati et al., 2006). The inspection methods such as the heuristic evaluation, cognitive walkthrough, and walk through inspection help in detecting the usability of a software product. These methods are mainly involving system developers or experts to do the test. The user testing methods employ techniques for collecting data while the actual users utilize the product to perform representative tasks. In this case, usability can be specified in terms of users’ performance and satisfaction when interacting with the system (Butler, 1996). For example, Nielsen (1993) suggests that efficiency, learnability, errors, memorability and satisfaction are the main components of usability. ISO (1998) defines usability evaluation in terms of effectiveness, efficiency and satisfaction. Shneiderman (1998) evaluates usability by measuring speed of performance, time to learn, retention over time, rate of errors by users and satisfaction.

10Jordan (1998, p. 8) argues that “a product which is usable for one person will not necessarily be usable for another”. Maguire et al. (1998) provide a list of user characteristics that affect usability. It includes: knowledge, skill, experience, education, training (about product’s usage), physical attributes of user, habits, motor and sensory abilities. Jordan (1998) offers a similar list of user characteristics that influence usability including: experience, domain knowledge, cultural background, disability, age and gender. Thomas and Bevan (1996) suggest that experience in both the usage of product and of other products which have similar main functions should be considered. Bevan (1995) adds that experience in doing a certain task should also be considered.

11There is some usability research done in the context of PPGIS or Web-PPGIS. Haklay and Tobón (2003) demonstrate the connection between human-computer interaction (HCI) and usability evaluation. They argue that ease of use and user friendliness are characteristics of software which are more elusive than one may expect. Even if the PPGIS designers believe that they have managed to create something that is easy to use, only appropriate testing will show if the design is successful in meeting users’ needs (Haklay and Tobón, 2003). Sidlar and Rinner (2007) provide a case study involving 11 student participants concerned with planning issues on the University of Toronto (UofT) downtown campus. The analysis of this case study focuses on examining various aspects of Argumentation Maps’ usability, such as: efficiency, interactivity, connectivity, and learnability etc (Rinner, 2001; Keßler, 2004). However, the UofT case study does not show how to measure and analyze various aspects of the system usability with statistical methods, partly because the number of participants (11) is too small.

A Case Study in Canmore, Alberta

Study Area

12Canmore, Alberta (see is located in the Canadian Rocky Mountains. It is approximately 4 km east of Banff National Park and about 100 km west of the city of Calgary. The town has a population of about 12,000, and it is the administrative and commercial center for residents, employees, and employers of the Banff National Park, Kananaskis Country, and the Bow Valley. It is undergoing rapid change and growth as a result of tourism promotion and facility development. These changes contribute to parking service shortage in the downtown area. The Planning and Engineering Department of Canmore plans to build a new parkade in the downtown to provide adequate parking facilities in the area.


13ArgooMap is based on the concept of Argumentation Maps (Rinner, 2001). The concept provides the theoretical foundations for PPGIS tools that support the deliberative aspects in spatial decision-making. Rinner’s (2006) Argumentation Maps model defines argumentation elements and geographic reference objects as independent entities. The model describes the relationships between a user initiated discussion and the discussion related place on a map. The model also includes user-defined graphic reference objects (e.g., creating a new point or area). Many-to-many relationship between any kinds of objects is supported (see Figure 1). Argumentation Maps implementation was developed based on Java Applets for the user interface and a combination of PHP and a MySQL database on the server side (Keßler et al., 2005a; Keßler et al., 2005b; Rinner et al., 2008).

Figure 1: Conceptual Model for Argumentation Maps (Source: Rinner, 2006).

14The development of ArgooMap has mainly been driven by the objective to improve the usability of the tool for non-experts (Rinner et al., 2008). Accordingly, ArgooMap employs Google Maps API (Google, 2008). ArgooMap was customized by Boroushaki and Malczewski (2009). The main map section of the system contains two elements: “tutorial” and “main decision map” (see The “tutorial” component describes the goal and objectives of the parkade site selection problem and provides a detailed explanation of the properties and geospatial characteristics of candidate sites along with their photos. It also contains the explanation of terminologies, evaluation criteria and their units of measurement. In addition, the “tutorial” provides a step by step walkthrough on how to join the online public participatory decision-making process by selecting the preferred location or participating in the online discussion, debates and communication with other users.

15The “main decision map” consists of a multicriteria decision analysis (MCDA) module (Malczewski, 1999). Users can input their preferences regarding the relative importance of each criterion using a set of linguistic terms. The set of six linguistic terms include: none, very low, low, medium, high and very high importance (see Boroushaki and Malczewski, 2009). In addition, users should choose a linguistic quantifier (such as “at least one”, “most”, “half”, “all”) to define how many of the evaluation criteria ought to be satisfied by an acceptable location. Given the input information, a MCDA procedure generates the final score for each alternative. Within the “main decision map” element, users can switch to the “group decision map” component. The “group decision map” shows the ranking of the decision alternatives based on the majority’s preferences. In the “main decision map”, users can explore the study area using functions (e.g., zoom in/out) or change the map view background map to satellite image or map-satellite image hybrid view module. The layer of alternative sites for locating a new parkade can be loaded and removed. The attributes of each alternative site can be retrieved by clicking the site (see


Data Used for the Parkade Location Problem

16The dataset for the parkade location decision-making can be categorized into two broad categories: (i) data related to the demand for parking service, and (ii) data on alternative sites and evaluation criteria. The demand for parking service in the downtown of Canmore comes from two groups of people: local residents and tourists. Tourists can be further categorized into day-visit tourists and stay tourists based on whether they stay overnight in Canmore. The day-visit tourists usually enter into the community by Benchlands Trail Overpass on Highway No. 1, stop in Canmore for a meal, gas or other short time activities, and then head to other places without staying overnight. The stay tourists spend at least one night in Canmore. The number of hotel units is used to describe their demand. Private vehicles are the main transportation mode for residents in Canmore. Canmore vehicle ownership statistics are the best data to describe the demand from local residents for parking service. However, the local government does not collect relevant data. Therefore, this study employs the 2006 population statistics (Canmore Census, 2006) as an approximate measure of the demand. The Local Delivery Units (LDU) (the smallest postal delivery zones, see Figure 2) are used for identifying the distribution of population and stay tourists. The centroïds of LDU areas and location of Benchlands Trail Overpass are used as demand points. Canmore consists of 410 LDUs.

Figure 2 : Study Area

17The Planning and Engineering Department of Canmore has identified four alternative locations for a new parkade in the downtown area (see Figure 2). The suitability of the candidate sites is evaluated on the basis of two objectives: (i) accessibility to local residents, and (ii) accessibility to stay tourists. The concept of accessibility can be operationalized in terms of the average distance and maximum distance (e.g., McAllister, 1976; Morrill and Symons, 1977; Hodgart, 1978). In this study, the average and maximum distance is weighted using population and the number of hotel units to represent different demand levels from local residents and stay tourists in each zone. Therefore, the objectives are measured by four attributes: (1) weighted average distance to local residents, (2) weighted maximum distance to local residents, (3) weighted average distance to stay tourists, and (4) weighted maximum distance to stay tourists. Most day-visit tourists enter Canmore by Benchlands Trail Overpass on Highway No.1 and then drive into the downtown area. If the future parkade is located closer to the Benchlands Trail Overpass, it will possibly decrease the number of day-visit tourists’ vehicles traveling on the downtown streets. It will also provide easy access to Highway No. 1. Hence, (5) the distance to Benchlands Trail Overpass is used to consider the demand from day-visit tourists. The parkade is targeting users who park their cars in the downtown and look for various services. Main Street is the community’s main shopping area with various amenities. Therefore, (6) the distance to Main Street is used to describe the suitability of a candidate site. All distances were measured using the road network-based distance between the centroïds of the LDUs (the middle of Benchlands Trail Overpass, or the center of Main Street) and the location of the candidate sites. The parkade will generate some negative impacts (i.e., noise, traffic, poor air quality etc.) for surrounding residents in the future; consequently, (7) the number of people living within 100m of a candidate site is used to measure the suitability of the candidate site. In addition, (8) the size of a candidate site, and (9) cost of land acquisition are used to evaluate the suitability of the candidate site. Except the size of a candidate site, all attributes are to be minimized.

18Both of the data used for parkade location decision-making and ArgooMap system were uploaded at for use by the general public between September 1st and December 30th, 2008. Local residents in Canmore Alberta were invited to identify their concerns, ideas, suggestions and/or preferences over the candidate sites and evaluation criteria for locating a new parkade in the downtown of Canmore. The website and the proposed parkade project were advertised on local community newspaper – Rocky Mountain Outlook. The advertisement was also posted on the websites of the Department of Local Economic Development and Department of Planning and Engineering.

Data Used for Evaluating the ArgooMap System

19For testing the usability of ArgooMap (Rinner, 2001; Keßler, 2004; Rinner et al., 2008; Boroushaki and Malczewski, 2009), the users’ every move on the website was recorded with UsaProxy (Atterer et al., 2007). The software is based on an HTTP proxy approach working with existing server and browser setups. UsaProxy makes it possible to obtain detailed and useful information about the actual usage of the website holding ArgooMap. Actions such as moving the mouse pointer around, scrolling a page, or fill out a form in a specific order were all recorded. Events such as opening a website, logging in the system and clicking a button are stored as well. Such information is highly valuable for evaluating the system usability. The tracked actions and events together with the ordinary HTTP communication were directly logged on the server with the UsaProxy software.

20Figure 3 shows an example from the log data that the UsaProxy produced during the public participatory decision-making process. The events in the figure include mouse move (pointer position changed), mouse over (the pointer was moved over a HTML element or similar), focus (the cursor moved into an input field) and others. The events are recorded together with users’ IP, the time of the events, and coordinates of mouse pointer.

21The users are asked to fill out a questionnaire at the end of the online participation period. The questionnaire asks for the user’s socio-economic and demographic status such as: age, gender (male or female), education (ordinal scale from 1 to 5, where 1 represents high school diploma, 2 represents college diploma, 3 represents university degree, 4 represents Master degree and 5 represents Ph.D. degree). Questions about their GIS experience, web surfing experience, public participatory planning experience and satisfaction of using ArgooMap are also included in the questionnaire. Previous GIS experience was collected as binary data where 1 represents that users have used GIS software and 0 represents that they do not have. Public participatory planning experience was also collected as binary data where 1 represents that users have joined the public participatory planning before and 0 represents that they do not have. Finally, data on web surfing experience were collected by asking users how many hours they spend weekly on the Internet.

Figure 3: A Sample of the Log Output Produced by UsaProxy.

Generating Usability Metrics

22There is no general rule for how usability measures should be chosen or combined (ISO, 1998). The choice of measures and the level of details of each measure are dependent on the objectives of the parties involved in the measurement. ArgooMap is designed for “walk up and use”. This means that the system is intended to be used by first-time users who need to be able to effectively use the system without any training; for example, in public information kiosks, museum displays, and ticket-purchasing systems. Therefore, metrics employed for evaluating the usability of ArgooMap are a combination of measures from previous research (Nielsen, 1993; Haklay and Tobón, 2003; Sidlar and Rinner, 2007) and “walk up and use” measures (ISO, 1998). The measures of usability include: effectiveness, learnability, efficiency and satisfaction.

23Effectiveness refers to “the accuracy and completeness with which users can achieve their goals” (ISO, 1998, p. 19). In this study, it is measured by the number of power tasks completed successfully on first attempt (ISO, 1998). In the online public participatory decision-making process, users are expected to perform three tasks. The first task is to read comments or suggestions from previous users. The second task is to perform a MCDA for resolving the site selection problem by giving their preferences on the evaluation criteria. The third task is to initiate (or reply to) a map-based discussion to suggest other land parcels as candidate sites or express their concerns related to the parkade project. The second metric used to measure effectiveness in this project is the number of functions used on first attempt (ISO, 1998). ArgooMap system supplies a number of functions that help users to explore GIS map/image and candidate sites. Functions include: zoom (zoom in and zoom out), background view change (map view, satellite image view, and map-satellite hybrid view), group decision-making outcomes inquiry and site attribute inquiry.

24Efficiency refers to the system’s ability to fulfill its functions and objectives while taking a minimal amount of resources, albeit time or hardware (ISO, 1998; Sidler and Rinner, 2007). It is measured by the time to perform a pre-specified task on first attempt. As mentioned, users were suggested to perform three power tasks. In terms of initiating or replying a comment, the length of comments posted by the users and their typing speed are quite different. Also, only a limited number of users posted comments. Hence, the time used to post a comment on first attempt can not be used to measure efficiency. The time used to read comments posted by other users depends on available comments and the length of comments. In addition, only a limited number of users read comments from others. Every user performed the MCDA (see Section 3.2) and the time spent on the task is comparable. Therefore, the time to perform a pre-specified task (that is, the MCDA) on first attempt is chosen to measure the efficiency in this study.

25Learnability focuses on “how easy it is to understand and recognize the usefulness of the prototype or tools in the prototype” (Sidler and Rinner, 2007, p. 6). It is quantified by the time to learn how to use the system to participate. In this project, no immediate assistance was provided via phone or the Internet. Users have to go through the second part of the “tutorial” page to learn by themselves if they do not know how to use the system to participate. Therefore, the time spent on reading the instructions (about how to use the system to participate) is used to represent learnability.

26Satisfaction is defined as “freedom from discomfort and positive attitudes to the use of the product” (ISO, 1998, p. 20). It is a response of users when interacting with the product. In the questionnaire section, the users were asked to rate their overall experience with ArgooMap on a six point scale ranging from 0 to 5 (0 is the lowest score and 5 is the highest score).

Results and Analysis

Descriptive Statistics

27Table 1 shows descriptive statistics for user characteristics: gender, GIS experience, and public participatory planning experience, and Table 2 gives descriptive statistics for age, education, and web surfing experience. In this research, 58 people joined the online public participatory decision-making process; 47 of them filled up the questionnaire at the end of the online participation period. Among those users who filled up the questionnaire, 25 are males and 22 are females, with ages ranging from 21 to 67 (Table 2). 16 users have used GIS software, and 33 users have participated in public participatory planning projects. All of the users have at least college degree (Table 2), and they spend very different amount of time on web surfing weekly.


GIS experience

Public participatory planning experience

The number of male users: 25

The number of users who have GIS experience: 16

The number of users who have public participatory planning experience: 33

The number of female users: 22

The number of users who do not have GIS experience: 31

The number of users who do not have public participatory planning experience: 14

Table 1: Descriptive Statistics by Gender, GIS experience, and Public Participatory Planning Experience.




Web surfing experience





Std. deviation












Table 2: Descriptive Statistics by Age, Education, and Web Surfing Experience.

28Descriptive statistics for the five usability metrics are presented in Table 3. The time to perform a pre-specified task on first attempt varies from 26 seconds to 375 seconds. In addition, most of users finished the task within 150 seconds. The time spent on reading the instructions varies from 0 seconds to 395 seconds, and the majority of the users spent less than 100 seconds to read the instructions about how to use the system to participate. In terms of the number of functions used on first attempt, most of the users employed at least one function provided in this project. When it comes to the number of power tasks completed successfully on first attempt, more than half of the users only performed one task (that is, MCDA). The subjective satisfaction of using ArgooMap is fairly high. The majority of the users gave a rate higher than 2.





Std. Deviation

The time to perform a task on first attempt





The time spent on reading the instructions





The number of functions used on first attempt





The number of power tasks completed successfully on first attempt










Table 3: Descriptive Statistics for the Five Usability Metrics.

Examining the Relationships between User Characteristics and Usability Metrics

29For the analysis of the impacts of user characteristics on the system usability, the Mann-Whitney U test (Mann and Whitney, 1947) at a 0.05 significance level was applied to compare the usability metrics regarding users’ gender, GIS experience, and public participatory planning experience. This non-parametric test was selected as an appropriate tool because the characteristics are defined as binary data and the distributions of the usability metrics do not follow the normal distributions. A summary of the Mann-Whitney U test results is presented in Table 4. The Spearman’s rank correlation (Spearman, 1904) tests were conducted to assess the relationships between the usability metrics and the levels of users’ education, users’ age, and their web surfing experience. The results are shown in Table 5.

Table 4: A Summary of Results from Comparing the Usability Metrics Regarding User Characteristics with Mann-Whitney U Test.

Table 4: A Summary of Results from Comparing the Usability Metrics Regarding User Characteristics with Mann-Whitney U Test.

Table 5: A Summary of Correlations between the Usability Metrics and User Characteristics with Spearman’s Rank Correlation Test.

Note: ** Correlation is significant at the 0.01 level
* Correlation is significant at the 0.05 level


30The Mann-Whitney U test suggests a significant difference in the number of functions used on first attempt with regards to the gender difference. Males use significantly more functions than females on first attempt (Table 4). However, significant differences have not been found for other usability metrics regarding the gender difference. Both the Internet and GIS have been male dominant fields since their beginnings (King et al., 1997; Gilbert and Masucci, 2004). Some studies (Sherman et al., 1999; Gilbert and Masucci, 2004) indicate that the gender gap in the Internet and GIS usage has narrowed in recent years but has not closed entirely. Our results support this conclusion by demonstrating that gender does not have significant impacts on most aspects of the system usability.

GIS Experience

31Our analysis of the data does show significant differences for all usability metrics with regards to the GIS experience difference. For the effectiveness and satisfaction metrics, the means of users who have GIS experience are significantly higher than those who do not (Table 4). For the efficiency and learnability metrics, the means of users who have GIS experience are significantly lower than those who do not. In other words, the system usability is significantly higher for users who have GIS experience than those who do not. These findings are consistent with research by Jordan (1998, p. 8) suggesting that “experience with other similar products will affect how usable a product is for a user”.

Public Participatory Planning Experience

32The Mann-Whitney U test indicates that there is a significant difference for the number of functions used on first attempt with regards to public participatory planning experience difference. For the number of functions used on first attempt, the mean value of users who have public participatory planning experience is significantly higher than those who do not (Table 4). Users who have the experience employ significantly more functions on first attempt than those who do not. However, comparing the means of other usability metrics regarding the public participatory planning experience difference, significant differences have not been found. This suggests that previous public participatory planning experience does not have significant impacts on the system usability in general. Since this was the first time that the Department of Planning and Engineering in Canmore used a Web-PPGIS to support public participatory planning, this innovation made previous public participatory planning experience (public meetings, poster presentation, etc.) unhelpful to improve performance and satisfaction in the course of using ArgooMap for resolving the site selection problem.


33According to the Spearman’s rank correlation coefficients, there is a significant correlation between the learnability metric and users’ age (Table 5). The older the user, the more time he/she spends on reading the instructions about how to use the system to participate. There are, however, insignificant correlations between age and other usability metrics. These findings are inconsistent with previous research conclusion, which points out that “older people may be less accepting of computer-based products and could be deterred from using them” (Jordan, 1998, p. 11), since different generations have grown up with different types of technology. While younger generations are more likely to have had exposure to new technologies (computers, IT, GIS, etc.), older generations are left behind. Our results show that age is not a major factor that would impede senior users to use ArgooMap for resolving the site selection problem.


34The Spearman’s rank correlation test shows that there are significant correlations between all usability metrics and the levels of users’ education (Table 5). The levels of users’ education are positively correlated with the number of functions used on first attempt, the number of power tasks completed successfully on first attempt, and the level of satisfaction. It is negatively correlated with the time it takes to perform a task on first attempt and the time spent on reading the instructions. In other words, the system usability is significantly higher for users who have high education degrees. These findings show that the design of ArgooMap has not passed the technical barrier that impedes users with lower levels of education to interact with the system.

Web Surfing Experience

35The Spearman’s rank correlation test shows significant relationships between all usability metrics except the number of function used on first attempt and users’ web surfing experience (Table 5). The more time people spend on web surfing, the larger the number of power tasks completed successfully on first attempt, the less time it took to perform a task on first attempt and read the instructions, and the higher the level of satisfaction. Web surfing experience does not have a significant relationship with the number of function used on first attempt, but it still has significant impacts on most aspects of the system usability. The Google map interface, discussion forum, simple GIS functions (zoom-in, zoom-out, pan, etc.) are widely used in various web applications nowadays. People with more web surfing experience are expected to be familiar with the elements of the website. Accordingly, our findings indicate that ArgooMap system usability is higher for users with more web surfing experience.

Discussion and Conclusions

36This paper has demonstrated a new way of performing usability test for a Web-PPGIS – ArgooMap. UsaProxy (Atterer et al., 2007) was used for documenting how people use ArgooMap for resolving a real world planning problem. This kind of tools can be easily deployed and used for various Web-PPGIS applications. Usability tests performed in such a way successfully resolve a conflict: software usability is mainly tested in lab environment and ArgooMap is developed to facilitate public users to access it from a variety places. In addition, the test is time-efficient and cost-effective.

37This research has also demonstrated a framework to evaluate the system usability for different users quantitatively. Despite the fact that the ArgooMap interface has limited capabilities compared to a standard GIS interface, one could detect significant differences in users’ performance and satisfaction. The system usability has been analyzed in conjunction with user characteristics. As shown in this study, the usability of ArgooMap system depends significantly on the participants’ previous GIS and web surfing experience. The results indicate that the experience with other GIS products or web-based GIS elements has positive effects on the system usability. The finding provides important clues for designers of a new Web-PPGIS. On the one hand, there are many commonly used commercial GIS (e.g.: ArcGIS, MapInfo, IDRISI etc.) and well-recognized web-based GIS interfaces available (e.g.: Google Map, ArcIMS, MapQuest etc.). Hence, introducing a completely new Web-PPGIS or radical changes in system design does not necessary improve the product usability from users’ perspective, because “the inherent usability benefits of compatible with other products may be lost” (Jordan, 1998b, p. 9). The importance of the consistency with existing systems is noticed by Rinner et al. (2008) who adopt Google Maps API (Google, 2008) instead of Java Applet for the new version of ArgooMap. On the other hand, future Web-PPGIS projects would adopt diverse web and GIS technologies using various interfaces, because the technologies will continue to evolve in the foreseeable future. It is very challenging in terms of how to integrate the new technologies to a Web-PPGIS and design the system interface in a user acceptable way. Web-PPGIS developers have to trade-off between the introduction of the new technologies and/or interfaces and consistency with existing products.

38As we mentioned earlier, every participant in this study have at least college degree. However, based on the data from Statistics Canada (2006), 72.75% of Canmore residents have at least college degree or equivalent. The absence of the participants with the lowest level of education in this study indicates that while the application of Web-PPGIS creates a new opportunity for engaging average people in the decision-making process, it could bring about new risks of excluding some citizen groups. We suggest that Web-PPGIS practitioners should ensure the inclusion of each marginalized group in the decision-making process. Possible solutions include sending advertisements to a certain group of disadvantage people (low education, low income, etc), supplying computers and Internet access at libraries and community centers to attract people who do not have them at home, and providing face-to-face helps to the people lacking basic computer and GIS skills, etc. However, some of the suggestions may undermine the efforts of Web-PPGIS to ease the spatial and temporal constrains brought by conventional participation methods. In addition, we found that the levels of users’ education have significant effects on the system usability. Poor usability of a Web-PPGIS would cause issues such as the waste of user’s time, worry and frustration, and eventually could discourage people with low levels of education to engage in the decision-making process. As a result, those people may not be convinced that the new technologies are “better” than conventional methods and this can lead to a further division with respect to public participation using Web-PPGIS in the future.

39Despite the optimistic claims that Web-PPGIS becomes more prevalent and readily used, Web-PPGIS practitioners may encounter similar usability issues emerged in this study. There is simply no list of absolute rules that the Web-PPGIS developers can follow to make the system usable for every user, but one can minimize common usability mistakes by following existing design guidelines or principles (e.g.: Nielsen and Tahir, 2001; Pearrow, 2006; IBM, 2009). To address the issues, we also suggest that usability testing approaches should be integrated into the system design process, so Web-PPGIS designers can detect the usability problems and make changes before the public participatory planning process. Furthermore, traditional public participation methods (public meetings and poster presentations etc.) should be used as a complementary solution for supporting people who have immense difficulties when interacting with a Web-PPGIS.

40In this project, we found that it is extremely difficult to inform every resident in Canmore of the parkade project and the Web-PPGIS website because of the limited advertising budget, time press, and strict policy regarding access to personal information. Consequently, they were advertised on the local community newspaper and governmental websites. The advertising methods actually limit the participants from all Canmore residents to the newspaper readers, people who regularly browse governmental websites, and possibly some people connected with them. It should be noted that the bias brought by the advertisement methods is a limitation of this research. However, it is a very challenging task to eliminate the bias because every major media has its own targeting customers. The advertising constraints we faced in this research are common in the Web-PPGIS applications. Therefore, there is a need for developing innovative methods to advertise the Web-PPGIS website and to analysis the extent to which the advertising methods affect public participation.

Haut de page


Aitken S.C. and Michel S., 1995, “Who Contrives the ‘Real’ in GIS? Geographic Information, Planning and Critical Theory”, Cartography and Geographic Information Systems, Vol.22, No.1, 17-29.

Atterer R., Schmidt A. and Wnuk M., 2007, “A Proxy-based Infrastructure for Web Application Sharing and Remote Collaboration on Web Pages”, Proceedings of the 11th International Conference on Human-Computer Interaction, Rio de Janeiro, Brazil, September 2007, 74-87, (last visited April 04, 2009).

Banati H., Bedi P. and Grover P.S., 2006, “Evaluating Web Usability from the User’s Perspective”, Journal of Computer Science, Vol.2, No.4, 314-317.

Bevan N., 1995, “Measuring Usability as Quality of Use”, Journal of Software Quality, Vol.4, No.2, 115-130.

Boroushaki S. and Malczewski J., 2009, “ A WebGIS-based Collaborative Multicriteria Decision Analysis”, Submitted to URISA Journal, (last visited April 29, 2009).

Butler K.A., 1996, “Usability Engineering Turns 10”, Interactions, Vol.3, No.1, 59-75.

Canmore Census, 2006, “Population Frequency Table 2006”, Town of Canmore, (last visited April 23, 2009)

Chua Y.L. and Wong S., 2002, “Data Intermediation and Beyond: How the Web Modifies the Dissemination of GIS Information”, Proceedings of the 2nd Public Participation GIS Conference, New Brunswick, U.S.A, July 2002, (last visited April 29, 2009).

Craig W.J. and Elwood S.A., 1998, “How and Why Community Groups Use Maps and Geographic Information”, Cartography and Geographic Information Systems, Vol.25, No.2, 95-104.

Curry M., 1995, “Rethinking Rights and Responsibilities in Geographic Information Systems: Beyond the Power of the Image”, Cartography and Geographic Information Systems, Vol.22, No.1, 58-69.

Dragićević S., 2004, “The Potential of Web-based GIS”, Journal of Geographical Systems, Vol.6, No.2, 79-81.

Evans A., Kingston R., Carver S. and Turton I., 1999, “Web-based GIS Used to Enhance Public Democratic Involvement”, Proceedings of the 4th International Conference on GeoComputation, Fredericksburg, United States, July 1999, (last visited April 20, 2009).

Evans A., Kingston R. and Carver S., 2004, “Democratic Input into the Nuclear Waste Disposal Problem: The Influence of Geographical Data on Decision-making Examined through a Web-based GIS”, Journal of Geographical Systems, Vol.6, No.2, 117-132.

Gilbert M.R. and Masucci M., 2004, “Moving beyond ‘Gender and GIS’ to a Feminist Perspectives on Information Technologies: The Impact of Welfare Reform on Women’s IT Needs”, in Nelson L. and Seager J. (eds.), A Companion to Feminist Geography, Malden, Blackwell, 305-321.

Google, 2008, “Google Maps API”, (last visited April 14, 2009).

Haklay M. and Tobón C., 2003, “Usability Evaluation and PPGIS: Towards a User-centred Design Approach”, International Journal of Geographical Information Science, Vol.17, No.6, 577-592.

Haklay M. and Zafiri A., 2008, “Usability Engineering for GIS: Learning from a Screenshot”, The Cartographic Journal, Vol.45, No.2, 87-97.

Harris T.M., Weiner D., Warner T. and Levin R., 1995, “Pursuing Social Goals through Participatory GIS: Redressing South Africa’s Historical Political Ecology”, in Pickles J. (eds.), Ground Truth, New York, Guilford, 196-222.

Healey P., 1997, Collaborative Planning: Shaping Places in Fragmented Societies, London, MacMillan.

Hodgart R.L., 1978, “Optimizing Access to Public Services”, Progress in Human Geography, Vol.2, No.1, 17-48.

Hopkins L.D., Twidale M. and Pallathucheril V.G., 2004, “Interface Devices and Public Participation”, Proceedings of the 3rd Annual PPGIS Conference of Urban and Regional Information Systems Association, Madison, United States, 71-83.

IBM (International Businness Machines), 2009, Web Design Guidelines, (last visited September 09, 2009).

ISO (International Organization for Standardization), 1998, Ergonomic Requirements for Office Work with Visual Display Terminals -- Part 11: Guidance on Usability, ISO.

Jankowski P., Nyerges T., Smith A., Moore T.J. and Horvath E., 1997, “Spatial Group Choice: A SDSS Tool for Collaborative Spatial Decision-making”, International Journal of Geographical Information Systems, Vol.11, No.6, 577-602.

Jankowski P. and Nyerges T., 2001, “GIS-supported Collaborative Decision-making: Results of an Experiment”, Annals of the Association of American Geographers, Vol.91, No.1, 48-70.

Jordan P.W., 1998, An Introduction to Usability, Boca Raton, CRC.

Keßler C., 2004, “Design and Implementation of Argumentation Maps”, Diploma Thesis, Westfälische Wilhelms Universitäty, Münster, Germany, (last visited April 17, 2009).

Keßler C., Rinner C. and Raubal M., 2005a, “An Argumentation Map Prototype to Support Decision-Making in Spatial Planning”, in Toppen F. and Painho M. (eds.), Proceedings of the 8th Conference on Geographic Information Science, Estoril, Portugal, May 2005, 135-142, (last visited April 19, 2009).

Keßler C., Wilde M. and Raubal M., 2005b, “Using SDI-based Public Participation for Conflict Resolution”, Proceedings of the 11th EC-GI and GIS Workshop, Sardinia, Italy, June 2005, (last visited April 09, 2009).

King J.L., Grinter R.E. and Pickering J.M., 1997, “The Rise and Fall of Netville: The Saga of a Cyberspace Construction Boomtown in the Great Divide”, in Kiesler S. and Mahwah N.J., Culture of the Internet, Mahwah, Lawrence Earlbaum, 3-33.

Kingston R., Carver S., Evans A. and Turton I., 2000, “Web-based Public Participation Geographical Information Systems: An Aid to Local Environmental Decision-making”, Computers, Environment and Urban Systems, Vol.24, No.2, 109-125.

Krek A., 2005, “Rational Ignorance of the Citizens in Public Participatory Planning”, Proceedings of the 10th Symposium on Information and Communication Technologies in Urban Planning and Spatial Development and Impacts of ICT on Physical Space, Vienna, Austria, February 2005,
(last visited April 26, 2009)

Maguire M., Kirakowski J. and Vereker N., 1998, Respect: User-Centred Requirements Handbook, Loughborough, HUSAT Research Centre.

Malczewski J., 1999, GIS and Multicriteria Decision Analysis, New York, Wiley.

Mann H.B. and Whitney D.R., 1947, “On a Test of Whether One of Two Random Variables is Stochastically Larger than the Other”, Annals of Mathematical Statistics, Vol.18, No.1, 50-60.

McAllister D.M., 1976, “Equity and Efficiency in Public Facility Location”, Geographical Analysis, Vol. 8, 47-63.

Morrill R. and Symons S., 1977, “Efficiency and Equity Aspects of Optimum Location”, Geographical Analysis, Vol. 9, 215-225.

Nielsen J., 1993, Usability Engineering, Boston, Academic.

Nielsen J. and Tahir M., 2001, Homepage Usability: 50 Websites Deconstructed, Indianapolis, New Riders Press.

Nyerges T., 1995, “Design Considerations for a Group-based GIS: Transportation Improvement Project Decision-making as an Example”, Proceedings of the 8hth Annual Symposium on Geographic Information Systems for Transportation, Sparks, U.S.A, June 1995, 261-282.

Obermeyer N.J., 1998, “PPGIS: The Evolution of Public Participation GIS”, Unpublished UCGIS white paper.

Pearrow M., 2006, Web Site Usability Handbook, Rockland, Charles River Media.

Pickles J., 1995, “Representations in an Electronic Age: Geography, GIS, and Democracy”, in Pickles J. (eds.), Ground Truth, New York, Guilford, 1-30.

Preece J., 1993, A Guide to Usability: Human Factors in Computing, Reading, Boston, Addison-Wesley.

Preece J., Yvonne R. and Helen S., 2002, Interaction Design: Beyond Human-computer Interaction, New York, John Wiley and Sons.

Rinner C., 1999, “Argumentation Maps -- GIS-Based Discussion Support for Online Planning”, Ph.D. Thesis, GMD Research Series Nr. 22/1999, University of Bonn, Sankt Augustin, Germany,
last visited March 28, 2009).

Rinner C., 2001, “Argumentation Maps – GIS-based Discussion Support for Online Planning”, Environment and Planning B: Planning and Design, Vol.28, No.6, 847-863.

Rinner C., 2006, “Argumentation Mapping in Collaborative Spatial Decision-making”, in Dragicevic S. and Balram S. (eds.), Collaborative GIS, Hershey, Idea Group Publishing, 85-102.

Rinner C., Keßler C. and Andrulis S., 2008, “The Use of Web 2.0 Concepts to Support Deliberation in Spatial Decision-Making”, Computers, Environment and Urban Systems, Vol.32, No.5, 386-395.

Rundstrom R.A., 1995, “GIS, Indigenous Peoples, and Epistemological Diversity”, Cartography and Geographic Information Systems, Vol.22, No.1, 45-57.

Sawicki D. and Peterman R., 1998, “Understanding the Depth and Breadth of PPGIS Supply”, Proceedings of PPGIS Specialist Meeting, Santa Barbara, U.S.A, October 1998, (last visited March 26, 2009).

Schroeder P., 1997, “GIS in Public Participation Settings”, Proceedings of University Consortium for Geographic Information Science, Annual Assembly and Summer Retreat, Bar Harbor, United States, June 1997.

Sherman R.C., End C., Kraan E., Cole A., Campbell J., Birchmeier Z. and Cohara C., 1999, “The Internet Gender Gap: Is It Narrowing or Widening?” Proceedings of the 107th Annual Convention of the American Psychological Association, Boston, U.S.A, August 1999, (last visited April 06, 2009)

Shneiderman B., 1998, Designing the User Interface, Boston, Addison-Wesley Longman.

Sidlar C. and Rinner C., 2007, “Analyzing the Usability of an Argumentation Map as a Participatory Spatial Decision Support Tool”, URISA Journal, Vol.19, No.1, 47-55.

Simão A., Densham P.J. and Haklay M., 2009, “Web-based GIS for Collaborative Planning and Public Participation: An Application to the Strategic Planning of Wind Farm Sites”, Journal of Environmental Management, Vol.90, No.6, 2027-2040.

Spearman C., 1904, “The Proof and Measurement of Association between Two Things”, The American Journal of Psychology, Vol.15, 72-101.

Statistics Canada, 2006, Education Highlight Tables,
(last visited September 03, 2009).

Talen E., 1999, “Constructing Neighbourhoods from the Bottom up: The Case for Resident-generated GIS”, Environment and Planning B: Planning and Design, Vol.26, No.4, 533-554.

Tang M.Y. and Coleman D.J., 2005, “Design of a GIS-enabled Online Discussion Forum for Participatory Community Planning”, Proceedings of the 98th Annual Conference of the Canadian Institute of Geomatics, Ottawa, Canada, June 2005.

Thomas C. and Bevan N., 1996, Usability Context Analysis: A Practical Guide, Teddington, Serco Usability Services.

Traynor C. and Williams M.G., 1997, “A Study of End-user Programming for Geographic Information Systems”, Proceedings of the 7th Workshop on Empirical Studies of Programmers, Alexandria, United States, 140-156.

Zhao J. and Coleman D.J., 2006, “GeoDF: Towards a SDI-based PPGIS Application for E-Governance”, Proceedings of the GSDI 9 Conference, Santiago, Chile, November 2006, (last visited February 06, 2009).

Zhao J. and Coleman D. J., 2007, “An Empirical Assessment of a Web-based PPGIS Prototype”, Proceedings of the 45th Annual Conference of the Urban and Regional Information Systems Association, Washington, United States, August 2007, (last visited April 03, 2009).

Haut de page

Table des illustrations

Fichier image/png, 49k
Fichier image/jpeg, 472k
Fichier image/png, 697k
Fichier image/png, 278k
Titre Table 4: A Summary of Results from Comparing the Usability Metrics Regarding User Characteristics with Mann-Whitney U Test.
Fichier image/png, 62k
Haut de page

Pour citer cet article

Référence électronique

Yunliang Meng et Jacek Malczewski, « Usability evaluation for a web-based public participatory GIS: A case study in Canmore, Alberta », Cybergeo: European Journal of Geography [En ligne], Cartographie, Imagerie, SIG, document 483, mis en ligne le 17 décembre 2009, consulté le 04 juillet 2022. URL : ; DOI :

Haut de page


Yunliang Meng

Ph.D. Candidate, Department of Geography, The University of Western Ontario, London, Ontario, Canada

Articles du même auteur

Jacek Malczewski

Professor, Department of Geography, The University of Western Ontario, London, Ontario, Canada

Articles du même auteur

Haut de page

Droits d’auteur

Licence Creative Commons
La revue Cybergeo est mise à disposition selon les termes de la Licence Creative Commons Attribution 4.0 International.

Haut de page
Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search