1Rapidly changing social environments often require public sector organizations to adapt the way they produce public services, thus creating new ways of addressing policy problems. Learning is a fundamental component of the adaptation process. Yet, public sector organizations are often not designed to support intentional policy learning. Policy innovation labs (PILs) have emerged in recent years as venues that can enhance the capacity for learning how to solve public policy problems and achieve policy objectives (Wellstead et al., 2021; Kim et al., 2023). Among the diverse types of PILs, ‘data-based’ PILs address the growing interest in big data and advanced technologies to address public policy issues (Lee & Ma, 2020; Wellstead et al., 2022). New digital technologies and advanced data analytics programs are being employed to manage administrative data (McGann et al., 2018). Specifically, these refer to machine learning, artificial intelligence, and advanced algorithmic methods that facilitate data processing.
2While many types of PILs have expertise in developing innovative public services and products, data-based PILs support the production of knowledge and ideas for improving public policies and services through a specific focus on the use of data and technology. By emphasizing the importance of data and new technologies, data-based PILs theoretically provide a critical case for studying policy learning. Policy learning processes require mechanisms, such as formalized venues and strategic processes, for acquiring, translating, and disseminating data and information across decision-making actors (Heikkila & Gerlak, 2013), which data-based PILs are arguably designed to do. Additionally, data-based PILs and government agencies collaborate by collecting and sharing data, providing information, and producing public programs and services. As such, data-based PILs can extend learning opportunities beyond their internal boundaries. Despite their potential contributions to policy learning, few studies have examined whether and how data-based PILs contribute to and support policy learning.
3To our knowledge, this paper is the first to investigate the role of policy learning in data-based PILs. Specifically, we investigate how learning occurs internally with labs, and externally from other agencies or policy actors. After introducing concepts of policy learning, learning processes, and learning products, we discuss how policy learning occurs in PILs. We then present the data and methods employed, which are grounded in an exploratory study design and include key informant interviews with PIL managers and staff, followed by a discussion of the findings. Our findings show that data-based PILs engage in learning processes and contribute to the sharing of information and knowledge across a range of organizations.
4Learning and knowledge are central to policy processes. This was first noted by Helco (1974), who built on Lasswell’s (1965) idea of “enlightenment,” which is associated with the development, communication, and use of knowledge in policy processes. These knowledge activities are associated with learning, mutual understanding, and decision-making around social problems (Ostrom, 1997).
5Policy learning occurs through individual policy actors and can be translated through organizations or institutions to advance understanding of policy problems, their environments, and policy objectives (Sabatier, 1988; Heikkila & Gerlak, 2013). The growth of new knowledge through policy learning creates possibilities for policy actors to collectively anticipate and respond to future events (Ostrom, 1997). It can therefore be the trigger or source of policy change (Bennett & Howlett, 1992). Such change is more likely when policy actors update their policy beliefs, ideas, and information (Sabatier, 1987). In addition to policy change, learning can potentially lead to more efficient, legitimate, and democratic policy processes and outcomes.
6Continuous learning practices enable policy actors to maintain and improve their knowledge of policy issues and capabilities (Ostrom, 1997; Dunlop, 2017). However, a single policy actor has limited capacity to accumulate knowledge alone for effectively influencing policy outcomes. Accumulating knowledge and skills across multiple policy actors requires processes and commensurate efforts to transmit this knowledge and skills. Therefore, policy learning requires considerable deliberate effort to communicate with others, to share and absorb knowledge with others, and a willingness to learn (Bennett & Howlett 1992; Ostrom, 1997).
7How and whether policy learning occurs in policy and governance processes has been studied through diverse frameworks and lenses (see Bennett & Howlett, 1992; Dunlop, 2017; Gerlak & Heikkila, 2011). Across this literature, policy learning can be classified as learning processes, learning products, or analytical work (Dunlop, 2017). Learning products emerge from and are linked to the learning process (Heikkila & Gerlak, 2013). In addition, scholars have examined the analytical levels of learning at the micro, meso, and macro levels (see Dunlop, 2017; Moyson et al., 2017). Learning can occur at various individual or collective levels (e.g., within organizations of policy actors, within policy venues, and across policy subsystems) (Heikkila & Gerlak, 2013). In this study, we investigate who learns and where learning processes and products occur in the context of data-based PILs. Specifically, our central research question is: how do data-based PILs acquire, translate, and disseminate data, information, and experiences in terms of learning processes and their products?
8Understanding public policy processes and outcomes can involve the acquisition and use of knowledge (Helco, 1974). Yet, policy learning is not limited to knowledge acquisition. It requires the transmission and dissemination of knowledge (Ostrom, 1997). Thus, the process of policy learning includes phases of acquiring, translating, and disseminating knowledge, information, and experience (Heikkila & Gerlak, 2013).
9The acquisition phase of learning involves the accumulation of information (Heikkila & Gerlak, 2013). This phase can result from deliberate actions, such as when policy actors intentionally seek out information about a policy problem or issue. For instance, seeking information could involve establishing monitoring or tracking policy activities (e.g., organizational strategies to influence policy outcomes). Acquiring information could also result from intentional experimentation to determine what new policies or programs are needed (Heikkila & Gerlak, 2013; Lipshitz et al., 2002). Information can be gained from internal and external policy actors or venues. Individuals learn from interactions among members of a policy organization or from outsiders with diverse knowledge and through interactions or active dialogue (Weber, 2009; Heikkila & Gerlak, 2013). As individuals have limited capacity to absorb diverse knowledge alone, collective learning across diverse policy actors is often necessary to access relevant information for a given policy problem (Ostrom, 1997).
10The translation phase of a collective learning process involves the interpretation and use of new information. Heikkila and Gerlak (2013) have noted that through the translation process, acquired information becomes “knowledge” (p. 489). The translation of information may involve an active process, such as the analysis of data or information. Such processes may require technical experts to help interpret accumulated data. The interaction of experts in a policy field or across diverse fields helps policy actors learn about various aspects of the policy problem and experiment with different methods to achieve policy objectives (Helco, 1974; Sabatier, 1988).
11Acquired and translated knowledge requires dissemination in collective learning processes. Disseminating information and knowledge across groups, organizations, or key individuals is vital to producing better plans, programs, or policies. Shared collective routines can facilitate the agile transfer of ideas, information, and knowledge (Heikkila & Gerlak, 2013). Diverse forums or venues enabling communication and shared dialogue can facilitate dissemination. These venues may include formal and informal meetings, sessions, or performance evaluation processes (Heikkila & Gerlak, 2013).
12While this study focuses on learning processes, it is important to acknowledge how these processes can lead to learning products, such as cognitive or action-based changes through expanded strategies, plans, programs, policies, or institutional arrangements (Heikkila & Gerlak, 2013). Of course, there is no guarantee that learning processes will produce learning, often due to time delays and policy resistance (Ghaffarzadegan et al., 2011). Political support is important for policy development and the successful production of learning products, particularly in the policy sector dealing with technological change. Policy-induced technological innovation and change is often observed, particularly in areas such as climate and energy, which serve as examples of proactive policy interventions (Schmidt & Sewerin, 2017). Furthermore, this technological change also influences subsequent technological development.
13Analytical levels of learning enable the examination of relationships between learning processes and products (Dunlop, 2017). Examining the levels at which data-based PILs engage in learning processes is important for assessing not only how learning occurs internally within a PIL, but also how learning can extend beyond the PIL to the broader policy system. Such learning may offer insights into how learning can be institutionalized across a policy system. Yet, as discussed below, there are also barriers to learning in any policy. Our secondary question is: to what extent can data-based PILs enable learning both internally and externally?
14Internally, within a PIL, learning can occur at the micro-level or among individual policy actors. McLaughlin (1987) has highlighted the individual as the smallest unit in the policy process, stating, “[a]t each point in the policy process, a policy is transformed as individuals interpret and respond to it” (p. 174). Individuals may be driven by institutional incentives or personal motivation to learn from new information and take relevant action (McLaughlin, 1987; Sabatier & Mazmanian, 1980). Dunlop and Radaelli (2017) argue that individual learning processes rely on decision heuristics that follow individuals’ past experiences. Yet, past experiences and beliefs can also create cognitive biases (e.g., motivated reasoning) that constrain learning (Heikkila et al., 2023).
15Internally, PILs can also enable more “meso” or organizational-level learning. Learning at the organizational level is mainly concerned with organizations adapting to changes in their operating environment (Cyert & March, 1963; Moyson et al., 2017). Such learning involves detecting and correcting errors or anomalies in organizational processes to achieve objectives. Yet, institutional conditions, authority, and power dynamics of organizations can also potentially impede learning.
16At the macro level, learning can occur across an entire government or policy system. System-level studies have analyzed how one government’s policy decisions affect other governments (Moyson et al., 2017). Such processes can be associated with both micro- and meso-level learning - or learning within a group or organization in a policy system. For instance, policy makers in one institutional setting may learn from policy decisions in another setting, which is referred to as policy transfer (Dolowitz & Marsh, 2000), and such policy transfer can have systemic effects. However, at the system level, political forces and structural challenges (e.g., lack of connectivity within the system) can create barriers to learning (Heikkila & Gerlak, 2024). Thus, venues and other organizational structures, such as PILs, can theoretically support learning across policy systems.
17Policy labs co-create and co-produce policy products through processes of designing, testing, and modifying public services and programs (Lee & Ma, 2020). Ideally, PILs should be designed to support learning processes and learning products at multiple levels, but empirical analyses of how PILs enable such learning are under-studied. Studies have acknowledged the relationship between PILs and policy learning processes and products in different ways. For instance, Dekker et al. (2020) describe the role of PILs in policy experimentation and innovation. Others recognize how policy labs help policymakers test and modify a policy before implementation or disseminate policy ideas by bringing together government and stakeholders, generating scientific evidence, and prototyping and simulation (Lee & Ma, 2020; Williamson, 2015). Policy labs are also described as a platform for developing policy solutions based on co-learning processes of piloting, pretesting, and prototyping potential policy solutions (Ojha et al., 2020). They are also described as laboratories for promoting learning across multidisciplinary organizations or stakeholders (Wanner et al., 2018).
18In general, policy experimentation is a key function of policy labs, often using digital data or innovative methods to help spur novel policy ideas across government (Lee & Ma, 2020). Engaging in evidence-based experiments allows governments to acquire, translate, and disseminate relevant knowledge that promotes innovation and collaboration. For example, prototyping enables policy lab participants to engage in iterative learning cycles. Furthermore, exploratory experiments and prototyping produce proof of policy concepts that have a greater likelihood of implementation (Kimbell, 2015). Experimentation with proposed policy solutions is important because policies are difficult to modify or may often be irreversible during the implementation phase (Ghaffarzadegan et al., 2011). PILs provide a space for policymakers and stakeholders to test the potential of proposed solutions to improve policy outcomes.
19PILs can also support learning as venues where diverse policy actors come together, share their ideas, and co-produce knowledge to solve a public problem (Ojha et al., 2020; 2021). Furthermore, data-based PILs can promote citizen and community engagement to create a better user experience from the perspective of a human-centered design. The importance of involving local communities in the design and implementation of policy activities and programs has been noted as a way of developing effective policy outcomes (McNamara et al., 2020). Diverse policy actors bring their own knowledge and expertise to a policy problem, which can be a key facet of learning for policy development and innovation processes. While many challenges to genuine community involvement exist across the public sector, there is a growing need to integrate alternative forms of knowledge into policy to address pressing public problems (Ojha et al., 2021). Policy labs can promote networked governance and collaboration to devise policy solutions for complex challenges (Brock, 2021).
20Studies of PILs also recognize that they promote policy learning across agencies as a means of knowledge transfer (Lee & Ma, 2020). Knowledge transfer is key to policy learning across organizations or agency networks (Janowicz-Panjaitan & Noorderhaven, 2008). PILs can use training and mentoring as modes of knowledge transfer across agencies to support policy innovations (Lee & Ma, 2020). Policy labs also undertake activities such as prospective studies, creativity workshops, or empowering civil servants through training, which can theoretically foster learning. Based on these insights from the literature, we expect that PILs, particularly data-based PILs, have the potential to support learning processes and products internally within their organizations (e.g., at the micro and meso levels) and externally across policy systems. This study explores how these processes might occur within the context of data-based PILs. Specifically, we suggest that data-based PILs may facilitate learning processes by helping individuals, groups, or organizations acquire information and share knowledge within their organization or with other organizations. Additionally, while this study focuses on learning processes in data-based PILs, we also consider the possibility that data-based PILs may contribute to learning products, such as expanded strategies, plans, and programs.
21The above literature on policy learning and policy innovation labs informed our analysis of learning processes in data-based PILs. We examine how data-based PILs acquire, interpret, and disseminate data, information, and experience to understand their role in policy learning processes. We also investigate whether new programs or plans are proposed or adopted as learning products and how PILs support learning internally within PILs and externally across policy systems. While the literature provided an initial guide for our research, we designed the study to remain open to emergent themes and concepts that may be missing given the nascency of research on learning in PILs.
22Using interview data, we analyzed policy learning in nine data-based PILs in the US, representing all levels of government and the non-profit sector. A total of 12 semi-structured key informant interviews with key policy lab personnel were conducted via Zoom between December 2022 and January 2023. The breakdown of interview participants by PIL affiliation is presented in Table 1.
Table 1. Interviewed policy labs and participants
Policy lab
|
Location
|
Number of labs
|
Number of participants
|
Federal government
|
Washington, D.C.
Washington, D.C.
|
2
|
4
|
State government
|
Denver, Colorado
|
1
|
1
|
Municipal government
|
Louisville, Kentucky
Syracuse, New York
Washington, D.C.
|
3
|
4
|
Non-profit organizations
|
Los Angeles, California Sacramento, California
Denver, Colorado
|
3
|
3
|
Total
|
|
9
|
12
|
Source: the Authors
23Following Yin (2009), we examine differences in learning processes and products across diverse types of data-based PILs. These include interviewees from federal, state, local, and non-profit data-based PILs in the United States. We focus on data-based PILs in the US, as they are important in resolving public problems and producing desirable policy outcomes through collaboration with government agencies and relevant organizations. The interviews were conducted and recorded on Zoom and were 30 to 45 minutes long.
24We developed semi-structured interview questions to investigate learning processes, particularly who learns and how they learn (e.g., Bennett & Howlett, 1992), and learning products – or what they learn – and at what levels. Open-ended, semi-structured questions help avoid biasing responses toward pre-conceived expectations and support a more inductive analytical approach. We also contextualized this information in terms of the purpose of each PIL and its role in different policy processes.
25Interview questions about the functions of the labs provided a basis for understanding how learning processes are enabled in data-based PILs:
-
What does your organization do to help solve a public problem and produce innovative policy outcomes?
-
Could you provide an example of how your organization is asked to address an issue or problem?
-
Could you explain the types of data your organization uses and the process of acquiring and interpreting data?
26Additional interview questions allowed us to elicit information about processes that support different levels of policy learning, both internally and externally:
-
What types of processes or channels does your organization use internally to share and disseminate knowledge and ideas, individually or collectively?
-
Could you think of any examples of sharing knowledge or ideas with other organizations, or disseminating your lab’s knowledge or ideas to other organizations?
27Content analysis of the interview transcripts identified codes and categories from the learning literature and emergent themes. We used a conventional content analysis approach, which allows for the emergence of new insights (Kondrachi & Wellman, 2002; Hsieh et al., 2005). Transcripts were reviewed in an iterative way to gain meaning and derive meaningful codes associated with learning processes. ATLAS.ti was used for interview data management, coding, and analysis.
28From our review of the first interview question posed to the PILs, respondents described a range of PIL functional activities that they employ to understand public problems and support innovative policy outcomes. These can be seen as indicators of both the processes and products of learning embedded in the functions of data-based PILs (see Table 2).
Table 2. Data-based PILs Functions
PIL functions
|
Federal government
|
State government
|
Municipal government
|
Non-profit organization
|
Total
|
Program evaluation
|
1
|
1
|
4
|
1
|
7
|
Better implementation
|
2
|
1
|
3
|
1
|
7
|
Human-centered design
|
1
|
1
|
2
|
0
|
4
|
Government Relations
|
1
|
0
|
0
|
2
|
3
|
Procurement
|
0
|
1
|
1
|
0
|
2
|
Research partnerships
|
0
|
0
|
0
|
1
|
1
|
Data Science
|
0
|
0
|
1
|
0
|
1
|
Data management
|
0
|
0
|
1
|
0
|
1
|
Cross-source linking
|
1
|
0
|
0
|
0
|
1
|
Consultation
|
0
|
0
|
0
|
1
|
1
|
Source: the Authors
29Looking at what data-based PILs do, there is evidence of learning processes built into their daily work practices. Several work with other government agencies and help to solve public problems and produce new policy programs. Respondents often engaged with other agencies to generate and use evidence to inform policies and programs and, ultimately, to improve implementation. The most popular activity was program evaluation – a key tool for the acquisition phase of learning processes – which included methods such as randomized evaluations, quasi-experimental methods, and descriptive studies. In many cases, existing administrative data from government agencies were utilized.
30Respondents from federal data-based PILs highlighted the importance of human-centered design approaches to develop better user experiences. This is another way of acquiring information that is often overlooked in the policy learning literature. They pointed out that human-centered design approaches have effectively built applications that benefit users. For example, one lab built a user-friendly online application for veterans’ disability benefits. The lab consulted with veterans about their challenges with existing applications so they could improve the application. In this process, iterations on design and testing products were performed to satisfy users' needs. In doing so, they developed built-in feedback effects from the process for not only acquiring information but also helping translate what that information means for those likely to be affected by a policy. The importance of human-centered design often outweighs the quantitative evaluation approach, as noted by the following respondent from a federal PIL:
“A lot of the work we do, especially from a human-centered design approach, is not always about being statistically significant in terms of numbers especially when we’re looking for the human element and the real-life story of the people that we’re talking to, in the experiences that they’re having.” (DPIL 1)
31A state government respondent also emphasized the necessity of adopting a human-centered design perspective when developing new programs or enhancing existing programs, services, and products. This respondent found that the use of technology and data supported the translation and dissemination of knowledge, permitting them to launch a program for residents.
32Municipal data-based PIL respondents stated that they endeavor to improve administrative processes – a learning outcome – through various processes. Both municipal data-based PILs focus on performance management, increasing efficiency, and improving administrative processes, including procurement and data warehousing. The PILs collected data to track and examine performance and identify gaps between current performance and expectations, an important step in the information translation phase of the learning process. In the case of procurement, the municipal PILs responded that they try to make their procurement functions efficient and effective, with a focus on equity. This reflects an important consideration in the policy learning literature, which is how normative goals can steer or illuminate the goals of learning. The municipal PILs supported these goals by building data platforms, similar to a data warehouse, to better manage data, with much easier and more timely access to data when needed in administrative processes. These efforts contributed to building an ecosystem for the responsible use of data in government and supported learning at the system level, as highlighted by one of our respondents from a municipal PIL.
“Most of the data science work that we've done so far has been around using predictive modeling to target resources to the people or places that we think can benefit most from them or that the agencies think can benefit most from them.” (DPIL 7)
33We also interviewed data-based PILs that were more research-focused and independent of government (e.g., non-profit PILs). One of these PILs responded that they had launched over 200 research projects in collaboration with public agencies in six policy areas: health, criminal justice, homelessness, labor and workforce, social safety net, and education. Thus, they have often used external sources of information in the learning process. This raises important questions about how learning across the policy system can be enabled through data-based PILs that are more independent of other policy actors. One mechanism to support such learning is by establishing long-term research partnerships with public agencies and co-creating research agendas. Such collaboration with public agencies can enable PILs to help to solve urgent policy problems that communities face. The type of learning products that can emerge from such processes, according to one of our respondents from a municipal PIL, include “…solving policy problems and directly informing government decisions with the most rigorous evaluation and evidence we can.” (DPIL 4)
34Our interviews also focused on the types of data and information that PILs used for data gathering (acquiring information for learning) and approaches to interpreting data (translating information for learning), key steps in collective learning processes. The federal data-based PIL respondents stated that they tended to use administrative data already collected from a specific agency or a pair of agencies (see Table 3). Examples of existing data include census data, privately purchased data, and Medicaid budget data. One federal PIL respondent pointed out that they also generated original data as part of a pilot project.
Table 3. Types of data used by data-based PILs
Types of data
|
Federal government
|
State government
|
Municipal government
|
Non-profit organization
|
Total
|
Administrative data
|
2
|
1
|
3
|
1
|
7
|
Existing data
|
1
|
0
|
1
|
3
|
5
|
Interview data
|
1
|
0
|
1
|
1
|
3
|
External survey
|
0
|
0
|
0
|
3
|
3
|
Internal survey
|
0
|
0
|
1
|
1
|
2
|
Source: the Authors
35State and municipal data-based PIL respondents also used administrative data collected by state agencies. Coordinating with actors outside the system can therefore help enable not only internal learning, but also learning across policy actors. For instance, these data can be employed to develop a particular service or learning product, such as simplifying or modernizing the childcare application process or veteran services. Using state agency performance data, one municipal data-based PIL developed and managed an open data performance statistics portal program. They mentioned the importance of connecting systems in a meaningful way using existing or newly collected quantitative data, which data-based PILs need to obtain policy-relevant information. In many cases, however, the divergent software or technical systems that government agencies use for data storage and management can impede both the acquisition and dissemination of knowledge in the learning process. As one respondent from a state PIL noted, “…the systems with meaningful data are rarely connected in a way that allows you to analyze that information” (DPIL 6). In other words, the technical and structural context within which data-based PILs are embedded can be an impediment to learning without intentional efforts by PILs or other actors to overcome the barriers.
36PILs also acquired and translated information in learning processes through qualitative analysis, focusing on human-centered design. This involved gathering qualitative information about people’s experiences in terms of problems they face, existing products, or needs for improvement, using professionals skilled in human-centered design. One respondent in a municipal PIL described how they used a human-centered approach that took into account citizens’ experiences to obtain a city permit. The non-profit data-based PIL exclusively used government administrative data or privately purchased data, such as credit bureau data. A few PIL projects, however, included interviews or focus groups, such as with homeless individuals, for a qualitative perspective. One respondent from a municipal PIL pointed out the strategic importance of conducting interviews for their learning capacity:
“A lot of times, what we’re doing in interviews of focused groups is not true qualitative empirical research. It’s like strategically enriching our understanding of the processes and people that we’re seeing in the data.” (DPIL 4)
37The respondents highlighted several barriers to acquiring information and therefore learning. For instance, one respondent from a municipal PIL mentioned the lack of standardization in data systems across agencies, illustrating their role in linking data from different sources. This municipal PIL respondent described in detail the challenges to aligning data:
“And a big part and a big time-consuming part of our work is establishing data use agreements between our larger office the office of the city administrator and different government agencies sometimes outside vendors but mostly district agencies to make sure that you know we're aligned and have an agreement on the objectives of what we are doing… that we have….you know the laws that govern the sharing of the data and the use of the data and kind of clearly laying out what data we will use and for what purpose and how we will publish the data in the end. So it's a big mix of different things. And you know some of the programs are a couple hundred people that we're working with. Some of the other ones are more like millions of records.” (DPIL 7)
38We also asked about the processes or channels the PIL uses internally for sharing knowledge, ideas, and information, which helps us unpack the dissemination phase of policy learning processes (see Table 4). While such channels can vary widely across data-based PILs, the most common channel for dissemination is through weekly or monthly meetings or seminars. These channels underscore how important face-to-face communication can be for the uptake of knowledge in the learning process. One leader from a federal PIL mentioned “information sharing sessions with each other, which we call nitty gritty, where we will talk about a range of different topics every week” (DPIL 1).
39One effective approach to disseminating knowledge within PILs to foster collective learning was for policy lab members to present their ongoing projects within their labs at different stages of the process. During these meetings, valuable feedback was provided. The respondents also highlighted their internal learning opportunities from sub-groups of experts, often organized by areas of specialty community or practices, such as product design, engineering, data science, or procurement. This illustrates how the structure of organizations can play a critical role in shaping how knowledge is communicated and disseminated within a data-based PIL.
Table 4. Internal learning processes in data-based PILs
Internal
learning processes
|
Federal government
|
State government
|
Municipal government
|
Non-profit organization
|
Total
|
Knowledge sharing meetings
|
2
|
2
|
3
|
2
|
9
|
Communication channels
|
3
|
1
|
1
|
2
|
7
|
Professional development
|
0
|
0
|
1
|
1
|
2
|
Newsletter
|
0
|
0
|
1
|
0
|
1
|
Q and A session
|
0
|
0
|
0
|
1
|
1
|
40Source: the Authors
41Data-based PILs mentioned other important communication channels, such as Slack, for effective information exchanges, including file storage systems. The data-based PILs manage and share their own file storage systems, where members of the PIL can access data, documents, and resources on other projects. One respondent from a federal PIL noted: “In the file storage system in our organization, almost everything that members work on at any point is accessible to team members because of the file storing structure.” (DPIL 2). Another respondent from a municipal PIL offered a detailed example of how such sharing is enabled:
“For example, we have a social science channel, a data science channel, and a civic design channel where people and these are both for people who are in the lab and some people that are external to the lab at other agencies or kind of people who have worked here previously and have gone elsewhere but are still on our communication channels where they can be like oh this is an interesting article or I'm encountering this issue does anyone know how to solve it or dealt with it? And so a lot of that kind of ad hoc learning, but having like a small community to draw upon for advice in different ways.” (DPIL 7)
42Some respondents pointed to their regular newsletter systems with an active email list. Communications directors sent out the regular newsletter and members received summaries of current projects and trends, as well as publications of academic affiliates.
43To help us better understand the capacity for learning at the system level, we asked respondents to identify effective approaches to sharing knowledge or ideas with other organizations or disseminating knowledge to other organizations. As shown in Table 5, federal data-based respondents mentioned that they learn organically from external agencies or groups as they collaborate with other agencies on a project basis. Knowledge transfer happens in multiple ways when the PIL collaborates with agencies and when it helps them with product or service work.
Table 5. Learning approaches with external agencies in data-based PILs
Learning approaches with external agencies
|
Federal government
|
State government
|
Municipal government
|
Non-profit organization
|
Total
|
Knowledge sharing meetings
|
1
|
1
|
1
|
2
|
5
|
Conference attendance
|
2
|
0
|
2
|
1
|
5
|
Professional development
|
2
|
0
|
1
|
0
|
3
|
Other city examples
|
0
|
1
|
1
|
0
|
2
|
Networking events
|
0
|
0
|
1
|
1
|
2
|
Government relations
|
1
|
0
|
0
|
1
|
2
|
Engagement sessions
|
0
|
0
|
1
|
1
|
2
|
Public health education
|
0
|
0
|
0
|
1
|
1
|
Podcast
|
0
|
0
|
1
|
0
|
1
|
Newsletter
|
1
|
0
|
0
|
0
|
1
|
Legislative sessions
|
0
|
0
|
0
|
1
|
1
|
Community-based partners
|
0
|
0
|
1
|
0
|
1
|
Source : the Authors
44Academics from non-profits, research institutes, or universities often join data-based PILs for one to four years with the goal of knowledge transfer. Data-based PILs also initiate training or education sessions for public employees to develop knowledge across governments. The goal is to support collective learning among individuals who have different knowledge or skills. As one respondent from a federal PIL noted:
“We run a training series for federal employees about how to do evaluation. A few different topics are really aimed at federal people, employees who arrive in different ways, with different levels of technical skills.” (DPIL 2)
45Municipal data-based PILs often held regular meetings with leaders from other municipal PILs. Respondents added that they could learn from partnerships between cities, private companies, local non-profit organizations, universities, and community-based organizations. Respondents highlighted the importance of established networks, such as a digital services network and a civic analytics network, which mostly focus on data and technologies. Members of PILs received new knowledge and information from colleagues at a variety of organizations in their networks. Respondents described how learning about effective services, such as a procurement system developed by other cities and counties, helped data-based PILs provide better services to their own citizens. One municipal PIL respondent provided a specific example:
“We actually went to Seattle a few months ago to learn about what they are doing in procurement, and they get to learn from what we are doing in procurement.” (DPIL 5)
46Conferences are another way that data-based PILs describe learning from other organizations. There, they could talk openly about the state of the service and address topics such as technology and innovation. The non-profit data-based PIL mentioned that knowledge exchange depends on working groups, but also mentioned conferences for PILs, including urban labs, crime labs, and education labs, to learn about operating procedures and products.
47Various contextual factors were identified that played a key role in learning in data-based PILs. We found that innovation as learning products can be achieved when there is sufficient political support for data-based PILs. This confirms previous studies that political support is critical for learning products to be safely accomplished (Girod, 2016). For example, municipal data-based PILs emphasized that their operation depends on how mayors support data-based PILs and how they value the importance of technology and innovation in developing public programs, products, and services. As one municipal respondent explained:
“I do want to stress that our mayor was genuinely so supportive of innovation and risk taking. I know many other mayors wouldn't let me do some of the things that I've done, but it's paid off. I would even argue even though the riskier things we did actually created more public trust because we were so transparent and inclusive in the process….Our mayor also wants his staff to have both the vision, the capacity and professional development, the support from their managers and supervisors, to do both continuous improvement and breakthrough innovation.” (DPIL 4)
48Additionally, community trust in government and data-based PILs is critical to moving forward with plans for data-based PILs. As data-based PILs use advanced technology and innovative services, community trust, which is indicative of community readiness or acceptance, must be assessed to proceed with policy. An example of the importance of such trust and willingness to learn was provided by the following municipal respondent:
“There might be a lot of work we did back in 2018 around drones. As you can imagine, the FAA regulates how drones are to be operated. In fact, one of the major barriers, although they're working towards experimentation on how to responsibly let cities operate in this manner, are letting drones operate at night, allowing them to fly beyond the visual line of sight, and actually the hardest thing to do is flying over people. As you can imagine, in an urban environment, you're definitely going to be flying over people. However, you could imagine how much public utility there could be unlocked by being able to use drones. So, is the community ready for this? How do you make sure that you have community trust and a policy that does its best at balancing utility for agencies while protecting residents at the same time? Community trust is significant; it can be a significant barrier to launching a project ultimately.” (DPIL 4)
49Municipal data-based PILs and non-profit data-based PILs mentioned barriers relating to funding their organizations. Municipal data-based PILs valued the funding that enabled the PILs to operate and launch projects, as they relied primarily on philanthropic contributions. While this was a challenge, it gave them a certain degree of autonomy and flexibility. They were able to focus on the most interesting or impactful questions and implement novel methods to solve public problems. In this respect, data-based PILs emphasized the importance of external funding and expertise in securing funding sources to undertake meaningful and innovative projects, as well as to maintain flexibility. Respondents in a non-profit data-based PIL and a municipal data-based PILs mentioned funding problems as follows:
“Another barrier is funding. I mean, there are times when you demonstrate that this is an amazing project, you've collected the evidence, and yet the need is so great while the funding is still so small compared to the need that perhaps your project does not rise to the priority level that gets it funded.” (DPIL 3)
“Expertise for our funding sources is critically important because we tend to rely primarily on philanthropic dollars which gives us flexibility. If you are only responding to government contracts, they're typically not written well and they're not focused on the most interesting or impactful questions, and they don't allow for the best methods. So, you need that sort of external funder so that we can be flexible and really sort of turn on a dime.” (DPIL 4)
50From the interviews, it appears that data-based PILs have taken diverse approaches to acquiring and utilizing information and data, and to disseminating information and knowledge. To some extent, data-based PILs collect information and coordinate with government agencies and other organizations to develop or expand programs and services that better address the needs of citizens.
51Data-based PILs use data and analytical expertise to resolve public problems, often with the goal of improving the experience of citizens affected by policies or government programs. Using semi-structured interviews, we investigated policy learning within data-based PILs. This research was guided by both the literature on policy learning and PILs, while allowing for emergent themes to inform our findings. We examined how the goals and functions of data-based PILs as a whole relate to learning processes and products. We also traced how PILs acquire, translate, and disseminate knowledge and information both within and outside of PILs. Although only a small number of interviews were conducted, the evidence suggests that data-based PILs may support a more efficient accumulation of information and help establish connections that facilitate the sharing of new information and experiences. Additionally, the data-based PILs we studied often try to institutionalize knowledge to effectively adopt knowledge and information across agencies or policy actors in a policy system.
52We observed more explicit learning processes than learning products (or outcomes) through our key informant interviews. For example, in acquiring information, data-based PILs often used administrative data but also sought human-centered knowledge from surveys and interviews. Different types of data and knowledge are central to effective learning processes (Heikkila & Gerlak, 2024). In the translation phase of the learning process, we found that data-based PILs played an important role in adding analytical capacity with experts in data, technology, human-centered design, and strategic management. Based on our findings, the inception of the work of data-based PILs usually began with government requests. Hence, the phases of acquiring and translating data and information tended to be handled internally among the sample of PILs in our study.
53Our findings also indicate that data-based PILs have, to some extent, served to disseminate information and experiences outside their labs to the broader policy system. Data-based PILs are part of a broader policy system within which learning occurs This study provides evidence that policy labs can support learning across agencies and enable stakeholders to share their knowledge and information through a range of activities, including workshops, training, and mentoring. These findings align with recent studies that have highlighted similar activities by policy labs (see Lee & Ma, 2020; Tõnurist et al., 2017).
54Relatedly, we found that the PILs in our study used more diverse channels to share knowledge with external organizations compared with modes of knowledge-sharing within the lab, but they also appear to be more ad-hoc relative to their internal knowledge sharing channels. Although the data-based PILs in our study contributed to building standardized systems for different data networks and formats dispersed across agencies, they also faced difficulties in learning externally with other organizations. This was due to the lack of more systematic learning mechanisms that they have established internally.
55A key insight for the literature from these findings is that data-based PILs appear to be well-equipped for learning at the meso (or organizational) level. While our interviews found that data-based PILs interact with other organizations to share knowledge and export technical expertise, suggesting their potential to support macro-level or system-wide learning, such learning can be difficult. As noted above, system-wide structural issues can create barriers to information acquisition and dissemination. That said, data-based PILs can play a role in the learning potential of policy systems by adapting to rapidly changing environments with their data and technology expertise.
56Learning at the micro level is also likely to occur through data-based PILs through their efforts at professional development, learning about new technology, and sharing knowledge and experiences with others. However, we did not directly measure individual-level learning in this study, whether internally or externally to the PIL. Future research could examine how data-based PILs support individual learning among policymakers or even among policy recipients. For instance, some policy labs offer assistance to citizens who struggle with low accessibility to rapidly changing technologies or data-related resources. Whether these individuals learn to navigate such technologies more effectively can be important for understanding equity in local communities and the ability of PILs to address issues that the government alone may not be able to manage.
57While data-based PILs engage in learning processes, they also face constraints. A focus on “data”, especially quantitative data, can obscure relevant information for learning how to improve policies. Also, the data-based PILs we studied often used administrative and existing data from public agencies, with more limited use of new data from outside sources. This reliance on existing data and focus on inward processes could limit learning, especially at the system level. However, respondents highlighted the importance of user experience and human-centered design when dealing with human elements and real-world experiences. Such efforts may play a key role in supporting a richer and more relevant policy learning process, at least internally within data-based PILs.
58Additionally, when data-based PILs sought to develop and manage open data systems using multiple data sources, they faced difficulties due to different data systems across agencies. These institutional differences may create structural constraints to system-level policy learning from the data and information produced by data-based PILs (Heikkila & Gerlak, 2024). While data-based PILs may serve as a venue for supporting connectivity in a policy system, it is important to understand how the system, or their own internal processes, may limit their capacity to serve in this role.
59In examining the learning processes of data-based PILs, we found that they followed the phases of acquiring, translating, and disseminating data, information, and experiences across the organizations with which they work or to similar types of organizations. However, it was impossible to trace learning in each phase in detail from the interview data. Additionally, we did not examine learning products in depth in this study. Learning products can result in new programs, services, or systems that emerge from the work of data-based PILs. Whether or not such products function well over time is an important question. Research on learning can be biased by assuming that learning products are positive or productive. Yet, as one of our respondents indicated, data-based PILs can create products that lead to unanticipated burdens or costs to citizens if they are not designed well. As our research focused more on understanding the tools and structures that data-based PILs use to support learning processes, we did not evaluate such outcomes or learning products.
60Finally, our study was limited to a small number of PILs and a particular type of PIL in the US. We acknowledge that the sample size may limit the ability to make strong claims about how learning processes function in data-based PILs. However, this study can serve as a starting point for future research. Extending this research is essential for theory testing and building. Future studies investigating data-based PILs across different countries should be conducted to observe any similarities or differences between data-based PILs in different countries. This investigation would provide information about possible barriers to learning, such as structural challenges across agencies or support for learning within PILs and with external organizations. As learning helps individuals, groups, and societies to develop, it is important to understand what elements support learning and what limits learning as we scale up from individual learning to collective learning in different policy settings. While more research is needed on the role of PILs in policy learning, this research provides a starting point for understanding how data-based PILs engage in learning processes. Through their work and collaboration with government agencies using data and technology to produce improved public products and services in rapidly changing environments, data-based PILs can play an important role in supporting the acquisition, translation, and dissemination of knowledge about emerging public sector issues and the tools for addressing them.