1 Introduction

Growing understanding of sea-level rise, and its significance for coastal communities, has prompted burgeoning interest from coastal decision makers in the data, information and knowledge required to manage coastal hazards into the future. Managers of the impacts of coastal hazards include decision makers at all levels of government and from business, industry, non-government organisations (NGOs) and the community groups, and businesses or industries that depend on a coastal location or need to access coastal resources or populations. Such diversity of decision makers has focussed attention on the spectrum of knowledge needs and capacities and how decision-makers are best supported to effectively manage coastal hazards under a changing climate. Web-based decision support to address climate change includes platforms such as web portals (Sanderson et al. 2016), decision support tools and frameworks (DSFs) (Moss 2016) and online databases (Mitchell et al. 2016). These platforms enable provision of a range of data and information in flexible and accessible formats. The growing literature on DSFs for climate change adaptation acknowledges that if such tools and frameworks are to be usable and useful, and thus long-lasting, then their development requires collaboration between creators and potential users (Kopke and Dwyer 2016; Romsdahl 2011; Romsdahl and Pyke 2009).

Parallel with the development of a DSF is growth in broader concepts of scientific knowledge production that combines a plurality of knowledge types to build an integrated understanding of the system and takes account of the social and political context in which knowledge will be applied (Lemnos and Morehouse 2005; Meadow et al. 2015). The process—known as coproduction of knowledge—weaves together ‘three dimensions: interdisciplinarity, interaction with stakeholders, and production of usable science’ (Lemnos and Morehouse 2005, p.58) and involves ‘working iteratively and interactively toward collaborative learning, shared understanding of key concepts, and coevolution of common purpose and action’ (Nel et al. 2016, p.178). Cash et al. (2003), through related concepts of bridging ‘boundaries’, propose that when people are meaningfully involved in producing knowledge, they are likely to consider this knowledge as credible, salient, and legitimate and, in turn, are more likely to use this knowledge for implementation. These three attributes of credibility, saliency and legitimacy have become important benchmarks of effectiveness of the translation of science into policy or practice. However, as Heink et al. (2015) describe, they (and their inevitable trade-offs) are vague and depend on individual framing and thus require more specificity to be useful.

This paper reports on a web-based decision support and information delivery framework, CoastAdapt (www.coastadapt.com.au), developed during 2014–2017 to support coastal decision makers in Australia to adapt to climate change. The development and elements of CoastAdapt (also called here ‘the Framework’) are described by Palutikof et al. (2019a), while the consultation activities that focused on understanding and addressing user needs are described by Palutikof et al. (2019b). Here, we expand on these papers to provide further detail on the involvement of coastal decision makers in developing CoastAdapt. We use four types of feedback—an online survey, consultation workshops, advisory group feedback and participant observations by the development team—to reflect on how these activities supported the Framework to be usable. We provide a theoretical contribution through the analysis of an application of Cash et al.’s (2002) framework and an engagement tool (the IAP2 spectrum; IAP2 2007) in the context of the development of a DSF.

2 Rationale for, and development of, CoastAdapt

CoastAdapt was developed to underpin effective decision-making to support adaptation at all stages, from simple recognition of the risks through to planning, delivery, monitoring and evaluation in the coastal zone in Australia. Projections for Australia indicate a likely increase in coastal hazards associated with rising sea level, such as erosion, storm surge and inundation (McInnes et al. 2016) as well as changes in storm patterns (Christensen et al. 2013). Current estimates of sea-level rise for Australia lie in the range 6–19 cm by 2030 relative to the 1986–2005 baseline (CSIRO and BoM 2016). The Australian coast is particularly vulnerable, as the majority of the population, industry and infrastructure is located in the coastal zone: For example, the most comprehensive study to date estimated that AUD$63 billion in residential housing stock is at risk from 1.1 m rise in sea level (Department of Climate Change 2009).

Following production of two national assessments of climate change risks to coastal Australia (Department of Climate Change 2009; Department of Climate Change and Energy Efficiency 2011), in 2014, the Australian Government Department of the Environment (DoE) commissioned the National Climate Change Adaptation Research Facility (NCCARF) to ‘produce a coastal climate risk management tool to support coastal decision makers to understand future climate risks and provide practical guidance on how to manage the associated physical, social and economic risks’ (taken from the unpublished Funding Agreement).

NCCARF was selected for this project because of its role as a boundary organisation (Guston 1999), connecting science to practice and policy (Graham and Mitchell 2016). Increasingly, the concept of boundary work by such organisations has evolved from separating science from nonscience, to mediation between knowledge and action (van Kerkhoff and Lebel 2006; Clark et al. 2016). Since 2007, NCCARF has been generating, collating, synthesising and communicating knowledge to support decision makers throughout Australia as they prepare for and manage the risks of climate change (NCCARF 2014). For this project, NCCARF was given broad guidance from DoE through the funding agreement that the coastal risk management tool should be useful to coastal decision makers; however, the form and function were to be decided through an extensive consultation and co-production process with potential users. Given that the tool was required to meet the needs of a diverse range of coastal decision makers across all states and territories (and thus different geographic and jurisdictional contexts), online delivery through a DSF was considered the most appropriate mechanism.

Formally launched in May 2017, the coastal risk management tool became known as CoastAdapt and referred to as a framework rather than a tool. CoastAdapt fits Moss’s (2016, p. 146) definition of a DSF: ‘a knowledge framework that structures decision support processes of knowledge exchange to enable learning and experimentation’. CoastAdapt contains information and guidance to increase understanding of climate change and potential responses: Specifically, it includes:

  • Basic and advanced information and explanations of climate change and its impacts

  • Datasets on historical flooding, present-day coastal sensitivity to erosion, and future climate extremes, sea-level rise, and inundation for each coastal council

  • Tools for risk assessment, costing and planning action

  • Guidance on selection of adaptation actions and building support for implementation

  • A risk management framework supporting users through the six stages of adaptation from identifying the challenges through to monitoring and evaluation

CoastAdapt included (from April 2016 to June 2017) an interaction space to support a community of adaptors, but this area is no longer active: see discussion in Palutikof et al. (2019a).

In Australia, the key institutions for coastal management lie with state governments, yet local governments are day-to-day managers of the coast through their role in land use planning, asset management and ecosystem management. The national government’s role is the development of strategic policy for the coast (e.g. NRMC 2006), management of World Heritage sites such as the Great Barrier Reef, coastal defence and offshore dumping and shipping (Harvey et al. 2012). Other coastal decision makers include privatised utility providers, NGOs, community groups and businesses or industries that depend on a coastal location or need to access coastal resources or populations. Users of CoastAdapt are broadly defined to include managers in all of these groups, in particular local government and small businesses, which tend to have limited human and financial resources to undertake climate change adaptation. These are the key groups we sought to involve in co-production of knowledge for the Framework.

This paper reports on user involvement processes associated with CoastAdapt development, and how these aimed to contribute to CoastAdapt being usable and used through being credible, salient and legitimate. In doing this, we draw on four types of feedback as sources of data:

  1. I.

    An online survey conducted during the scoping phase, which included an open survey online for 3 weeks in March 2015, and completed by 131 individuals (see Palutikof et al. 2019b)

  2. II.

    Three rounds of consultation workshops, a total of 13 held in urban and regional population centres across Australia and attended by 330 individuals, conducted throughout the scoping, development and roll out phases (see Palutikof et al. 2019b)

  3. III.

    Feedback from three project governance and advisory groups (52 individuals in total) gathered at or after each meeting or workshop through a monitoring and evaluation process managed by an independent consultant (see Sect. 4.1)

  4. IV.

    Participant observation by the eight NCCARF members of the CoastAdapt development team: a multidisciplinary team including specialists in climate adaptation, coastal science, IT and graphic design.

3 ‘Usable and useful’: building relevance through engagement

3.1 Engagement of users in decision support

The growth of DSFs across many environmental domains has been matched by increased discussion about potential users and their needs (McIntosh et al. 2007; Romsdahl and Pyke 2009; Romsdahl 2011), concerns about low take-up by target users (Valls-Donderis et al. 2014) and the need for user involvement in testing so that frameworks will be usable and used (O’Dea et al. 2007; Romsdahl 2011; Voinov and Bousquet 2010). Increasingly proposed is that developers and potential users should collaborate in DSF design and content, for three principal reasons: First, so that content meets users’ expectations, contains legitimate and relevant information, for example, at appropriate spatial and temporal scales, and can be integrated with existing organisational knowledge systems (Dilling and Lemnos 2011; Valls-Donderis et al. 2014; Meadow et al. 2015); second, so that the process of negotiating the DSF and its content builds trust in the developers and faith in the final product (Lemnos et al. 2012; Valls-Donderis et al. 2014); and finally, so that users become active rather than passive participants, helping to build the DSF community and improve practice through social learning (Jakku and Thorburn 2010; Prokopy et al. 2017). Jakku and Thorburn (2010) define social learning in the decision support context as the ability to support stakeholders to interact, share different views and experiences, learn collaboratively and make collective decisions.

Despite increasing numbers of DSFs to support climate adaptation, few examples describe how these were developed and which or how users were involved. For example, Newman et al.’s (2017) review of 77 papers of natural hazard DSFs identified only 32 papers that reported user involvement. Of these, 11 were steered by user input, 11 evolved with user input, 7 reported end user training and 3 reported using champions. Kopke and Dwyer (2016) developed an online digital atlas of the global coastline and stressed the need to ‘identify potential audiences … at the start of the development process, and to remain engaged with them throughout’. In our reading of the literature, we identified four types of user engagement in DSFs (Table 1). Our aim for CoastAdapt was to use a ‘sustained’ approach that we defined as ‘users participate through a combination of engagement activities including formal references groups, testing, and training’.

Table 1 Types of user engagement in coastal decision support tools
Full size table

3.2 Using the IAP2 spectrum to design user engagement

In designing user involvement for CoastAdapt, we drew on the International Association for Public Participation spectrum (IAP2 2007); see NCCARF (2015a, 2015b). This framework for participation was chosen because, although generally used to describe public engagement, it is also used to describe involvement of users in DSF development (e.g. Valls-Donderis et al. 2014). In addition, it is widely adopted by Australian local governments for planning public engagement and so is familiar to many of our stakeholders (e.g. Serrao-Neuman et al. 2014).

The IAP2 spectrum outlines five levels of participation—inform, consult, involve, collaborate and empower—based on increasing levels of information, consultation and collaboration, with tools and processes suggested for each level (Fig. 1). To these, we added a pre-engagement step termed ‘information gathering’ that we defined as ‘collating existing information to plan informed engagement’. Our rationale for this step was to, first, demonstrably acquire extensive knowledge of current practice, so as to inspire confidence that the team was up to date with current issues and, second, minimise consultation fatigue (Crase et al. 2005; Brackertz and Meredyth 2009), as many of our stakeholders describe themselves as time poor (Palutikof et al. 2019a) and are frequently involved in engagement activities.

Fig. 1
figure 1

The IAP2 spectrum adapted to describe participation in CoastAdapt (adapted from IAP2 2007)

Full size image

3.3 Developing an effective DSF that is salient, credible and legitimate

Boundary organisations typically use boundary objects—collaborative products such as reports, models and maps—to negotiate knowledge between the realms of science, policy and practice (White et al. 2010; Clark et al. 2016). Increasingly, DSFs have potential as boundary objects: They enable users to participate in developing a decision tool, develop a shared understanding and social learning, and yet also adapt the tool for their own context (Jakku and Thorburn 2010). To ensure CoastAdapt usability and usefulness for decision-making, the development team drew on the three attributes identified by Cash et al. (2002, 2003) as essential for the effective translation of science into action—credibility, saliency and legitimacy. Following Cash et al. (2002), credibility refers to the integrity of the information and whether it is considered as authoritative, believable, and trusted. Saliency refers to the relevance of the information to decision makers, particularly in terms of timeliness and local applicability. Legitimacy reflects fairness, particularly of the process used to produce the information and whether it takes account of appropriate values, concerns and perspectives of different actors. These three attributes are interdependent but also require trade-offs (White et al. 2010). For example, providing local information can increase the relevance (saliency) of a DSF to coastal managers working in local government. However, if that information in any way deviates from or contradicts the user’s own perception of their local environment, that will impact upon credibility and legitimacy for that user (Leitch and Robinson 2012).

For a wicked problem such as climate change, it can be difficult to establish these three attributes (Leitch 2017). For example, climate change involves various types of uncertainty, e.g. scientific, epistemological, ethical and scalar uncertainty (Leitch and Robinson 2012), that can make it difficult to maintain credibility. Climate change typically highlights the conflict between different stakeholder values (Adger et al. 2009) and so whose knowledge is included or excluded from the process to produce a DSF can build or undermine perceptions of legitimacy. Climate change impacts depend on the context (Adger and Kelly 1999; Measham et al. 2011), and without resources to undertake an equitable process (legitimacy) to develop locally relevant information, the final DSF may lack saliency, which in turn undermines credibility.

4 The phases of development and engagement activities

The pragmatic goals for CoastAdapt were to have an easily-navigable DSF containing authoritative, relevant and effectively communicated information. It was also important to build users’ capacity to understand more about climate change and adaptation, as well as to plan for and take action, and to create a community of adaptors. Achieving these goals required significant and sustained involvement of potential users in the design and development processes, which meant that the need for adequate interaction was explicitly built into the project design from the beginning (Beiers et al. 2017). Engagement to develop the Framework and its content occurred in four phases—scoping, prototype, usability, and testing and training (which are similar to the four phases of DSF development described by Valls-Donderis et al. 2014).

We now describe the engagement processes of CoastAdapt and how they contributed to usability through support of the credibility, legitimacy and saliency of the content as well as supported action and community building by coastal decision makers. Table 2 provides more detail about the information gathering and engagement activities for CoastAdapt through the three governance committees and the four National Adaptation Networks (NANs).

Table 2 Information gathering and engagement activities for CoastAdapt through the three governance committees and four National Adaptation Networks
Full size table

4.1 Phase 1: scoping

The ‘scoping’ phase began with the ‘kickoff’ meeting (Beier et al. 2017) in November 2014, bringing together a diverse range of potential users and stakeholders to provide input directly as well as to build support through their networks for engagement in DSF development. This meeting discussed the need for a coastal framework and its aims, potential form, function and content.

Next, the CoastAdapt development team sought input from a range of coastal decision makers to better understand their context, needs and aspirations for managing the coast. This occurred through a series of short- and longer-term consultation events and engagement processes, as described below.

4.1.1 Three governance/advisory committees

Beier et al. (2017) recommend establishing technical and steering committees to help guide co-production of actionable science. Three committees were set up to guide the overall NCCARF program within which CoastAdapt development formed a major part. These committees participated broadly at the IAP2 ‘involve’ level as they were engaged throughout CoastAdapt development, mostly through quarterly 1-day meetings. Committee members were selected to represent all coastal jurisdictions in Australia, and from a wide range of experience in policy, practice and/or science, in a process designed to build credibility and legitimacy of CoastAdapt and its contents. Committee members also raised awareness of the Framework among their own stakeholder networks to help develop a community of adaptors.

The three committees are described below:

  1. 1.

    The Project Review Committee (PRC) consisted of state and sector representatives to provide national oversight of CoastAdapt content. These representatives ensured inclusion of knowledge and priorities from all the coastal state-level jurisdictions. The Australian government through the Department of the Environment and Energy had observer status. The PRC provided a national perspective (legitimacy and saliency). As a governance committee, it was required to endorse key deliverables (credibility), a process that signalled consistency with state policy and was an essential step if local governments were to adopt the Framework.

  2. 2.

    The End User Reference Group (EURG), an advisory group, consisted of representatives from local government and relevant businesses (i.e. with exposure to coastal hazards, such as utility operators). Members were drawn from across Australia and were selected for their local knowledge and experience to support legitimacy and saliency.

  3. 3.

    The Technical Reference Group (TRG), also advisory, consisted of relevant experts from the bio-physical and social sciences; their expertise added credibility and saliency.

4.1.2 The National Adaptation Networks

Four NANs were established to connect researchers and research users in government, business and communities, to build and maintain capacity to adapt to climate change. Each NAN focused on a different domain: Natural Ecosystems; Settlements and Infrastructure; Social, Economic and Institutional Dimensions; and Vulnerable Communities. Although they had a role independent of CoastAdapt, their involvement and input helped ensure saliency: They contributed content relevant to their different interests and sectors and also raised awareness within their networks to help build the community of adaptors.

4.1.3 Scoping processes

Three activities in the scoping phase aimed to elicit information on key problems for coastal managers caused by sea-level rise and climate change; understand what is needed to support coastal zone management under climate change and sea-level rise; and determine how a coastal DSF might help address these challenges. These were a desktop review or State of Play report (SOP), an online survey and a series of regional consultation workshops (supplementary information Table SI-1).

  • The SOP (NCCARF 2015a) collated and reviewed existing science, adaptation activities and regulatory frameworks, to ensure the saliency of CoastAdapt information and grow the development team’s knowledge (credibility). The draft report was reviewed by the PRC, EURG and TRG.

  • An online survey solicited user views on knowledge gaps and barriers to adaptation, in order to better understand where the focus of CoastAdapt should lie. It was distributed via the NCCARF website, the NCCARF mailing list of approximately 5000 recipients, the PRC, EURG and TRG, and NANs. There were 313 completed responses (Palutikof et al. 2019b).

  • Eleven scoping workshops with coastal decision makers were held across Australia, with 330 participants (Palutikof et al. 2019b). To encourage attendance and lend credibility, a local champion provided advice on potential participants and suitable timing, venue, and promotion. These champions also played a key role in the workshops through presentations on local issues and the local potential for a DSF.

4.2 Phase 2: prototype

The ‘prototype’ or development phase (whereby ‘an early concept of the look and feel of the Tool is developed and tested by the stakeholders’ (Valls-Donderis et al. 2014, p.73)) established the content and aesthetics of CoastAdapt (supplementary information Table SI-2). The PRC, EURG and TRG met regularly to discuss the evolving content of CoastAdapt. Some TRG members prepared technical, in-depth content including information manuals (e.g. on community engagement, coastal engineering and coastal planning). The NANs organised several capacity building events—mainly workshops for early career researchers and practitioners—which inter alia prepared and/or reviewed draft content.

4.2.1 Tool development partners

A Tool Development Partnership (TDP) was established as an advisory group of local-scale practitioners to ensure CoastAdapt relevance and accessibility. Membership was drawn from nine coastal local governments and one agricultural peak body. Members were selected through a competitive process, and their host organisations received a small grant to compensate for their time and ensure support for their continued involvement. Through contributing their knowledge of local context, organisational and practitioner issues, challenges and opportunities, the TDP made an important contribution to CoastAdapt salience and credibility.

The TDP met for 2 days approximately every 2 months during the prototype and usability (Sect. 4.3) phases to discuss progress, review content and contribute to user experience (UX) testing. This group shared deep experience with each other and the development team—perhaps more than was the case for the more formalised PRC, EURG and TRG. This deep interaction was attributed to the lack of a formal role with no terms of reference: The lack of an externally driven agenda allowed the issues that mattered to the group to emerge. Nevertheless, participation was at the involve level for the TDP—the same as for the three governance/advisory committees and the four NANs. In post-meeting feedback gathered by the independent consultant (Coutts and Coutts 2017), TDP members were very positive about their involvement and the resulting opportunities they experienced for national-scale networking at a level rarely available to local government and small business employees.

4.2.2 Case studies

The development team sought examples of ‘adaptation in action’ from the coastal community through case studies (and shorter ‘snapshots’) of adaptation that illustrated lessons learned, effective strategies and/or potential pitfalls. TDP members each contributed a written or video case study of local adaptation within their own organisation. Other individuals, groups and organisations were also encouraged to contribute such examples through offers of prizes and recognition. The case studies—currently around 80—aimed to foster a culture of taking action and increased the credibility and relevance of CoastAdapt by providing examples of ‘real adaptation’ already underway.

4.2.3 Monitoring current topics of interest

Through the consultation workshops, users expressed the need to digitally share knowledge and experience through on-line forums to support communities of practice (see NCCARF 2015b). An online forum, known as CoastExchange, was established in response. Over its 14 months of operation, it provided opportunity for moderated discussion, and carried 67 news and feature articles prepared by a communication consultant. It also encouraged users to pose questions each month that could be selected for expert responses: 27 questions were answered by eight experts. In addition to these roles, CoastExchange provided an opportunity for the development team to monitor topics of current interest to users (increasing saliency).

4.3 Phase 3: Usability

The usability phase—the Framework being ‘improved in terms of its appearance and ease of use’ (Valls-Donderis et al. 2014, p.73)—followed launch of the beta version of CoastAdapt in August 2016 and involved review of and refinement to content and usability (supplementary information Table SI-3). In this phase, the TDP changed to a Tool Implementation Partnership (TIP) that included six additional organisations and had an increased emphasis on outreach for CoastAdapt and mentoring for users.

Review and consultation on the beta version of CoastAdapt involved further feedback workshops, conducted in 19 city and regional locations (and where possible in the same locations as the scoping workshops). At each workshop, a TIP member presented on their adaptation experiences and the uptake of CoastAdapt in their own organisation, which added credibility and saliency to the Framework and its contents. A web-based survey also elicited feedback from users. Additionally, all pages on CoastAdapt had a feedback tab enabling users to provide comments. Major changes to CoastAdapt resulting from this beta version feedback are reported by Palutikof et al. (2019b).

4.3.1 Test cases

Test cases were carried out in two phases: usability and testing (see Sect. 4.4). In the usability phase, test cases contributed to the review process and led to modifications to CoastAdapt. Groups worked with an independent facilitator to apply CoastAdapt to a specific problem over 6 weeks and provide feedback and a case study of their experience. A small budget was provided to cover involvement costs. There were seven test cases in the usability phase: five councils, an airport operator and an aquaculture company. While feedback was positive overall, there were a number of suggested improvements on Framework utility and navigability, which were implemented where time and resources permitted (Palutikof et al. 2019b). This activity provided legitimacy through testing CoastAdapt in diverse contexts and credibility through the independence of the process.

4.4 Phase 4: testing and training

A key milestone of this phase, and a contractual requirement, was the endorsement of the final Framework by the PRC prior to final release. This meant that state governments would support decisions based on the Framework, which enhanced its credibility and saliency for local governments.

Six training workshops were held in state capitals (supplementary information Table SI-4), plus one training webinar to enable remote involvement at any time. Participants worked through two exercises: first, a first pass risk assessment for an imaginary coastal asset (see Tonmoy et al. 2019); second, an exercise in stakeholder mapping, prioritisation and engagement.

Eight further test cases were carried out in five councils, one grouping of councils, one aquaculture company and a credit union. These had the primary role of further demonstrating the utility of CoastAdapt, and so building its saliency among potential users. These test cases also produced case studies published in CoastAdapt.

Finally, this phase included the formal launch of CoastAdapt in May 2017, an activity additional to those of the testing phase outlined by Valls-Donderis et al. (2014). The event aimed to promote CoastAdapt to stakeholders considered key influencers among policy makers and practitioners, as well as to acknowledge and celebrate the involvement of the many contributors to the Framework.

5 Reflections on the engagement processes

Cash et al. (2002) described how knowledge into policy requires satisfying conditions of credibility saliency and legitimacy. We now reflect on how each of these attributes was relevant in development of CoastAdapt.

5.1 Ensuring credibility of the Framework creators and content

Establishing the credibility, or scientific integrity, of the development team and CoastAdapt content was a vital ingredient for project success. For example, a key need expressed by junior council staff during consultation was for advocacy—they wanted to use CoastAdapt to convince senior managers of the need for adaptation planning—the scientific integrity of CoastAdapt information and its developers is essential if it is to be used for this purpose.

Establishing credibility began at the pre-engagement phase. An important factor was the reputation of NCCARF as a climate adaptation knowledge brokering facility hosted by Griffith University. Stakeholders indicated that the ‘brand’ of NCCARF and Griffith University encouraged them to participate early in Framework development because they understood that an established and respected research facility would have buy-in from other recognised experts. Credibility was also enhanced by the pre-engagement preparation of the SOP report (NCCARF 2015a), which assembled the current state of knowledge on coastal issues nationally. This ensured that the team’s knowledge was current, aiming to inspire stakeholder confidence in the team’s competence and minimise potential frustration or consultation burnout. In later phases, local champions and the TDP/TIP members—as trusted local practitioners—lent credibility to the team through being involved in the workshops and explaining the local relevance of CoastAdapt.

Credibility of the content was supported during writing and review. Topic experts from academic, policy and practitioner backgrounds were engaged to write material and develop data sets and tools. An extensive peer review process followed with all material reviewed by a technical expert to ensure accuracy, a practitioner to ensure relevancy, and a communication specialist to ensure clarity and consistency. While a large logistical task for a small team, this review process was considered essential for credibility and saliency. Also, for a development team, an awareness of ensuring credibility is likely to motivate team members to sustain a focus on engagement (White et al. 2010).

One issue that emerged consistently across all consultation phases was the longevity of CoastAdapt. At all workshop and events, participants expressed concern that continuing support was uncertain. Without sufficient support, it would not be possible to carry out the maintenance and up-dating that ensures the continuing credibility of the Framework. Financial support to maintain and update CoastAdapt continues to be available. This is not assured beyond the middle of 2019, although plans are in hand to ensure that a library of CoastAdapt content is maintained should the structure of the Framework be discontinued. Concerns around longevity of publicly-available datasets and resources are common across the natural and physical sciences (see, for example, Moran et al. 2009).

5.2 Ensuring saliency of the Framework through relevance and accessibility

The key to ensuring saliency was facilitating input from diverse stakeholders from a wide range of organisations, job roles, locations and experience in coastal adaptation so that CoastAdapt would be widely relevant. The development team identified that the most effective engagement mechanisms were the consultation workshops and online survey during the prototype phase; the UX testing during the prototype phase; and the TDP/TIP input during the prototype and usability phases.

The consultation workshops and online survey gathered views from coastal managers and communities across Australia, highlighting their differences (and similarities) in perceptions of climate change risks, and thus in the potential value of a DSF that could provide content to satisfy a broad range of users. Palutikof et al. (2019b); Table 1) describes these perceptions and responses in detail and how some of these views helped to shape design and content. Other views raised issues that were more challenging but important to address if CoastAdapt was to be relevant nationally. For example, internet connectivity, while not an issue for capital cities, poor internet connections in some remote communities would restrict access to the moderately data-intensive Framework. In addition, whereas CoastAdapt is a national framework seeking for national consistency, planning issues in Australia are handled at state level, leading to concerns that CoastAdapt information might not match the regulatory and legislative contexts of individual states. NCCARF worked to overcome these concerns, for example, by providing each state with the opportunity to contribute a webpage to CoastAdapt populated with relevant state-specific information.

The UX testing was vital to support CoastAdapt navigability and accessibility. Its development of profiles of typical users, or ‘personas’, helped to refine an understanding of users through the formal testing process. The development team also reported value from in-team discussion of refining these personas.

The TDP was established to help ensure the relevance and accessibility of CoastAdapt. Members’ local scale and pragmatic input, and the trust that developed through their prolonged involvement throughout CoastAdapt scoping and roll out, allowed the Partnership to provide insightful and forthright perspectives that the development team reported as invaluable. Its successor, the TIP, directly contributed to saliency through raising awareness of CoastAdapt through its networks, and as trusted independent participants in CoastAdapt consultation and promotion activities.

One activity that perhaps did not merit the expenditure of resources given its contribution to saliency was the online forum CoastExchange. This forum had 6000 sessions and 27,000 page views over its 15 months of operation (Palutikof et al. 2019a): It was reported as valuable and gaining traction by local government users (Coutts and Coutts 2017). Nevertheless, over its 14 months of operation, the volume of actual discussion on CoastExchange was low: 250 people signed up but most were ‘lurkers’ (i.e. a member of an online community who posts rarely, if ever) with few postings or comments, which is a common feature of online communities (Sun et al. 2014). However, the weekly articles provided a mechanism for awareness raising of contemporary adaptation issues, helping CoastAdapt to remain salient (and therefore supporting its legitimacy).

5.3 Ensuring legitimacy of the Framework through extensive and diverse consultation

The development team aimed to foster the legitimacy of CoastAdapt through broad geographical consultation with diverse stakeholders at NCCARF-organised workshops and in conference sessions. The online survey with around 300 responses—representative of the types of anticipated users for the Framework with respect to their location, place of work and job level—enabled remote input from anyone with an interest in coastal adaptation (Palutikof et al. 2019a, 2019b). Thirteen workshops held across seven capital cities and four regional centres (330 participants) canvassed views of attendees, including academics, state and local government officers and elected representatives, consultants and students. Where possible, consultation workshops in the scoping and usability phases were held in similar locations to enable NCCARF to report back to participants on their original feedback. Legitimacy was also facilitated through sustained engagement with the PRC, EURG, TRG TDP/TIP and NANs throughout the process.

A final ‘check’ of legitimacy (and also saliency) was through final signoff by the PRC, which signalled a commitment to CoastAdapt by state and federal agencies. Obtaining this final approval meant that the information relevant to each state or sector was relevant to that jurisdiction and therefore ‘endorsed’ as legitimate.

5.4 Using the IAP2 spectrum

Using the IAP2 spectrum helped the development team to work through the purpose of consultation and engagement activities. Most of these activities lie in the ‘inform’ to involve area of the spectrum (Fig. 1), with the PRC, EURG, TRG and TDP/TIP groups having the highest participation level (see supplementary information Tables SI-1–4). This level of engagement is common in DSF development (Valls-Donderis et al. 2014; Fig. 1). Also, while individual workshops are at the ‘consult’ level, returning to the same locations to report back on feedback and get further guidance from the same community nudged them into the involve level of higher contribution. This observation highlights the importance of considering activities as both individual and cumulative events in terms of their engagement effect.

6 Discussion

This paper reports on the user involvement process of developing and implementing CoastAdapt, and how this process contributes to the Framework being usable through being credible, salient and legitimate. Through reflecting on the process, we can make some practical suggestions that may provide some guidance on processes for user involvement in development of a DSF.

First, establishing a strong foundation for the process is essential. For CoastAdapt, NCCARF’s existing reputation, together with the pre-engagement activities of the scoping phase—researching and writing the SOP review—were important factors in building credibility. Similarly, so did the strong foundation for the content through extensive consultation of user needs, and preparation and review by technical, practice/policy and communication experts. This strong foundation facilitated on-going support from local champions and strong willingness to participate in engagement activities.

Second, the process of stepping through engagement phases provided the opportunity to tweak activities at each new phase. For example, in the prototype phase, the TDP was very effective in supporting development of salient content so that in the usability phase, this group could be expanded to become the TIP to assist roll out of CoastAdapt and therefore support legitimacy.

Third, strong and transparent governance and advisory structures build stakeholder trust and understanding (credibility). By soliciting participation in governance and advisory groups from governments, business, technical and local scale interests, NCCARF attempted to incorporate diverse views to ensure broad legitimacy and saliency. The longer-term engagement provided through these groups complemented the shorter-term workshop activities, enabling us to better understand stakeholder needs, thus contributing to credibility and saliency.

Finally, the endorsement of CoastAdapt by state and national government was a key consideration in the Framework’s perceived legitimacy. Legitimacy reflects fairness, particularly of the process used to produce the information and whether it takes account of appropriate values, concerns and perspectives of different actors. Without this formal endorsement, local councils would not be encouraged to contribute to and use CoastAdapt.

It is well established that the attributes of credibility, saliency and legitimacy are interrelated (Cash et al. 2002), and this study confirms the finding that a focus on one attribute may support or compromise one of both of the other attributes (Cash et al. 2002; White et al. 2010; Kirchhoff et al. 2013). We found that engagement activities that built credibility also fostered saliency and legitimacy. For example, facilitating multiple rounds of workshops across Australia showed a genuine commitment to work with, and integrate, diverse knowledge and disparate views of a range of coastal stakeholders. Such intensive efforts can both build credibility as well as co-produce content that is salient.

Conversely, we found that issues that affected saliency could also undermine credibility. For example, where local knowledge conflicted with information provided in CoastAdapt, the credibility of the Framework was questioned. Similarly, for remote communities where there was little or no local scale data in CoastAdapt (reducing legitimacy), users could form the opinion that the rest of the content would not be salient, despite alternative information and ‘work arounds’ provided for these situations. This demonstrates the importance of providing information within CoastAdapt that matches the experience of users and is applicable in their local context.

7 Conclusion

As the need grows to deliver effective online DSFs to support climate change adaptation, so does the need to involve potential users in their development. It is important to document and reflect on the experience of engaging with potential users to enable sharing and learning among the DSF development and stakeholder communities, including funding agencies. This paper advances knowledge practically and theoretically by outlining our approach and documenting our reflections. The IAP2 spectrum is a well-used approach in community engagement processes and therefore has traction with many of our stakeholders. Although less used in development of decision support systems, nevertheless, we found that it helped the development team as a heuristic tool for our consultation activities as individual events and also cumulatively. It also helped us sharpen our understanding of our own role as knowledge brokers.

Cash et al.’s (2002) framework of credibility, saliency and legitimacy is well-established, and helped us consider our consultation and how activities could contribute to a useful CoastAdapt. The three attributes are each important and require consideration separately and together, because activities, when carefully implemented, can simultaneously address more than one. Furthermore, future research that probed CoastAdapt users’ perceptions of these attributes within the context of the Framework would be valuable.

Finally, it is important to consider the future of CoastAdapt. The attributes of credibility, saliency and legitimacy are dynamic, and so any future improvements to CoastAdapt will need to ensure that these attributes are considered. As the adaptation community expands and matures, what is considered to be relevant is also likely to expand. However, growth of DSF content and complexity comes with increasing challenges of accessibility which, if not addressed, will undermine credibility.