About Jody Peters

Ecological forecasting is going to transform our understanding of ecology. I am thrilled to have the opportunity to help coordinate efforts to improve and move the field forward.

From Communities to Topologies, Forecasting is a Social System

January 30, 2024

Post by: Nicholas R. Record, Tandy Center for Ocean Forecasting, Bigelow Laboratory for Ocean Sciences

Does the removal of urchins and perrywinkles lead to brown water?

This question is one of hundreds of hypotheses I’ve heard over the years working with community groups on forecasting projects. It’s one of the reasons that community-centered science is fun. The ideas I encounter in communities are far more wide ranging than what’s found among scientists, who are also interesting, but are often much more in lock-step with each other (McClenachan et al. 2022).

Questions and ideas that come from communities can broaden the scientific perspective, but they can also give an indication of what’s important to that community. To me, this resonates, at a time when we know science needs to be more equitable and inclusive. It’s especially pertinent to environmental forecasting, which can have direct impacts on communities.

So how can a community’s knowledge shape a forecasting system?

Thinking about this question took me down a strange rabbit hole recently. Or maybe it was more of a complex network of gopher holes. Or a termite mound. However esoteric, I popped out the other side with some new info.

The starting point was remembering that making a forecast is more than just solving a math problem. The techniques of forecasting might be learned in a quantitative context, but a forecasting system is a social system. As a social system, there can be complex social dynamics, like reflexivity and environmental justice (Record et al. 2021, Wilson et al. 2023). There’s lots of potential for unintended consequences and other ethical pitfalls if the social dimensions are brushed over (Boettiger 2022, Hobday et al. 2019). There are plenty of cases where well-intentioned forecasts have caused harm (see the aforementioned references).

But what does this forecasting / social system look like?

Most forecasting papers are mainly quantitative, but it’s common to include a figure that diagrams how the quantitative exercise–the forecasting algorithm–fits within a social context. For example, the “ecological forecasting cycle” traces steps from hypothesis generation to model building, uncertainty quantification, forecast generation and communication, assessment and updating, and back to hypothesis generation (Moore et al.2022). The cycle diagram codifies the implicit social system, which in turn shapes what we decide to forecast and how. This particular system has communication to and feedback from groups that are influenced by the forecasts (generally managers). The cycle is taught and learned by forecasters as a way to iteratively improve forecasting systems within their social contexts.

A cycle is not the only model for diagramming a forecasting system within its social network. Because I’m curious and a bit of a nerd, I recently reviewed a collection of papers with flowcharts diagramming their forecasting systems for ecological forecasting (Figure 1). You can see the “forecasting cycle” of Moore (2022) in the bottom center. There’s a range of configurations, including pure cycles, unidirectional flows, trees, meshes, nearly fully connected networks, and combinations of patterns. Personally, I like the one in the middle row, from Petchey, that looks like a flux capacitor.

Figure 1 Some examples of topologies of ecological forecasting systems, diagrammed from the figures provided in a subset of the papers reviewed. Each figure was distilled as a network graph based on nodes and edges indicated in the diagram.

I think I used the word “esoteric” earlier, but these so-called topologies of networks can be pretty informative. The shape of a network influences the flow of information and the resultant emergent knowledge. For example, stronger connectivity leads to faster consensus, and under certain conditions, regular networks (same number of edges and nodes) can increase the probability of reaching a less biased consensus (Fernandes 2023).

Some patterns emerged from an analysis of this collection of networks. For example, diameter appears to increase with network size at a rate similar to that for cyclic networks–i.e. networks that are basically circular, like the “forecasting cycle” (Figure 2). To find the diameter of a network, you look at the shortest distances between all pairs of vertices, and take the largest of those (wikipedia explains it in more detail). In principle, larger networks don’t necessarily have to have larger diameters. Fully-connected networks have small diameters no matter how large, and other shapes (e.g. tree-like networks) fall somewhere in between. But for this group of papers, larger networks had larger diameters, which means information has to pass through many steps to get through the social network. Multiple steps can lead to situations where the information is not well connected with other parts of the system. One potential explanation for this is that as we build larger forecasting systems that include more social components, we have a tendency to overlook the importance of the connectivity of these components.

Figure 2 The relationship between network size (edges) and network diameter (greatest shortest node-node distance) across eighteen forecasting topologies (r = 0.64, p < 0.005). In comparison are lines showing what this relationship would look like for cyclic (bi-directional) networks, tree-like networks, and fully connected networks.

To put this knowledge to use, we redrew the schematics that we use for our own forecasting projects at the Tandy Center for Ocean Forecasting (Figure 3). The idea was to have a system that is fully connected, without long chains of arrows that information has to follow to get from one place to another. This exercise emerged through the process of writing this EFI blog post on barriers to inclusivity. The system has properties that aim to reduce confirmation bias and to speed consensus (Fernandes 2023)–i.e. full connectivity and regularity (equal number of nodes and vertices). In practice, the idea is to include forecast users and those influenced by forecast-based decisions–i.e. communities–as participants throughout the process. More details on this approach are in a technical document (Record 2022).

We don’t know for sure that our approach will lead to more equitable outcomes in forecasting. That’s just the working theory. There’s an important trend in ocean science (and many geosciences) developing the role of co-producing knowledge through collaborations across social systems (Liboiron et al. 2021, Schreiber et al. 2022). The schematic devised here seeks to address issues of accessibility to forecasting science and practice, though there are tradeoffs to this approach (Record et al. 2022).

Ideas like, “Does the removal of urchins and perrywinkles lead to brown water?” should be able to propagate quickly through a well-connected network and be incorporated, or not, in a forecasting system. By centering communities of people who use or are influenced by forecasts, it can help to kickstart forecasts in places that might be under-resourced or otherwise outside of the mainstream of ocean forecasting applications. It might help avoid unintended consequences of forecasts. And it should get a community’s wide range of hypotheses and ideas into the mix faster.

Figure 3 Schematic diagram of the framework used by the Tandy Center for building forecasting systems. Arrows indicate two-way movement of information between any of the components of the framework, potentially multiple times.

Do you have your own sketch of what a forecasting system network should look like? Or are you encouraged to try to make one? I suspect there are lots of different configurations that could work in different circumstances.  If folks are willing to share, I’d be happy to collect them and post in a followup blog post. You can send it to forecast@bigelow.org, with a subject heading “topology”, or share on the EFI #inclusion Slack Channel (if you are not on the EFI Slack group and would like to join, reach out to info@ecoforecast.org to be added). 

Note: Some of this content appeared in a blog post from 2023 (https://seascapescience.github.io/posts/2023/10/topologies/)

Acknowledgements: The EFI Diversity, Equity, and Inclusion Working Group provided helpful feedback on the post during the January 2024 call. Participants included: Alyssa Willson, Anna Sjodin, Antoinette Abeyta, Jason McLachlan, Jody Peters, John Zobitz, Rachel Torres, and Saeed Shafiei Sabet

References

Boettiger C. The forecast trap. Ecology Letters. 2022 Jul;25(7):1655-64.  https://doi.org/10.1111/ele.14024

Fernandes MR. Confirmation bias in social networks. Mathematical Social Sciences. 2023 May 1;123:59-76. http://dx.doi.org/10.2139/ssrn.3504342

Hobday AJ, Hartog JR, Manderson JP, Mills KE, Oliver MJ, Pershing AJ, Siedlecki S. Ethical considerations and unanticipated consequences associated with ecological forecasting for marine resources. ICES Journal of Marine Science. 2019 Sep 1;76(5):1244-56. https://doi.org/10.1093/icesjms/fsy210

Liboiron M, Zahara A, Hawkins K, Crespo C, de Moura Neves B, Wareham-Hayes V, Edinger E, Muise C, Walzak MJ, Sarazen R, Chidley J. Abundance and types of plastic pollution in surface waters in the Eastern Arctic (Inuit Nunangat) and the case for reconciliation science. Science of the Total Environment. 2021 Aug 15;782:146809. https://doi.org/10.1016/j.scitotenv.2021.146809

McClenachan L, Record NR, Waller J. How do human actions affect fisheries? Differences in perceptions between fishers and scientists in the Maine lobster fishery. Facets. 2022 Feb 10;7(1):174-93. https://doi.org/10.1139/facets-2021-0030

Moore TN, Thomas RQ, Woelmer WM, Carey CC. Integrating ecological forecasting into undergraduate ecology curricula with an R shiny application-based teaching module. Forecasting. 2022 Jun 30;4(3):604-33. https://doi.org/10.3390/forecast4030033

Record NR, Pershing AJ. Facing the Forecaster’s Dilemma: Reflexivity in Ocean System Forecasting. Oceans 2021 Nov 12 (Vol. 2, No. 4, pp. 738-751). MDPI. https://doi.org/10.3390/oceans2040042

Record N. Early Warning Systems for Harmful Algae: A Stakeholder-Centered Framework. 2022. http://dx.doi.org/10.13140/RG.2.2.24501.14568

Record NR, Evanilla J, Kanwit K, Burnell C, Cartisano C, Lewis BJ, MacLeod J, Tupper B, Miller DW, Tracy AT, White C, Moretti M, Hamilton B, Barner C, Archer SD (2022) Benefits and Challenges of a Stakeholder-Driven Shellfish Toxicity Forecast in Coastal Maine. Frontiers in Marine Science. https://doi.org/10.3389/fmars.2022.923738

Schreiber MA, Chuenpagdee R, Jentoft S. Blue Justice and the co-production of hermeneutical resources for small-scale fisheries. Marine Policy. 2022 Mar 1;137:104959. https://doi.org/10.1016/j.marpol.2022.104959Willson AM, Gallo H, Peters JA, Abeyta A, Bueno Watts N, Carey CC, Moore TN, Smies G, Thomas RQ, Woelmer WM, McLachlan JS. Assessing opportunities and inequities in undergraduate ecological forecasting education. Ecology and Evolution. 2023 May;13(5):e10001. https://doi.org/10.1002/ece3.10001

EFI at AGU 2023

December 1, 2023

Below is the list of poster and oral presentations for EFI’s hosted session at the American Geophysical Union (AGU) 2022 Conference in Chicago, as well as other ecological forecasting-related talks that may be of interest to the community. All times are listed in US Pacific Time.

EFI has name badges! EFI community members can find Mike Dietze at the Conference, during the EFI-hosted sessions, or at the Social to get a badge.

Tuesday EFI Social – Anyone who is available to meet up on December 12, Tuesday evening, we’ll have a group getting together from 7:00-9:00pm at 1015 Howard St, which is just an 11-minute walk from the Moscone Center.

Tuesday Poster and Oral Sessions – EFI’s oral and poster sessions on “Ecological Forecasting in the Earth System” will be held on Tuesday, December 12. The Poster Session is from 8:30am-12:50pm in Poster Hall A-C – South (Exhibition Level, South, MC). The Oral session is from 14:10-15:40pm in 3008 – West (Level 3, West, MC). We’re excited to have a great set of speakers that really span the full gradient from terrestrial to freshwater to marine. Come check out the following talks!

Tuesday Poster Session (9:00-12:30, Poster Hall A-C- South)

Tuesday Oral Session (14:10-15:40, 3008 – West (Level 3, West, MC)

Other Forecasting Presentations

If you are presenting an ecological forecasting-related talk or poster that you don’t see on the list, reach out so we can get it added!

EFI Book Club Suggestions & Actions Items

November 27, 2023

The following document was created by the DEI Working Group in October 2023 in response to the final discussion of the group that participated in the book club which read “97 Things About Ethics Everyone in Data Science Should Know”.  

Goals of this document:

  1. Provide a summary of action items suggested during book clubs. The summary is a living document kept on the EFI Book Club webpage and action items will be added from future book clubs. This document is to be used as a reference to support and inspire activities for EFI that can be developed/led as individuals have time, availability, and interest in doing so
  2. Provide an opportunity to collate and celebrate activities that have taken place and those activities that will take place in the future that have been inspired by or align with suggestions from the book clubs.

Books Read

Note: There was a group within EFI that read Spatio-Temporal Statistics with R. That book group was not led by the DEI working group and was not included in this assessment.

Have we done any of the stuff we planned to do?

  1. From 97 Things About Ethics
    1. Summarize ideas we’ve had in previous book clubs and do self-assessment
      1. This document!
  2. From Braiding Sweetgrass
    1. Gratitude statement regarding data as a gift from the community, posted in:
      1. Newsletter
      2. Book club page
    2. The book club included student participants from the Sustainable Land and Water Resources (SLAWR) REU which provided opportunities for discussions across career stages and sectors
  3. From Weapons of Math Destruction
    1. JEDI (justice, equity, diversity, inclusion) database assessing who is in EFI is a good place to start to contribute to the self-assessment
      1. Dave Klinges (University of Florida) has worked with EFI Community Manager, Jody Peters (University of Notre Dame) to develop a workflow to assess the demographics of EFI members and how that changes through time
  4. From The Years That Matter Most and from Caste
    1. DEI Workshop ideas – webinar series leading up to workshops. Whose voices do we want to hear ahead of time
    2. Culturally competent training
      1. In late 2020/early 2021 the EFI DEI working group hosted 4 panels for the “Conversations on Diversity and Data Seminar Series.” See the links below for the recordings for each panel and the resources shared.
      2. What do diverse students need to thrive in data- and computer-heavy settings? November 19, 2020
      3. What does culturally competent data science training look like? January 20, 2021
      4. How can 4-year data-driven institutions and programs support students transferring from community colleges? February 19, 2021 
      5. What kind of online tools are useful and work well for remote learning and teaching? March 25, 2021 
  5. From Caste:
    1. Mentorship networks and having a series of mentors
      1. EFISA co-working session
      2. Developing a mentoring network for minority students is one of the goals of the Alfred P. Sloan Foundation Equitable Pathways funding from the project “A Generation of Native American Earth and Environmental Data Scientists” that EFI is involved with
        1. Still working on developing this, but the idea is for BIPOC students in research to have mentors from their own institution as well as mentors from the institutions they are doing research with and that the mentors will continue to support the students beyond the original activity
  6. From the Years That Matter Most
    1. Train the trainers
      1. This is something that continues to be included in EFI-led proposals.  Freya Olsson is also thinking about updating her NEON Forecasting Challenge workshop material which is currently geared towards participants wanting to learn how to submit forecasts to the Challenge to be applicable for people want to teach others how to submit to the Forecasting Challenge
      2. On the October 2023 Education working group call, the group discussed the following two papers. The first is about the needs for instructor training for people teaching data science at the collegiate level. The second provides papers, discussion prompts, and R code materials that can be used or modified for teaching ecological forecasting at the upper undergrad/early grad student level
        1. Emery et al 2021: Instructor Training Needs for Data Science
        2. Ernest et al 2023: Ecological Forecasting & Dynamics Course
    2. Highlight the #jobs and #funding Slack channels in the newsletter and in the welcome letter for new EFI members.

Do any of the ideas overlap between books? 

  1. The Years That Matter Most, Caste, and 97 Things: mentorship, supporting involvement, inclusive trainings
    1. Ongoing EFI efforts with potential overlap:
      1. EFISA efforts to share grant applications
  2. Braiding Sweetgrass, Weapons of Math Destruction, and 97 Things: addressing unintended consequences of forecasts and taking time for self reflection
    1. Ongoing EFI efforts with potential overlap:
      1. Examining EFI efforts from the lens of Design Justice Principles
      2. Social scientist-ecologist matchmaking

A full list of the notes and recommendations compiled from the book clubs can be found in the living document on the EFI Book Club webpage.

Congratulations to Jacob Zwart on the 2023 ESA Ecological Forecasting Award!

The ESA Statistical Ecology section presented the 2023 Ecological Forecasting Outstanding Publication Award to Jacob Zwart and collaborators for their 2022 Journal of the American Water Resources Association paper:

“Near-term forecasts of stream temperature using deep learning and data assimilation in support of management decisions”

The award committee was impressed by the novel methodological contributions of the paper, in fusing machine learning with traditional data assimilation approaches. The committee also values the authors’ ability to put this forecast into operations, to tie a novel forecasting approach to actionable real-world decisions, and the overall readability and approachability of what is otherwise a very technical paper.

Nominate Papers for the 2024 Award

Individuals wishing to nominate papers published in the past  3 years for the 2024 award are encouraged to do so by the March 1, 2024 deadline. Additional information can be found at https://www.esa.org/stats/awards/ecological-forecasting-outstanding-publication-award/

Full List of Award Winners & Citations

2023 – Jacob Zwart (USGS)
Zwart, J.A., Oliver, S.K., Watkins, W.D., Sadler, J.M., Appling, A.P., Corson-Dosch, H.R., Jia, X., Kumar, V., and Read, J.S. 2023. “Near-Term Forecasts of Stream Temperature Using Deep Learning and Data Assimilation in Support of Management Decisions.” JAWRA Journal of the American Water Resources Association 59 (2): 317–37. https://doi.org/10.1111/1752-1688.13093.

2022 – Kelly Heilman (University of Arizona)
Heilman, K.A., Dietze, M.C., Arizpe, A.A., Aragon, J., Gray, A., Shaw, J.D., Finley, A.O., Klesse, S., DeRose, R.J., & Evans, M.E.K. (2022). Ecological forecasting of tree growth: Regional fusion of tree-ring and forest inventory data to quantify drivers and characterize uncertainty. Global Change Biology 28(7):2442-2460 doi.org/10.1111/gcb.16038

2021 – Sarah Saunders (National Audubon Society)
Saunders, S.P., F.J. Cuthbert, and E.F. Zipkin. “Evaluating Population Viability and Efficacy of Conservation Management Using Integrated Population Models.” Journal of Applied Ecology 55, no. 3 (2018): 1380–92. https://doi.org/10.1111/1365-2664.13080.

2020 –  Paige Howell (USGS)
Howell, P.E., B.R. Hossack, E. Muths, B.H. Sigafus, A. Chenevert‐Steffler, and R.B. Chandler. “A Statistical Forecasting Approach to Metapopulation Viability Analysis.” Ecological Applications 30, no. 2 (2020): e02038. https://doi.org/10.1002/eap.2038.

2019 – Maria Paniw (CREAF, Ecological and Forestry Applications Research Centre)
Paniw, M., N. Maag, G. Cozzi, T. Clutton-Brock, and A. Ozgul. “Life History Responses of Meerkats to Seasonal Changes in Extreme Environments.” Science 363, no. 6427 (February 8, 2019): 631–35. https://doi.org/10.1126/science.aau5905.

2018 – Quinn Thomas (Virginia Tech)
Thomas, R.Q., E.B. Brooks, A.L. Jersild, E.J. Ward, R.H. Wynne, T.J. Albaugh, H. Dinon-Aldridge, et al. “Leveraging 35 Years of Pinus Taeda Research in the Southeastern US to Constrain Forest Carbon Cycle Predictions: Regional Data Assimilation Using Ecosystem Experiments.” Biogeosciences 14, no. 14 (2017): 3525–47. https://doi.org/10.5194/bg-14-3525-2017.

Transporting Models Between NEON and non-NEON Systems

September 5, 2023

Brendan Allison1, Olufemi Fatunsin2, Jeffrey Mintz3

1University of Texas, Austin, 2University of Alabama, 3University of Florida

AS NSF NEON data becomes more prominent in forecasting and other forms of ecological modeling, these models may potentially become products in themselves, distilling some important characteristic of this continental-scale network. A natural follow-up question as we seek to use not only the raw data but these derived products is that of model transportability. In this project group started at the 2023 Unconference, we asked: 

1) How can we take models trained on NEON data and refine them for use in another context?

2) How can we take models trained on non-NEON data and refine them on NEON data?

Doing this effectively can empower a range of applications, including local field studies, adaptive management, and data fusion from multiple monitoring networks, enabling greater statistical power for big ecological questions. We realized that, whether transporting a model to or from NEON, the vast majority of challenges are the same. These included unbalanced data, different monitoring protocols, different predictors, and different site selection criteria. As anyone who has fused multiple datasets together can tell you, even ostensibly identical data products can differ in both subtle and dramatic ways. Naturally, models trained on one data product will inherit the distinct characteristics of the monitoring networks they are derived from. There is nevertheless remarkable potential in being able to leverage what are in many cases continental-scale models for anything from fusion with another continental-scale network to an informed prior for a small-scale field study, dramatically increasing statistical power.

In framing the problem, we found it helpful to consider similarities with longstanding efforts to effectively downscale global climate models to local forecasts. Here, the two main classes of approach are dynamic (re-running a tuned version of the full model with nested local components, ensuring that known physics are respected) and statistical (examining correlations between global climate model outputs and local history, with the benefit of much faster performance). For the greatest flexibility, including compatibility with black-box machine learning approaches with unknown dynamics, we took inspiration from the statistical approaches.

Our only hard requirement as input for the modeling process was thus an existing model that can make predictions for an arbitrary set of sites, given a set of predictor covariates for these sites, alongside the ability to measure the true value of the predicted variable at these sites. From here, we can subtract prediction from true value, generating a dataset of residual errors. This sort of additive bias correction is not the only approach to the problem, but it is simple and effective. Any skill in predicting these errors represents an improved model: simply take the sum of the original model and the bias correction term to make new forecasts. At the same time, one may examine the various machinery of the residual error model alone to learn something about the gaps and biases in the original model. Allowing models to be incrementally improved in this fashion will enable teams to improve on existing models by tailoring them to their particular ecosystem of interest and the data they have at hand.

Case Study

To focus our efforts, we picked the particular case study of bringing together NEON forest data with similar datasets generated under the Forest Inventory Analysis (FIA) program. Because FIA has multiple hundreds of thousands of plots measured across decades of operation, while the more recent NEON network offers more intensive monitoring within a smaller set of sites, both networks have the potential to bolster one another. Our basic setup was to train a model predicting forest productivity at FIA sites from a small set of environmental covariates. Though we would train this ourselves, we would not take advantage of our knowledge of its structure or details. Instead, it serves as a stand-in for any existing black box model, which in some sense represents the distillation of a large body of FIA data. Typically, it would be a model generated by some other research group or forecasting team, which we seek to leverage in our own work. Using predictions from this FIA-trained model, we calculate a set of error residuals at both NEON and FIA sites, and ask: can we leverage this for a more robust multi-system model?

Our first product was the development of a draft Bayesian multilevel model that would be equally capable of integrating multiple sets of continental or global-scale monitoring networks as it would be of making predictions at a single site of interest, but informed by some larger statistical structure. We show our first/guiding version of this in the figure below; it changed a little over development, but the basic idea remained. Hierarchical structures such as this are commonly described as partial pooling approaches. This is because FIA and NEON systems are not assumed to operate under the same model. Nor are they assumed to be fully independent. Instead, they will flexibly share a set of data-driven parameters, in this case through the top layer of a global forestry model, while having the flexibility to stray a little from this global model to account for system-specific differences. 

Figure 1 represents our original version of a partially pooled model. At the top layer, we have a global scientific model. This informs system-specific models in layer two, which then inherit ecoregion in layer three (eco_i) and plot-level parameters (p_j) in layer four. The bottom layer are the multiple observations at each plot. In this case, observations are not biomass or other direct data products, but residual error (e_k) after subtraction of predicted from actual values by some black-box model. The main modification to this model was to place the NEON and FIA models alongside the ecoregion layers as an additive source of variance, rather than nesting ecoregions inside the respective system models.

To account for spatial autocorrelation, we briefly considered Gaussian processes, but settled on building ecoregions into the model hierarchy for two main reasons. The first is that geographic distance is different from ecological distance: a mile might take you from prairie to mountaintop in one case, and from prairie to prairie in another case. The second is practical: because Gaussian processes and related kernel methods tend to scale poorly with increasing data, they tend to require extensive approximations for the fusion of multiple continental-scale datasets.

As mentioned, while this was our inspiration, we did make a few changes. The main change was to dial back on the nesting. In the above diagram, ecoregions are nested inside continental-scale monitoring networks, such that Level 3 – Ecoregion 1 – NEON plots are allowed to have different predictive models than Level 3 – Ecoregion 1 – FIA plots. This could be useful and true, but we realized that we were most interested in 1) the global forestry model at the top of the hierarchy, 2) any global differences between the types of sites or measurement protocols of NEON and FIA, and 3) any Level 3 Ecoregion variations in model structure. Thus the minor combinatorial explosion in parameters of nesting ecoregions inside monitoring systems seemed a cost without a corresponding benefit in scientific lessons learned. We decided to bring both ecoregion and monitoring system variables to the same level of the hierarchy. 

So far, the model has been tested on synthetic data, with code developed in Stan, using the CmdStanR library in R. Once we’ve had the chance to hone the model against real data, we’ll push all the code to a NEON repository for the community to begin playing around with it. From there, some version of the project is likely to continue and become a more formal manuscript. We may also reintroduce some complexity to the model in the form of nested ecoregions. 

With that said, we’d be interested in hearing from other members of the community. How to integrate NEON data with that of other monitoring networks, how to use it to inform local field studies, and how to leverage it for management goals at non-NEON sites are all open and important questions. What potential applications do you see? Are you working on similar projects?

Contact us at info@ecoforecast.org to share your interests in the project or similar work you are doing!

Reenvisioning EFI-RCN NEON Forecast Challenge Dashboard Visualization

August 22, 2023

Melissa Kenney1, Michael Gerst2, Toni Viskari3, Austin Delaney4, Freya Olsson4, Carl Boettiger5, Quinn Thomas4

1University of Minnesota, 2University of Maryland, 3Finnish Meteorological Institute,4Virginia Tech, 5University of California, Berkeley

With the growth of the EFI NEON Ecological Forecasting Challenge, we have outgrown the current Challenge Dashboard, which was designed to accommodate a smaller set of forecasts and synthesis questions. Thus, we have reenvisioned the next stage of the EFI-RCN NEON Forecast Challenge Dashboard in order to facilitate the ability to answer a wider range of questions that forecast challenge users would be interested in exploring. 

The main audience for this dashboard are NEON forecasters, EFI, Forecast Synthesizers, and students in classes or teams participating in the Forecast Challenge. Given this audience, we have identified 3 different dashboard elements that will be important to include: 

  1. forecast synthesis overview, 
  2. summary metrics about the Forecast challenge, and 
  3. self diagnostic platform.

During the June 2023 Unconference in Boulder, our team focused on scoping all three dashboard elements and prototyping the forecast synthesis overview. The objective of the synthesis overview visual platform is to support community learning and emergent theory development. Thus, the synthesis visualizations are aimed at creating a low bar entry for multi-model exploration to understand model performance, identify characteristics that lead to stronger performance than others, the spatial or ecosystems that are more predictable, and temporal forecast validity. 

You can view the prototype developed during the meeting HERE and in Figures 1 and 2. 

Figure 1.  Static image of an interactive map of aggregate forecast skill relative to climatology at each forecasted sites, here showing the water temperature forecasts for the aquatics theme. Bubble colour represents the continuous rank probability score (CRPS) skill relative to climatology with positive values (blues) showing submitted models on average perform better than climatology and negative values showing submitted models perform worse (reds). The size of the bubble represents the percentage of submitted models that outperformed the climatology null (i.e., larger sized bubbles have a higher percentage of skilled models). When hovered over, the bubbles show this percentage (perc_skilled), the site type (field_site_subtype), as well as the total number of models forecasting at that site (n_mod). 

Figure 2. a) Percentage of submitted models that are classed as ‘skillful’ (outperform the null climatology forecast based on the continuous rank probability score metric) at the river (n=27) and lake sites (n=6) for water temperature forecasts at each horizon from 1 to 30 days ahead. b)  Percentage of submitted models that are classed as ‘skillful’ for water temperature forecasts at six of the lake sites (https://www.neonscience.org/field-sites/explore-field-sites). 

Developing these graphics requires aggregation of skill scores. There are a multitude of metrics that can be used to calculate the skill score, which each have their own benefits and flaws. Thus, there should be multiple skill scores for different metrics with clear presentation of what metric is used at a given visualization. Additionally, in order to isolate what sites are more interesting from a model development perspective, there needs to be a comparison of how many of the models meet a baseline skill score at a given site at a chosen time frame. That allows isolating challenge areas and also easily informs which models really succeed at situations where others struggle. For better future analysis of how models perform at certain sites, we also envisage the visualization to include the skill scores for the relevant drivers (NOAA weather) for comparison. For example, if we see a drop in skill across models in water temperature projections after some time, there should be a direct method to assess if this reflects overall flawed model dynamics or if the weather forecast driving the water temperature loses its reliability. This also allows the user to approximate a maximum length in which the model performance analysis is at all useful.

In addition to the main synthesis overview, the goal of this platform is to support exploration of synthesis data. For all themes, there was general agreement that it would be useful to pull up at a glance, site characteristics, a photo, and basic summary statistics about the number of models and model performance. 

During the meeting, we worked with the Aquatics and Beetles Challenge teams to identify some of the key data aggregation groupings that will be important to facilitate exploration. One important distinction arose during the conversations – the baseline model, time scale, and data latency.  For Aquatics there is a long time series of data that create a climatology and data are provided relatively quickly via data loggers. For Beetles, there is a different null baseline model given the length of historic data that is different at each site and it takes a year to provide beetle abundance and richness assessment. There was also a desire to have specific types of synthesis visualizations including the species accumulation curve over years, 3-year running average, and indicating the lower and upper bounds of a particular variable (use in scale). Thus, for both Beetles and Aquatics there are similarities and differences in the types of groupings that would be most useful to support synthesis exploration. 

Table 1. Different data groupings that would be useful to facilitate easy-to-develop synthesis visualizations of the EFI-NEON Forecast Challenge models to facilitate learning and community theory development.

GroupingsAll ThemesAquaticsBeetles
Team / Challengetheme, site, model ID, customized classroom or team groupingsparticular variables (e.g., DO) within a theme
Spatial / Ecosystemssites, NEON domains, site type (river, stream, lake…), altitude (high vs lowlands)sites by distance, dominant NLCD classification
Temporal Scale average for past year, seasonal groupings,1 day, 5 days, 7 days, 15 days, 30 days14 days, growing season, multi-year (up to 5 year) forecasts
Modelsbest model at each site, model inputs, model structure, functional type, output uncertainty representationmodel run time, model computational requirements
Skill Scoringcurrent skill forecast approaches, better than climatology/null baseline, comparison of your model to the best forecast
Other Featuresenvironmental variables and weather forecast observationscomparison with weather/climate forecast skilldisturbance events (e.g., widlfire), growing season dates at each sites, site disturbance characteristics (e.g., mowing, fencing)

In addition to the synthesis overview, there were two complementary and linked platforms that will create the dashboard.  First, the objective of the forecast challenge overview is to provide a basic summary of metrics related to the overall EFI NEON Ecological Forecasting Challenge. Specifically, the metrics that would be included are: number of forecasts submitted, number of unique teams, percentage (or median of all) of models that are better than climatology or a null model per theme, and total forecast and observation pairs.

Second, the objective of the self-diagnositic platform is to provide an overview for individuals or team forecast contributions and performance. The types of summaries that will be provided for the forecasters are: confirmation of forecast submission, date of the most recent forecast submitted for a model, model performance relative to climatology or null model, model prediction versus observation, model performance vs other selected models, and model skill over a specific time horizon (to assess whether it performs better over time).

Overall, the goal of the re-envisioned visual dashboard is to create platforms that will allow us to track challenge engagement, individually or as a team diagnose any model submission problems and performance improvement opportunities, and support community theory development through a synthesis given the range of models submitted through the EFI NEON Ecological Forecasting Challenge.  Long-term, if this platform structure is useful and robust, it could be applied to other systems where there are multi-model predictions and there is a desire to collaboratively learn together to improve our theoretical understanding and forecasts to support decision-making.

We are looking for input from the EFI community on the synthesis dashboard for other themes, to discuss with individuals what synthesis would be most relevant to phenology, terrestrial, and ticks forecasters. Reach out to info@ecoforecast.org to share your thoughts or let us know you would like to join future conversations about updating the dashboard.