For many applications of Meteorological and Oceanographic (METOC) data, there are routine sensitivities to the environment ranging from local to global scales at forecast lead times from minutes to days and weeks. Considerable effort has been undertaken in many previous projects to validate the predictive skill for various operational limits such as wind, seas, turbulence, icing, and thunderstorms for various metrics and lead-times for specific use cases, as well as the underlying skill of “state variables” such as pressure, temperature and humidity that inform these derived products. However – with the proliferation of extremely high resolution regional/mesoscale models, increasing global model resolution and accuracy, longer range forecasting from subseasonal to seasonal timescales, more advanced low order modeling, machine learning and artificial intelligence forecasts, improved statistical predictions, more robust climatologies –the best forecast for an application is increasingly unclear.
There has been no systematic effort to develop a methodology to rank the hierarchy of available forecasts or degradation of skill when the best and latest forecast is unavailable (e.g., due to forecast center outages, data dissemination issues, etc.). As such, users must currently subjectively interpret the added uncertainty due to using older, coarser, or inferior methods of quantifying the current and future predicted states of the environment. Additionally, the old “rules of thumb” guidance has not used current methodologies such as machine learning to assess what aspects of degraded forecasts are most impactful to local short-term nowcast through large scale global multi-day and week forecasts.
Objective
The FACEOFF program intends to formalize methodology and provide objective metrics to inter-compare, evaluate, verify and validate forecasts of the earth system environment. The scope for our program is atmospheric forecasts (including weather to space weather), however interdisciplinary efforts that include interactions/coupling with ocean, ice, and/or land are welcome to contact the program about potential interest. We anticipate that proposed efforts would touch on one or more of the following focus areas:
- Improve theory and formalization of objective forecast comparisons and validation across spatial and temporal scales. This may include technical and/or peer reviewed report on best practices and/or convening a workshop of experts to build a consensus on desired approaches and the state of the science.
- Develop tools and infrastructure to characterize forecast skill across multiple datasets and platforms. This software development should be able to compliment and integrate with environmental scorecard efforts at operational production centers. An example project could include development, testing, and implementation of new metrics through the National Center for Atmospheric Research (NCAR) Model Evaluation Tools (MET) software package.
- Analyze a hierarchy of model fidelities with multiple degradation scenarios for specific use cases. This may include (but is not limited to) high resolution regional dynamical models, global forecast models, longer range earth system prediction, AI/ML models, statistical techniques, analogs, climatology, persistence, extrapolation, etc.
Demonstrate prototype software data quality and uncertainty assessment tool capabilities to help refine algorithmic use of METOC forecasts based on the fidelity of available data. This may take multiple forms: as large as a post-processing suite within an operational production center to account for non-linear changes in outcomes to as small as a reference look-up table in a specific application.
Request for Planning Letters
The first step in the proposal process is for prospective investigators to prepare planning, allowing investigators to submit a short (three pages maximum) summary of their ideas on this topic for ONR to evaluate, provide technical feedback and indicate whether a full proposal would have a reasonable chance of success.
Refer to the current Marine Meteorology Planning Letter guidelines and indicate your desire to work with the FACEOFF project and team. Please note "FACEOFF Planning Letter ‘Your Last Name’" in your email subject line. If you do not receive acknowledged receipt within 10 days, please follow-up with a resend.
Important Dates
Planning letters and full proposals for the FACEOFF program will be accepted on a rolling basis.
However, note that ONR funding cycles operate on the federal fiscal year. Funding amounts and timing may vary year-to-year. Please use the following dates as guidelines for full consideration of funding into the next calendar year.
September 15: Last date to submit planning letters (please submit by e-mail to below).
October 1: Estimated response deadline by ONR to all submitted planning letters with proposal recommendation.
November 15: Earliest anticipated commencement of awards made with new fiscal year funding, depending on availability of funding and grant processing if the grant application was received by October 1.
All planning letters should be submitted by email to Josh Cossuth (joshua.h.cossuth.civ@us.navy.mil).
Interdisciplinary efforts with other areas of ONR Marine Meteorology interest may also copy planning letters to: Dan Eleuterio (daniel.p.eleuterio.civ@us.navy.mil) and Kate Mulreany (katherine.l.mulreany.civ@us.navy.mil).