“Summer Rage” at the International Energy Program Evaluation Conference
-By Tim Guiterman
Earlier this month, I was invited to speak on a panel at the International Energy Program Evaluation Conference (IEPEC) in Long Beach, CA. This conference is held in the U.S every two years, and brings out the leading practitioners in the evaluation, measurement and verification (EM&V) field. The panel I spoke on was titled “Survival of the Fittest: Data Analytics and Evaluation.” The panel was designed to draw out the tension between data analytics and traditional evaluation. For those of you following our industry, this is a hot and somewhat divisive topic these days. I joined three other panelists including Dan Violette of Navigant, Ken Kolkebeck of First Fuel, and Ryan Bliss of Research Into Action.
Leading up to the IEPEC, a former colleague of mine told me that every summer, something in the energy efficiency industry becomes the “summer rage.” This summer’s rage appears to be EM&V 2.0. In June, Greentech Media published an article tackling challenges with measuring and tracking energy efficiency savings, and I wrote a follow-up response to that in my last blog post. I spoke with many in the industry after the article broke, including evaluators, utilities, regulators and advocates, and while there was some disagreement about the tone of the article, there appeared to be a consensus that the industry needs to address the core issues in the article. The article sparked a vibrant discussion around the challenges of measuring energy savings, touching the EM&V field as well as new technologies under the umbrella of EM&V 2.0.
So the IEPEC proved to be a perfect place to bring this all together, and I was honored to be a part of it. Each of the panelists provided a brief presentation on their companies, roles and perspectives on this important issue. Below is what I had to say:
I’m here today to offer you a vision of how we as an industry can adapt to the challenges of achieving energy savings, by using cloud computing and data analytics to unlock the power of measurement and provide continuous, actionable feedback on program performance.
Why am I talking about this? Because we can all acknowledge that the way things are done today results in real problems and frustrations for those designing, managing and approving energy efficiency programs.
The way we assess program performance is through a regulatory compliance mechanism, EM&V, and while this does a great job serving that function, it is less well-suited to providing the kinds of deep, timely and actionable feedback needed to continuously monitor performance, take corrective action and enhance and improve programs. And many in this field want to begin to understand what’s really happening at the meter, what’s happening in every premises, and dive into reams of data to measure and value the savings that our industry delivers.
These quotes represent some of the challenges we hear, and that we’re trying to address:
So how does our savings measurement software work? In a nutshell, we’re leveraging established M&V protocols to perform a continuous, automated billing analysis, across every project in a program, and using large comparison groups of non-participants to control for population-wide effects (See Figures 2 and 3).
Figure 2. Billing analysis for all projects in a program, in an on-going manner, with comparison groups.
Figure 3. Savings measurement software: how it works.
We’re taking the data from customer meters, either monthly or AMI, residential or small/medium businesses, with a focus on programs where we can (1) tie savings to a meter, (2) there’s enough participants and non-participants to build robust comparison groups, and (3) the savings are at least 2-3% of total usage, so we can see the signal from the noise. The results of the analysis feed into a web-based dashboard which provides program managers and interested stakeholders a real-time view of program performance and deep, granular insights into drivers behind that performance.
So imagine a world in the not-too-distant future, where every project in every program in every year is continuously monitored and the savings are measured at the meter. What does this mean for this industry? Well, at EnergySavvy we see a world where we have continuous program improvement based on a steady stream of data analytics from the grid edge, providing key performance indicators such as contractor metrics, insights into savings achieved by specific measures or measure groups, location, vintage, etc. and factors driving your program towards success as well as those moving the program farther from its goals (See Figure 4). These analytics unlock the power of measurement to turn quantitative analytics into qualitative findings and actions, and this internal monitoring function is most applicable to utilities in the audience today.
Figure 4. Identify factors of both positive and negative influence on performance.
And what does this mean for external, 3rd party evaluation? In its simplest form, it’s integrated EM&V with substantially more volume and granularity of data that can inform a responsive, targeted evaluation, focused on reducing uncertainty and investigating areas of concern uncovered through the continuous monitoring of the program. Streamlined and cleaned data can help reduce burdensome data requests to the utility and potentially reduce costs and/or leverage more value-added work for evaluators.
One one hand, evaluation serves a regulatory function to ensure that programs are meeting statutory requirements. On the other hand, utilities and program administrators would like to use evaluation to inform continuous program improvement. Program managers are clamoring for evaluation to serve a developmental function and help programs improve.
Brandy Brown, “Evolution of EM&V: Moving Towards a Systems Design,” 2014 ACEEE Summer Study
But will it replace EM&V? The simple answer is no, savings measurement software will no sooner replace 3rd party evaluation than QuickBooks displaced financial auditors. Will certain programs move away from assessing impacts through deemed savings and towards continuous billing analysis, with savings measured at the meter? Sure. And they should.
So to bring all this back, where does this leave us now?
In California, we have a push towards a rolling portfolio where what counts is timelier, actionable feedback. And utilities, commissions and stakeholders are clamoring to roll EE into grid operations, and the grid requires measurement of reductions from the grid edge, not proxy algorithms.
Nationwide, we just got the most profound ruling on reducing carbon emissions in US history handed down last week. EM&V is no longer utility by utility, but will be checked at the state and federal level. Think about that for a second: EM&V is going to be used to calculate how we’re doing relative to climate change. So now it’s no longer about measuring the impacts from a light bulb or a program, but about moving towards valuing EE in a new market for emission credits.
In conclusion, my request is this: As trusted advisors to utilities, help your clients understand how EM&V 2.0 fits into the current and future paradigm, and help them understand the importance of continuous monitoring and how this can enhance, complement and not conflict with third-party evaluation.
That concluded most of my presentation. As you might guess, calling EM&V 2.0 the summer rage seems over the top. Rather, EM&V 2.0 is an additional tool in the toolbox for utilities, evaluators, and regulators.
At the end of my presentation I challenged the audience to re-convene this panel in two years at the next IEPEC. I asked everyone to come prepared with fully fleshed out case studies demonstrating the power of EM&V 2.0 to yield deep insights, actionable feedback and “near real-time,” responsive/developmental evaluation. I am confident that the summer rage will quickly die down, and long before my next presentation at IEPEC, EM&V 2.0 will be the norm, and our industry will find these tools indispensable.