On the governance of models and data in the Brazilian electricity sector
Despite believing that the most relevant step towards solving a large portion of problems in the Brazillian power sector is more related to the gains in efficiency that moving towards a market design would bring, there is a tremendous amount of room for improvement in terms of the governance of models and data in the electric sector before doing that. And before implementing market mechanisms in the short term, these improvements are sine qua non for establishing a good environment for a smoother transition.
So, regarding the current way Brazil operates its generation fleet, most of the distortions observed in the sector's variables today - from the price that is extremely volatile and does not capture important signals in advance to inconsistent resource utilization (evidenced by frequent out-of-merit dispatch) - are amplified or partially originated by the optimistic bias that the official long-term hydrothermal dispatch planning models, called Newave (NW) and Decomp, and its data lead to the short-term model (called DESSEM). The short-term model receives the value function (cost-to-go function from a stochastic dual dynamic programming model) calculated through NW+Decomp with many simplifications.
By simplifying the view of the future within NW and Decomp, the value of the flexibility that water will bring to the future operation does not appear to the National System Operator (ONS, in Portuguese) in the short-term model DESSEM that schedule all the units for the next day. Thus, the ONS is induced to schedule more hydro generation than it should and spend the water even when it is clear to any agent in the market, even to the ONS team, that this resource needs to be conserved. The optimistic bias in the view of the future about the availability of water resources inevitably leads to risky and dangerous states of the system, as the costs are very asymmetric (deficit costs during months vs spillage and perhaps some cheap thermal generation - don't want to oversimplify here, but the costs are clearly asymmetric).
Strong and practical evidence of this detrimental mechanism of planning oversimplifying the future and then operating a much more complex system using the water opportunity cost (water value) from this simplified model manifests itself in pre-generation out-of-merit order (GFOM) situations. In these moments (which is still alive in our nightmare since the second semester of 2021 and which imposed a billionaire bill with unbelievably-expensive merchant thermal plants - not saying that the decision, given what happened, was wrong yet perhaps not well formatted in legal terms), it becomes a possibility that a more realistic (or even pessimistic) view of future inflows and system constraints should lead to more frequent and flat thermal dispatches, say, with cheaper gas-power plants, in order to avoid the exorbitant costs of activating all thermal plants, some of which costing more than 1000 R$/MWh.
Note that, at this point, various fronts that seek to justify, and in some instances even impose, firm gas dispatch could find an echo in the very nature of the potential benefit that these thermal plants could bring "within" the corrected merit order (dispatch using a more detailed and realistic view of the future). However, with the current chain of models in use, we are not able to measure such benefits. This is because the long-term models, NW and Decomp, are not able to
1) capture many of the benefits that hydropower brings in solving intraday inflexibilities (see reference 6, at the end of this post, for the case of simplifications in the network models) and
2) receive unrealistic projections about future water inflows.
As such, the value that the cheaper thermal plants bring by ensuring higher storage levels for the future is not fully captured. So, it is very likely that the intuition most of the market participants share of what would happen if we had more realistic view of the future and what is fair to expect given the knowledge about the operation of hydrothermal systems, such as more firm dispatch with gas, filling of reservoirs so that they operate in regions of greater efficiency, and greater valorization of reservoirs that bring important flexibilities to circumvent almost all the short-term constraints and the intermittency of renewables in real-time, should happen through the model, i.e., would become the new merit order by the improvements in the NW and Decomp.
However, there is little or no data-based study of actual operation that demonstrates the adherence between NW projections for a few months to a year ahead and what was actually done. And this is of great importance. This study is not simple, and for this reason it is not done. However, it is not, by any means, impossible to be carried out, and it is the role of the entity that chooses the operational model to carry out this monitoring in order to account to society about the goals that are being set and what has actually been accomplished. It is worth noting that what NW produces is an operation and water resources use planning. This planning has a direct impact on short-term operation. Therefore, corporate management best practices apply in this case. We should monitor the compliance of goals and plans so that we can correct the misuse of resources and operate better.
In the video below, my PhD student (Joaquim) and I show how this problem of simplifications affects the price and systematic emptying of reservoirs just by ignoring the electric restrictions in the calculation of the value of water within NW. My presentation puts the theoretical foundations to understand this problem (t=00:10:40) and then Joaquim shows a case study with real data from the Brazilian electric system (t=1:00:40). Video in portugues (perhaps automatic captions work?): https://www.youtube.com/watch?v=rX94PCS9eF8&ab_channel=P%26DNorus
To complement the point, in the charts below extracted from the presentation, we can see the effect of planning without the network and operating with the network (red line). This is what we do today. The electricity spot prices become much more volatile and higher and the reservoirs emptier. The interesting thing is that if we consider the long-term network model, the spot prices and the reservoir adapt and return to safe levels. But this is only for one of the simplifications (network model, which is currently simplified to only maximum energy transfer capacity of a fictitious copper plate network disregarding Voltage Kirchhoff law, losses, end the whole topology.
Figure 1 - Graphs extracted from Workshop "Computing Efficient Prices" (Rede means Network, N-1 means all the network and n-1 constraints, s/rede means no network or very simplified copper plate).
In 2021, I started to supervise Arthur Brigatto's Ph.D. thesis in a broader context (See the thesis summary - only in Portuguese). So, in this broader context, the same problem occurs due to the optimistic bias that the inflows model brings. In a highly simplified study, we calculate the average error between k-step-ahead forecast - reality, for k = 1 to 24 months (main periods that affect the calculation of the value of water in NW). We were able to assess through a rolling horizon forecasting study that on average, there is an optimistic bias of +/- 15% in the southeast for 12 steps ahead and an even greater bias in the Northeast in the last 10 years. This means that NW, since the end of 2011, the beginning of the rolling horizon, informs through optimistic water values to the short-term model DESSEM that it will receive 15% more water (on average) than it actually receives in the following year in the main subsystem.
Below: In solid lines, the historical natural energy inflows (inflows transformed into energy through another very questionable procedure, which we keep for another text) for a given fixed configuration of hydros cascades (July 2022) and in percentage of the long-term seasonal average. So, 100% means that the inflows were equal to the long-term average of that month. In dotted lines, the 12-step-ahead projections from each date (averages of the 2000 series of the deck of each date from 2011 to 2022). Sudeste = Southeast, Nordeste = Northeast. PARp stands for the periodic autoregressive model, and PARpA means PARp with annual (last 12 months average) component (another interesting topic to discuss as it is a particularization of a 12-month regular PARp).
Below, the average bias for each k step ahead forecast horizon in the rolling window out-of-sample evaluation.
These many optimistic approximations generate a value of water that is systematically lower than it should be. And given the complexity of the correctly characterized constraints in the short-term model, DESSEM, will tend to give preference to using water today (because the signal that the long-term model gives to the short-term model is that the future will be gentle and easy), even though everyone sees that this is completely wrong. This has the potential to lead to dangerous states that demand out-of-merit (out of the market) dispatches and pricing, which is only one dark side of many others that affect the very short-term prices and the hedging costs until the expansion of the system and its attractiveness.
References:
I would like to highlight the following articles for documentation on the topic:
1) An article on the economic value of this problem and presenting a possible improvement in incentives for entities in the sector to create more diligent governance of models and their data:
2) The contribution made to public call 121 of the MME https://www.dropbox.com/s/gtugvkgf8a0tqgg/Contribuicao%20Street%20CP%20CPAMP%202022-enviada.pdf?dl=0
3) In an interview with the energy channel, I present my personal opinions on the problems that I see existing in the governance of models: https://www.dropbox.com/s/7uxzasoxbmomxgt/Cepel%20come%C3%A7a%20a%20trilhar%20seu%20futuro_CanalEnergia%20-%20corrigido.pdf?dl=0
4) A recent publication in the newsletter of the stochastic programming society, where we raised this topic among big names and the discussions led to an invitation to write this message in the society's newsletter. It can be accessed on page 14. https://stoprog.org/sites/default/files/sps_newsletter_2.pdf
5) Article of SNPTEE about studies of the effects of transmission on prices and generation: https://www.dropbox.com/s/ah8s87rb1uzrkan/SNPTEE2017_Inconsistencia%20e%20PDDE%20com%20n-k%20-%20v20.pdf?dl=0
6) Recent paper on the inconsistency due to simplifications in the network model: https://www.dropbox.com/s/yebibp7712qalb6/Transactions_Andrew_Rosemberg_2020_final.pdf?dl=0
Recommendations:
For the very short term:
What we don't measure we don't control, so, the ONS should implement public monitoring dashboards to track the forecast errors based on actual observations, in both inputs (e.g., inflows, demand, etc.), and outputs (e.g., thermal and hydro generation per area, operation cost, prices, etc.).
For not that far in the future: Define clear performance targets, as well as consistency and agility in responding to crises, for the ONS, which should be held accountable for not meeting these goals in the same way that any CEO of a company would be held accountable for not meeting the business plan and strategic goals. Additionally, the role of the Permanent Committee of Model Monitoring (CPAMP) should be reevaluated, with a focus on creating incentives for effective governance of models and data. This could include the creation of a council made up of industry and academic professionals who are knowledgeable and impartial in the technical and regulatory aspects of model management. Additionally, the current governance of models and data within the ONS should be reevaluated, and changes should be implemented to improve its performance. Removing the obligation for the ONS to use the Newave, Decomp, and Dessem models and allowing them to choose the best model for their needs would create a healthy competitive pressure and allow the ONS to better meet its performance goals.