How close are urban scale building simulations to measured data? Examining bias derived from building metadata in urban building energy modeling
Introduction
With residential and commercial buildings in the United States contributing 39% of carbon emissions, 73% of electricity use, and 80% of electrical demand in 2020, buildings are the most significant opportunity for sustainability. Urban building energy modeling (UBEM) is a growing field within building energy modeling (BEM) for which thousands to millions of buildings are modeled to maximize modeling impact. Creating representative building energy models allows one to understand how different building types, systems, and technologies will perform under various conditions throughout a country. Modeling individual buildings has the potential to enable actionable decisions at the building or portfolio level, while stimulating private sector activity to refine such models toward investment grade status required for financing building retrofits. Modeling individual buildings at city- to nation-scale has the potential to impact energy, demand, emissions, and costs while informing coupling of buildings with the electric grid for resilience. Such building-specific models can be mapped to feeders, substations, or other critical infrastructure and constitute valuable tools for a utility to assess load throughout their service area and facilitate load shaping to cleaner or cheaper generation assets.
There is significant complexity when developing a representative building energy model as no two buildings perform exactly the same. Some of the bias that is attributable to the unique energy performance of each building can be eliminated through simulation adjustment or calibration to measured data, but performance data is rarely available for each building. For this reason, it is valuable to understand and quantify the sources of inherent bias for uncalibrated UBEM. Comparing measured and simulated (predicted) annual energy use intensity (EUI) is useful as it allows buildings of all sizes to be compared and better understand sources of bias. While previous UBEM has validated individual buildings for hundreds to two thousand buildings against monthly or annual data, this paper validates 50,843 building energy models against sub-hourly (15-min.) measured data and analyzes systemic bias.
This analysis investigates bias and the sources of bias in Automatic Building Energy Modeling (AutoBEM) [1], a UBEM framework that has been used to model hundreds of millions of buildings since it was developed including the creation and sharing of a model of 122.9 million buildings (97.8% of the United States) [2]. In one analysis, AutoBEM was used to model nearly 180,000 buildings of the Electric Power Board of Chattanooga, Tennessee (EPB). In close partnership with this utility, related research includes validation against measured data [3], [4], estimates of building retrofit technology savings [5], [6], and forecasting of building energy use under various climate scenarios [7].
AutoBEM is a bottom-up, physics-based UBEM framework. These modeling frameworks produce a physics-based building energy model for each building in the area of interest. This method has many advantages including individual building granularity and the ability to modify the building energy model to evaluate different building technologies in an effort to save energy, demand, cost, and emissions. As examples This allows the user to determine which buildings are good candidates for a certain technology or understand how grid resilience will be impacted by a heatwave. If individual building resolution is not necessary, top-down UBEM approaches can be used. These approaches are typically data-focused and are at a municipality scale, often incorporating economic, transportation, and other pertinent urban data into their predictions.
The measured data from the utility is used to examine the sources of bias in the AutoBEM models and input data. AutoBEM used a variety of data sources to generate a building energy model for every building in EPB’s service area, simulate with weather data from the same year as measured data (2019), and compare simulation outputs to measured data. Annual building EUI is used to compare buildings in this area with the models generated using AutoBEM as well as the typical U.S. building stock.
While this analysis covers a single UBEM framework, many sources of bias are applicable to common UBEM methods. Reviews such as [8], [9] contain many state of the art UBEM methods and a fairly comprehensive summary of UBEM efforts is contained in [10]. Many of the methods utilize similar physical building data, building energy modeling simulation engines, and related data that likely leads to similar sources of bias. The authors of this work anticipate and encourage other studies to reproduce and compare against the findings of this paper as a step toward establishment of best-practices and standards for UBEM.
UBEM biases are introduced into the simulated data from a variety of sources. Simulation results are likely to be heavily influenced by occupant behavior not just at the individual building level but at the urban level [11]. The influence of occupant behaviors on UBEM simulation results is primarily from the challenge of accurately and fully incorporating diversity in equipment schedules and other energy-related occupant activities into building models’ EUI or timing of energy use. This is relevant in residential districts where occupancy can significantly affect peak demand and energy use [12]. Some of the occupant behaviors vary from passive exchange of heat with space; opening and closing doors and windows; adjustment of thermostat settings, light setting, blinds and shades, or clothing levels; use of personal heating and cooling devices; and consumption of warm or cold drinks’ [13]. Many studies assert that the factors which motivate occupant activities in buildings are many and include occupants’ personal choice, environmental behaviors, social interactions and the general cultural context [14], [15], [16].
Another source of bias in UBEM simulation data is using typical weather environment (or representative city) for an entire district or city without considering the true meteorological conditions at that place and time—including microclimate variation between the building and closest weather data (usu. airports) [11]. The urban climate and microclimate, and how they interact with building models, can have a significant impact on building energy use [17]. These conditions can affect the air temperature and humidity of the local environment, movement of air, solar radiation as well as specular and diffuse reflections [18]. There is also a reverse action of buildings on the boundary conditions of urban canopy modes in the following ways: (1) the building geometries and layout have thermal effects on heat and airflow [19]; (2) heat exchange between the buildings’ exterior surface and the urban environment through simulation of sensible heat convection and radiation; and (3) the release of heat and moisture by buildings to the urban environment through cooling towers, condensing units and exhaust air [11]. The bidirectional impacts of buildings and the urban environment on one another needs to be factored properly into the assessment of UBEM. To demonstrate how the use of more weather stations can reduce temperature bias, [20] used all available weather stations within the IECC climate zone over the western U.S. in a UBEM and this led to a reduction of the average absolute summertime temperature bias from 4.0 °C to 1.5 °C.
The distribution of internal loads in buildings can vary widely from one building to the other, even when they are of the same type and vintage. This can lead to significant discrepancy between simulated energy use and what is actually consumed [21]. While energy use is in buildings are more complex, modeling uncertainties often lead to a simplification of mechanical systems and their controls which fail to fully capture the dynamic behavior or part-load use of the mechanical systems including the response from their control systems [22]. Parameters related to specifications for HVAC use, systems and plants scheduling, and casual gains can all lead to an incomplete or inaccurate modeling of buildings energy systems [23]. [24] detected performance gap in thermal modeling and linked it an underestimation of the values used for specifying fan powers in simulation compared to actual use. Also, [25] identified a similar underestimation of lighting loads as a significant source of source bias in simulated building energy use.
Several other building energy researchers have explored the relationship between measured and predicted EUI. A review of UBEM techniques in 2016 aggregated error rates for UBEM applications of the time and reported average error ranges well above 10% [26]. The UBEM applications in this study also were relatively small compared to today’s standard, with the majority of studies focusing on less than 100 buildings. Another more recent review of 78 UBEM analyses reported error ranges from 1% to 1000% for single buildings [27]. A recent study of 1.2 million London residential buildings compared measured EUI to the UK’s Asset Rating tool value [28] with a weak correlation indicated between the measured data and the building efficiency rating values. An older work compared measured and predicted daily EUI for the city of Houston, TX. It was found that the daily predictions were typically within 10% of existing building types. Because of their daily scale, a focus on hourly patterns was evaluated with general prototype building schedules specified as a primary source of error in the predictions. A study of 172 residential buildings in Kuwait emphasized the need for stochastic occupant modeling in UBEM, showing that deterministic models ignore uncertainties up to 30% when considering single buildings [29]. A 2016 review evaluated the measured vs simulated EUI of building energy modeling tools for various building types all over the world. While not a UBEM review, this work highlights several sources of bias in individual building energy modeling that naturally relate to UBEM [29].
While there are many comparisons of simulated to measured building energy use in the literature, few of these exercises have been done for multiple buildings, let alone at an urban scale. This analysis compares tens of thousands of simulations to measured building energy data to better understand the strengths and limitations of UBEM frameworks as well as typical data that must be used at the urban scale.
Section snippets
Materials and methods
To assess differences in simulated and measured data, individual building simulations were compared to measured meter electricity data for a sample of real buildings. In partnership with EPB, measured electricity data for the year 2019 was shared from smart meters in Chattanooga, Tennessee for 50,843 buildings used in this study. The data was shared as 15-minute electricity data but was aggregated and scaled by the area to be analyzed as annual electricity use intensity.
AutoBEM utilizes
Results
Measured data from 50,843 selected buildings in EPB’s service territory is compared to individual simulations with visualizations for illustrating how well the models represent the measured data. For the following visualizations, the EUI values consist of only electricity. The models were converted to all electric HVAC and water heating for a better comparison to the all electric data measured data. The lack of natural gas data is considered in a bias adjustment further in the document.
The data
Discussion
There are a few main sources of bias that are evident from the results. While there may be some bias in AutoBEM (or UBEM frameworks generally), the majority of the bias is derived from the input data, which has a huge impact on individual simulation performance compared to measured data. There are two main sources of bias in the input data: missing or limited data describing each building and mis-labeled data.
The first source of bias related to input data is missing or limited data. This data
Limitations
There are several major limitations of the UBEM study. First, reliance on (only) electricity data limits the broader applicability of the work. Moreover, having other energy types would allow for a more reliable comparison between models and measured data. Second, use of general building characteristic data sources is limited, difficult, and intractable for many organizations. While thousands of inputs are defined for each model, only a few are directly measured, and most are interpreted or
Conclusions
This article compares measured and simulated building energy data for more than 50 thousand buildings in Chattanooga, TN. There are several primary findings from this work: (1) An uncalibrated UBEM framework (AutoBEM) is compared to measured data to evaluate its performance, both on individual buildings and the aggregated building distributions. Individual results were compared using scatter plots and trendlines while the distributions were compared using box-plots. It was found that the
CRediT authorship contribution statement
Brett Bass: Conceptualization, Methodology, Software, Formal analysis, Investigation, Data curation, Writing – original draft, Visualization. Joshua New: Conceptualization, Writing – review & editing, Supervision, Project administration, Funding acquisition. Nicholas Clinton: Data curation, Writing – review & editing. Mark Adams: Software, Development. Bill Copeland: Data curation, Writing – review & editing. Charles Amoo: Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was funded by field work proposal CEBT105 under US Department of Energy Building Technologies Office Activity Number BT0305000, as well as Office of Electricity Activity Number TE1103000. This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive,
References (50)
- et al.
From concept to application: A review of use cases in urban building energy modeling
Appl Energy
(2020) - et al.
Urban building energy modeling: State of the art and future prospects
Renew Sustain Energy Rev
(2020) - et al.
Ten questions on urban building energy modeling
Build Environ
(2020) - et al.
Clustering and statistical analyses of air-conditioning intensity and use patterns in residential buildings
Energy Build
(2018) - et al.
Occupants’ window opening behaviour: A literature review of factors influencing occupant behaviour and models
Build Environ
(2012) - et al.
A critical review of observation studies, modeling, and simulation of adaptive occupant behaviors in offices
Build Environ
(2013) - et al.
Simulating the human-building interaction: Development and validation of an agent-based model of office occupant behaviors
Build Environ
(2015) - et al.
Microclimatic modelling in assessing the impact of urban geometry on urban thermal environment
Sustainable Cities Soc
(2017) - et al.
Stochastic modeling of overtime occupancy and its application in building energy simulation and calibration
Build Environ
(2014) - et al.
Urban building energy modeling – A review of a nascent field
Build Environ
(2016)
On the accuracy of urban building energy modelling
Renew Sustain Energy Rev
A method for using street view imagery to auto-extract window-to-wall ratios and its relevance for urban-level daylighting and energy simulations
Build Environ
Building instance classification using street view images
ISPRS J Photogramm Remote Sens
Modelling urban-scale occupant behaviour, mobility, and energy in buildings: A survey
Build Environ
A novel population-based occupancy modeling approach for district-scale simulations compared to standard-based methods
Build Environ
Annual maps of global artificial impervious area (GAIA) between 1985 and 2018
Remote Sens Environ
Data-driven urban energy simulation (DUE-S): A framework for integrating engineering simulation and machine learning methods in a multi-scale urban energy modeling workflow
Appl Energy
Scaling beyond tax assessor data
Model America - data and models of every U.S. building
Accuracy of a crude approach to urban multi-scale building energy models compared to 15-min electricity use
Quality control methods for advanced metering infrastructure data
Potential demand reduction from buildings in a simulated utility
Distribution of potential savings from urban-scale energy modeling of a utility
Future meteorological year weather data from IPCC scenarios
U.S. building stock characterization study: A national typology for decarbonizing U.S. buildingsNREL/TP-5500-81186
Cited by (9)
Correlating the urban microclimate and energy demands in hot climate Contexts: A hybrid review
2023, Energy and BuildingsEstimating cooling loads of Arizona State University buildings using microclimate data and machine learning
2023, Journal of Building EngineeringCitation Excerpt :They found that adding 5% mature trees reduced cooling loads by 2.7% in July. Results from these studies may be highly influenced by potential bias that exists within BES models that might not be evident [6]. For example, a modeled building could be validated with actual data using statistical indices, but the model might inaccurately simulate wall/roof insulation or air ventilation.
An eight-step simulation-based framework to help cities reach building-related emissions reduction goals
2023, Environmental Research: Infrastructure and Sustainability