1 Introduction

Exploration of various data sets on a set of residential virtual power plant installations. These have PV, batteries and hot water controllers.

2 Code

See https://github.com/CfSOtago/vpp/tree/master/scripts

3 To do

See the repo issues list.

4 Test Data

This will fail if you are not connected to the University of Otago HCS service where the data is archived.

4.1 RgAv_Sample_Data.csv.gz

This data appears to be regional averages. We don’t know how many dwellings are covered in each region.

The following shows the original variable names and descriptives.

## Skim summary statistics
##  n obs: 2883462 
##  n variables: 17 
## 
## ── Variable type:character ─────────────────────────────────────────────────────────────────────────────────────────────────
##         variable missing complete       n min max empty n_unique
##  record time utc       0  2883462 2883462  19  19     0  2457316
##           region       0  2883462 2883462   0  19 20152       14
## 
## ── Variable type:numeric ───────────────────────────────────────────────────────────────────────────────────────────────────
##                     variable missing complete       n      mean     sd p0
##               battery charge       0  2883462 2883462   0.00053  0.012  0
##             electricity used   70439  2813023 2883462   0.22     1.25   0
##         grid electricty used   70439  2813023 2883462   0.18     1.17   0
##                      power 1       0  2883462 2883462   0.16     0.9    0
##                     power 1n       0  2883462 2883462   0.039    0.25   0
##                      power 2       0  2883462 2883462   0.044    0.42   0
##                     power 2n       0  2883462 2883462 232.96    24.83   0
##                      power 3       0  2883462 2883462   0.0097   0.081  0
##                     power 3n       0  2883462 2883462   0.005    0.15   0
##                      power 4       0  2883462 2883462   0.079    0.25   0
##                     power 4n       0  2883462 2883462   0.0026   0.031  0
##   solar electricity exported  119899  2763563 2883462   0.04     0.25   0
##  solar electricity generated   49413  2834049 2883462   0.08     0.25   0
##       solar electricity used  119899  2763563 2883462   0.045    0.16   0
##                      voltage       0  2883462 2883462   4.27     0.04   0
##     p25    p50    p75    p100     hist
##    0      0      0       0.98 ▇▁▁▁▁▁▁▁
##    0.06   0.12   0.27 1613.72 ▇▁▁▁▁▁▁▁
##    0.02   0.08   0.21 1534.42 ▇▁▁▁▁▁▁▁
##    0.01   0.07   0.19  932.83 ▇▁▁▁▁▁▁▁
##    0      0      0.01  160.42 ▇▁▁▁▁▁▁▁
##    0      0      0     601.59 ▇▁▁▁▁▁▁▁
##  233.01 235.72 238.17  254.63 ▁▁▁▁▁▁▁▇
##    0      0      0      48.48 ▇▁▁▁▁▁▁▁
##    0      0      0     147.96 ▇▁▁▁▁▁▁▁
##    0      0      0.1   143.28 ▇▁▁▁▁▁▁▁
##    0      0      0       9.45 ▇▁▁▁▁▁▁▁
##    0      0      0.01  160.42 ▇▁▁▁▁▁▁▁
##    0      0      0.11  143.28 ▇▁▁▁▁▁▁▁
##    0      0      0.06   88.85 ▇▁▁▁▁▁▁▁
##    4.25   4.27   4.3     6.52 ▁▁▁▁▁▇▁▁

Table 4.1 shows the first 10 rows of data.

Table 4.1: First 10 rows for Auckland ordered by dateTime NZ
record time utc	region	electricity used	grid electricty used	power 1	voltage	power 2n	dateTime	dateTimeNZ
2017-07-03 10:23:19	Auckland	0.00	0.00	0.00	4.19	0.00	2017-07-03 10:23:19	2017-07-03 22:23:19
2017-07-03 10:23:33	Auckland	0.00	0.00	0.00	4.27	0.00	2017-07-03 10:23:33	2017-07-03 22:23:33
2017-07-03 10:55:23	Auckland	0.00	0.00	0.00	4.21	0.00	2017-07-03 10:55:23	2017-07-03 22:55:23
2017-07-03 10:55:37	Auckland	0.00	0.00	0.00	4.21	0.00	2017-07-03 10:55:37	2017-07-03 22:55:37
2017-07-03 11:57:32	Auckland	0.00	0.00	0.00	4.18	0.00	2017-07-03 11:57:32	2017-07-03 23:57:32
2017-07-03 11:57:40	Auckland	0.00	0.00	0.00	4.21	0.00	2017-07-03 11:57:40	2017-07-03 23:57:40
2017-07-03 13:26:41	Auckland	0.00	0.00	0.00	4.29	0.00	2017-07-03 13:26:41	2017-07-04 01:26:41
2017-07-03 13:27:06	Auckland	0.00	0.00	0.00	4.26	0.00	2017-07-03 13:27:06	2017-07-04 01:27:06
2017-08-02 03:07:47	Auckland	0.00	0.00	0.00	4.12	0.00	2017-08-02 03:07:47	2017-08-02 15:07:47
2017-08-02 03:09:35	Auckland	0.01	0.01	0.01	4.23	236.49	2017-08-02 03:09:35	2017-08-02 15:09:35

N observations: 2883462
Dates range: 2017-07-03 22:10:26 to 2018-01-01 12:59:00

There is no obvous pattern to the sequence of time stamps (Figure 4.1) although it appears that there are pairs of observations close together Table 4.1.

## 
## Skim summary statistics
## 
## ── Variable type:difftime ──────────────────────────────────────────────────────────────────────────────────────────────────
##       variable missing complete       n    min          max  median
##  dt1$datesDiff      14  2883448 2883462 1 secs 5487700 secs 19 secs
##  n_unique
##      1264

## Don't know how to automatically pick scale for object of type difftime. Defaulting to continuous.

Figure 4.1: Density plot of time difference between sequantial observations (all data < 30 minutes)

Figure 4.2 shows the number of observations per day and presumably increases with the number of households in the sample? In which case what are the data points an average of?

Figure 4.2: N daily observations by region egion

Figure 4.3 shows the mean solar electricity generated per day over time.

## Warning: Removed 355 rows containing missing values (geom_path).

Figure 4.3: Mean daily solar electricity generated by region

Figure 4.4 shows the mean solar electricity generated by time of day for December and January only. It appears to make sense.

## Warning: Removed 336 rows containing missing values (geom_path).

Figure 4.4: Mean solar generation by time of day and region

4.2 Sample Monitoring Data.csv.gz

This data appears to be a sample of the household level monitoring data. It appears to cover 917 differet Sites.

The following shows the original variable names and descriptives.

## Skim summary statistics
##  n obs: 85770 
##  n variables: 17 
## 
## ── Variable type:character ─────────────────────────────────────────────────────────────────────────────────────────────────
##           variable missing complete     n min max empty n_unique
##  RecordDateTimeUTC       0    85770 85770  15  16     0     1440
## 
## ── Variable type:integer ───────────────────────────────────────────────────────────────────────────────────────────────────
##  variable missing complete     n  mean  sd p0 p25 p50 p75 p100     hist
##      Site       0    85770 85770 458.2 264  1 230 457 687  917 ▇▇▇▇▇▇▇▇
## 
## ── Variable type:numeric ───────────────────────────────────────────────────────────────────────────────────────────────────
##                   variable missing complete     n      mean     sd   p0
##              BatteryCharge       0    85770 85770   4.25     0.041 3.31
##            ElectricityUsed    1813    83957 85770   0.39     4.36  0   
##        GridElectricityUsed    1909    83861 85770   0.35     3.64  0   
##                     Power1       0    85770 85770   0.32     3.59  0   
##                    Power1N       0    85770 85770   0.017    0.34  0   
##                     Power2       0    85770 85770   0.061    1.15  0   
##                    Power2N       0    85770 85770   0.0055   0.045 0   
##                     Power3       0    85770 85770   0.01     0.078 0   
##                    Power3N       0    85770 85770   0.002    0.026 0   
##                     Power4       0    85770 85770   0.05     1.25  0   
##                    Power4N       0    85770 85770   0.00072  0.025 0   
##   SolarElectricityExported    4160    81610 85770   0.018    0.35  0   
##  SolarElectricityGenerated    2347    83423 85770   0.05     1.26  0   
##       SolarElectricityUsed    4160    81610 85770   0.036    1.02  0   
##                    Voltage       0    85770 85770 233.34    12.8   0   
##      p25    p50     p75   p100     hist
##    4.22    4.25   4.27    4.39 ▁▁▁▁▁▁▇▃
##    0.073   0.22   0.52  961.19 ▇▁▁▁▁▁▁▁
##    0.048   0.18   0.49  800.48 ▇▁▁▁▁▁▁▁
##    0.037   0.16   0.46  800.48 ▇▁▁▁▁▁▁▁
##    0       0      0      87.2  ▇▁▁▁▁▁▁▁
##    0       0      0     269.99 ▇▁▁▁▁▁▁▁
##    0       0      0       0.85 ▇▁▁▁▁▁▁▁
##    0       0      0       2.98 ▇▁▁▁▁▁▁▁
##    0       0      0       0.87 ▇▁▁▁▁▁▁▁
##    0       0      0.037 247.91 ▇▁▁▁▁▁▁▁
##    0       0      0       6.29 ▇▁▁▁▁▁▁▁
##    0       0      0      87.2  ▇▁▁▁▁▁▁▁
##    0       0      0.039 247.91 ▇▁▁▁▁▁▁▁
##    0       0      0.029 216.39 ▇▁▁▁▁▁▁▁
##  230.99  234.25 237.27  254.22 ▁▁▁▁▁▁▁▇

Table 4.2: First 10 rows ordered by dateTime NZ
RecordDateTimeUTC	Site	ElectricityUsed	SolarElectricityGenerated	GridElectricityUsed	SolarElectricityUsed	Power1	Power4	Power4N	BatteryCharge	Voltage	dateTime	dateTimeNZ
27/05/2018 12:11	1	0.773	0.000	0.773	0.000	0.773	0.000	0.001	4.239	238.01	2018-05-27 12:11:00	2018-05-28 00:11:00
27/05/2018 12:26	1	0.487	0.000	0.487	0.000	0.487	0.000	0.001	4.277	241.40	2018-05-27 12:26:00	2018-05-28 00:26:00
27/05/2018 12:41	1	0.339	0.000	0.339	0.000	0.339	0.000	0.001	4.277	239.55	2018-05-27 12:41:00	2018-05-28 00:41:00
27/05/2018 12:56	1	0.704	0.000	0.703	0.000	0.703	0.000	0.001	4.271	240.48	2018-05-27 12:56:00	2018-05-28 00:56:00
27/05/2018 13:11	1	0.723	0.000	0.723	0.000	0.723	0.000	0.001	4.252	241.19	2018-05-27 13:11:00	2018-05-28 01:11:00
27/05/2018 13:26	1	0.693	0.000	0.693	0.000	0.693	0.000	0.001	4.265	239.25	2018-05-27 13:26:00	2018-05-28 01:26:00
27/05/2018 13:41	1	0.269	0.000	0.268	0.000	0.268	0.000	0.001	4.258	242.49	2018-05-27 13:41:00	2018-05-28 01:41:00
27/05/2018 13:56	1	0.549	0.000	0.548	0.000	0.548	0.000	0.001	4.277	240.58	2018-05-27 13:56:00	2018-05-28 01:56:00
27/05/2018 14:11	1	0.788	0.000	0.788	0.000	0.788	0.000	0.001	4.252	239.52	2018-05-27 14:11:00	2018-05-28 02:11:00
27/05/2018 14:26	1	0.676	0.001	0.675	0.001	0.675	0.001	0.001	4.239	238.44	2018-05-27 14:26:00	2018-05-28 02:26:00

These appear to be 15 minute observations:

N sites: 917
N observations: 85770
Dates range: 2018-05-28 to 2018-05-28 23:59:00

Figure 4.5 shows the number of sites per day. We clearly have just one day of data!

Figure 4.5: N sites by date

Figure 4.6 shows the mean solar electricity generated per half-hour by each site. There may be some outliers or data errors…

## Warning: Removed 1182 rows containing missing values (geom_point).

Figure 4.6: Mean solar generated (legend hidden for clarity)

Figure 4.7 shows the mean solar electricity exported by time of day.

## Warning: Removed 2089 rows containing missing values (geom_point).

Figure 4.7: Mean exported per site (legend hidden for clarity)

5 About

R packages used:

base R [baseR]
data.table (Dowle et al. 2015)
drake (Landau 2018)
ggplt2 (Wickham 2009)
hms [*hms]
kableExtra (Zhu 2018)
lubridate (Grolemund and Wickham 2011)
skimr (Arino de la Rubia et al. 2017)

Plus:

bookdown (Xie 2016a)
knitr (Xie 2016b)

Analysis completed in 59.896 seconds ( 1 minutes) using knitr in RStudio with R version 3.5.2 (2018-12-20) running on x86_64-apple-darwin15.6.0.

References

Arino de la Rubia, Eduardo, Hao Zhu, Shannon Ellis, Elin Waring, and Michael Quinn. 2017. Skimr: Skimr. https://github.com/ropenscilabs/skimr.

Dowle, M, A Srinivasan, T Short, S Lianoglou with contributions from R Saporta, and E Antonyan. 2015. Data.table: Extension of Data.frame. https://CRAN.R-project.org/package=data.table.

Grolemund, Garrett, and Hadley Wickham. 2011. “Dates and Times Made Easy with lubridate.” Journal of Statistical Software 40 (3): 1–25. http://www.jstatsoft.org/v40/i03/.

Landau, William Michael. 2018. “The Drake R Package: A Pipeline Toolkit for Reproducibility and High-Performance Computing.” Journal of Open Source Software 3 (21). https://doi.org/10.21105/joss.00550.

Wickham, Hadley. 2009. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. http://ggplot2.org.

Xie, Yihui. 2016a. Bookdown: Authoring Books and Technical Documents with R Markdown. Boca Raton, Florida: Chapman; Hall/CRC. https://github.com/rstudio/bookdown.

———. 2016b. Knitr: A General-Purpose Package for Dynamic Report Generation in R. https://CRAN.R-project.org/package=knitr.

Zhu, Hao. 2018. KableExtra: Construct Complex Table with ’Kable’ and Pipe Syntax. https://CRAN.R-project.org/package=kableExtra.

Virtual Power PLants: Explore initial sample data

Report v1.0

Ben Anderson

Last run at: 2019-12-10 13:50:08