ADED Testing ANALYSIS

Project canary's analysis of aded testing

Operators want to be alerted right when a leak starts at a facility so they can send a technician out to fix it. But given the significant resources involved in deploying leak mitigation teams, it can be just as important to avoid false alarms or “false positives.” Luckily, there’s a wide variety of incredible technologies and approaches capable of high-fidelity alerting.   

Satellite and aerial technologies have conducted single-source blind tests to determine if their technology can accurately detect leaks at facilities at various leak rates. These tests result in a probability of detection (PoD) curve, indicating the required leak rate (in kg/hr) for a 90% chance of their technology detecting the leak. The metrics presented below represent just that: What does the leak rate need to be at a facility for the Canary system to have a 90% chance of correctly alerting an operator? We’ve taken the 2023 ADED testing and normalized the single-blind anonymized, multi-source experiments to a single facility-wide emission event to calculate our 90% PoD. There are countless other ways to present this data, including normalizing for wind speeds (“wind normalized PoD”), requiring that an alert correctly identifies every individual leak at a facility to be counted as a successful outcome (“method detection limit”), and controlling for leak duration.  

To be fully transparent, we’ve published every experiment conducted in the 2023 ADED testing campaign, showing if Canary correctly sent an alert about a leak occurring at the METEC facility. Project Canary is technology “F” in the 2022 ADED results comparison linked below. 

90% Detection Limit

Computed via logistic regression using the below definitions for True Positives and False Negatives

0.6 kg/h

0.21 (kg/h) per (m/s) wind normalized

0.007 kg/h

Minimum Detected Release Rate

94.1% | 239

True Positives Tool Tip: True PositivesDefined as: Events where one or more releases were detected

5.9% | 15

False Negatives Tool Tip: False NegativesDefined as: Events where no releases were detected

0

False Positives Tool Tip: False PositivesDefined as: Event reports generated where no releases occurred

3.2% | 8

Overreported Tool Tip: OverreportedDefined as: Continuous experiments reported as multiple events by Project Canary (these are considered false positives by METEC)

254

Total Experiments

Project Canary = Solution F

ADED Results

Click (+) to Expand Table Featuring Detections for Every Experiment

= Correct localization of emission source
Equipment Group Release Rate (kg/h)
Experiment Start Duration (hrs) Wind Speed (mph) 4S
Pad 4 - Separator
4W
Pad 4 - Wellhead
4T
Pad 4 - Tank
5S
Pad 5 - Separator
5W
Pad 5 - Wellhead
Detected
02/08/2023 09:08 3.0 4.3 0.99 1.06 Yes
02/08/2023 14:08 4.0 15.2 2.22 1.63 Yes
02/08/2023 19:08 5.0 16.9 1.52 1.13 Yes
02/09/2023 02:08 7.0 18.4 1.13 1.29 Yes
02/09/2023 11:08 6.0 20.3 2.46 1.67 Yes
02/09/2023 18:08 8.2 4.2 0.72 0.92 1.69 Yes
02/10/2023 04:19 9.7 3.3 1.08 0.97 1.00 Yes
02/10/2023 15:00 2.0 3.7 2.17 1.59 Yes
02/10/2023 19:00 10.2 2.6 1.03 0.60 1.08 0.64 Yes
02/11/2023 09:11 5.2 2.5 0.68 0.88 1.59 Yes
02/11/2023 17:22 6.2 2.9 0.88 0.90 1.00 Yes
02/12/2023 01:33 9.2 3.5 1.03 0.59 1.07 0.63 Yes
02/12/2023 14:44 4.0 10.4 1.47 1.08 Yes
02/12/2023 21:44 3.0 2.6 2.26 1.66 Yes
02/13/2023 03:44 7.2 2.9 1.06 0.61 1.11 0.66 Yes
02/13/2023 13:02 5.0 5.2 0.60 3.64 Yes
02/14/2023 04:02 4.2 18.5 4.27 4.86 1.10 Yes
02/14/2023 11:16 7.0 21.7 1.57 Yes
02/14/2023 19:16 8.0 10.4 0.59 1.39 Yes
02/15/2023 05:16 6.0 7.1 0.66 2.58 Yes
02/15/2023 16:46 9.2 3.1 0.90 0.41 1.20 0.53 Yes
02/16/2023 03:57 7.2 2.7 1.14 0.40 0.76 Yes
02/16/2023 12:08 5.0 8.9 0.99 1.08 Yes
02/16/2023 19:08 4.2 2.2 0.89 0.41 1.21 0.53 Yes
02/17/2023 01:19 9.0 1.8 0.77 Yes
02/17/2023 11:19 5.0 6.1 0.56 0.60 Yes
02/17/2023 18:19 10.2 2.7 0.51 0.77 Yes
02/18/2023 07:30 10.0 3.3 0.44 0.41 Yes
02/18/2023 21:30 6.0 4.7 0.39 1.43 0.44 Yes
02/19/2023 07:30 5.0 18.3 1.08 0.44 Yes
02/19/2023 15:30 8.0 15.5 0.59 0.64 Yes
02/20/2023 03:30 7.7 6.4 0.50 0.75 Yes
02/20/2023 18:12 10.0 5.3 0.21 0.23 Yes
02/21/2023 06:13 9.0 5.2 0.13 0.18 Yes
02/21/2023 16:13 8.0 9.1 0.22 0.23 Yes
02/22/2023 02:13 10.1 11.3 0.10 0.14 Yes
02/22/2023 13:20 9.0 13.0 0.18 0.20 Yes
02/23/2023 00:20 10.0 3.7 0.14 0.20 Yes
02/23/2023 11:58 8.0 4.7 0.23 0.24 Yes
02/23/2023 21:58 10.0 2.1 0.14 0.20 Yes
02/24/2023 11:38 4.0 8.5 0.18 0.20 Yes
02/24/2023 19:38 7.1 3.3 0.10 0.10 0.14 Yes
02/25/2023 10:46 4.0 5.1 0.13 0.18 Yes
02/25/2023 22:46 4.1 2.6 0.13 0.10 0.10 0.12 Yes
02/26/2023 10:53 5.1 8.1 0.10 0.13 Yes
02/27/2023 00:00 1.5 16.9 0.17 0.19 Yes
02/27/2023 13:58 2.9 10.5 0.10 0.10 0.14 No
02/27/2023 18:50 4.0 10.8 0.13 0.19 Yes
02/28/2023 01:50 3.0 4.0 0.21 0.24 Yes
02/28/2023 06:50 4.1 10.8 0.10 0.14 Yes
02/28/2023 13:18 3.8 22.3 0.18 0.20 No
02/28/2023 20:03 3.1 2.5 0.10 0.10 0.14 Yes
03/01/2023 03:10 3.0 3.0 0.20 0.19 Yes
03/02/2023 15:08 2.0 7.2 2.50 Yes
03/02/2023 19:08 3.0 2.7 1.74 Yes
03/02/2023 23:08 4.0 2.3 1.80 Yes
03/03/2023 04:08 3.0 3.9 0.90 Yes
03/03/2023 08:08 4.0 3.0 0.01 No
03/04/2023 12:17 1.0 9.5 3.18 Yes
03/04/2023 16:17 1.5 4.7 2.52 Yes
03/04/2023 22:17 1.0 4.9 1.72 Yes
03/05/2023 03:17 1.5 4.2 3.05 Yes
03/05/2023 09:17 1.0 8.8 1.71 Yes
03/05/2023 14:17 1.0 21.7 3.21 No
03/05/2023 19:17 2.0 14.9 0.51 Yes
03/06/2023 01:17 1.0 1.8 0.77 Yes
03/06/2023 12:30 2.3 11.7 1.30 Yes
03/06/2023 17:01 2.0 8.3 0.64 0.77 Yes
03/06/2023 21:01 1.0 8.9 1.75 Yes
03/07/2023 01:01 2.1 10.3 0.34 0.68 0.30 Yes
03/07/2023 06:08 1.5 10.3 1.11 Yes
03/07/2023 10:38 1.5 10.4 0.58 0.75 Yes
03/07/2023 14:08 1.0 3.3 1.70 Yes
03/07/2023 18:08 1.5 3.9 0.63 0.76 Yes
03/07/2023 21:38 1.1 3.0 0.03 0.67 0.29 Yes
03/08/2023 01:45 1.0 5.2 1.17 Yes
03/08/2023 04:45 1.5 6.4 0.51 0.45 Yes
03/08/2023 11:01 3.0 7.5 0.49 0.44 Yes
03/08/2023 16:57 1.0 7.7 1.73 Yes
03/08/2023 19:57 2.1 6.4 0.33 0.67 0.29 Yes
03/09/2023 01:04 1.0 4.3 1.14 Yes
03/09/2023 05:04 2.0 4.3 0.50 0.45 Yes
03/09/2023 10:04 0.5 13.1 1.68 Yes
03/09/2023 12:34 1.6 13.4 0.31 0.65 0.26 Yes
03/09/2023 17:11 2.0 10.1 1.07 Yes
03/09/2023 22:11 1.5 3.3 0.51 0.44 Yes
03/10/2023 02:41 2.5 4.6 1.11 Yes
03/10/2023 13:51 9.0 3.6 0.05 0.05 Yes
03/11/2023 01:51 5.0 12.9 0.13 No
03/11/2023 09:51 4.1 21.3 0.10 0.10 0.13 No
03/11/2023 16:58 6.0 8.6 0.10 Yes
03/12/2023 00:58 5.0 5.7 0.04 0.09 Yes
03/12/2023 09:59 7.0 7.3 0.10 Yes
03/12/2023 19:59 4.0 5.0 0.13 Yes
03/13/2023 02:59 6.0 2.0 0.05 0.05 Yes
03/13/2023 21:01 2.0 4.6 0.03 0.02 Yes
03/14/2023 01:01 2.5 3.1 2.40 Yes
03/14/2023 06:31 0.8 3.3 2.41 Yes
03/14/2023 09:16 2.0 3.1 0.02 0.01 Yes
03/14/2023 14:16 1.0 7.4 5.87 Yes
03/14/2023 17:16 1.5 5.6 5.00 Yes
03/14/2023 21:46 1.0 2.5 2.18 0.01 Yes
03/15/2023 00:46 3.0 4.1 2.08 Yes
03/15/2023 06:47 2.0 4.6 4.12 0.02 Yes
03/15/2023 10:47 1.5 4.3 3.03 Yes
03/15/2023 19:17 1.5 9.2 3.16 Yes
03/15/2023 23:47 0.8 15.4 2.35 Yes
03/16/2023 02:32 2.0 15.0 0.03 0.04 Yes
03/16/2023 07:32 2.5 21.8 2.12 No
03/16/2023 12:02 0.3 18.2 4.03 0.01 Yes*Detected but not reported. Duration below reporting threshold.
03/16/2023 15:18 2.0 22.1 2.36 1.70 Yes
03/16/2023 19:18 0.8 13.1 6.05 Yes
03/16/2023 23:03 1.0 4.9 2.24 1.71 Yes
03/17/2023 02:03 1.0 2.6 5.26 Yes
03/17/2023 06:03 0.8 2.9 6.42 Yes
03/17/2023 08:48 2.0 6.6 2.35 1.72 Yes
03/17/2023 22:36 2.0 3.0 3.18 Yes
03/18/2023 05:36 3.0 2.1 2.40 3.39 Yes
03/18/2023 13:36 2.0 14.6 3.49 Yes
03/18/2023 20:36 2.5 3.9 2.41 2.22 Yes
03/19/2023 04:06 3.0 1.7 2.42 Yes
03/19/2023 12:06 2.5 6.0 1.64 3.29 Yes
03/19/2023 19:36 2.0 4.0 3.12 Yes
03/20/2023 02:36 2.0 2.2 2.38 3.35 Yes
03/20/2023 09:37 2.0 6.2 2.07 Yes
03/20/2023 16:00 0.5 9.7 3.91 Yes
03/20/2023 18:30 2.5 11.7 3.11 Yes
03/20/2023 23:00 0.5 5.5 5.83 Yes
03/21/2023 01:30 2.5 5.8 3.59 Yes
03/21/2023 07:00 0.8 5.3 3.53 Yes
03/21/2023 09:45 2.5 15.0 3.43 Yes
03/21/2023 15:15 3.0 12.5 5.87 Yes
03/21/2023 20:15 3.5 4.9 5.24 Yes
03/22/2023 02:46 3.5 4.2 5.59 Yes
03/22/2023 08:16 0.8 2.9 5.99 Yes
03/22/2023 12:01 3.5 4.7 4.01 Yes
03/22/2023 17:31 2.0 2.8 6.94 Yes
03/22/2023 22:31 2.5 2.9 5.54 Yes
03/23/2023 03:01 2.5 3.1 5.26 Yes
03/23/2023 08:31 3.0 2.4 5.54 Yes
03/23/2023 13:31 4.0 8.6 1.74 Yes
03/23/2023 20:31 3.0 3.6 6.01 Yes
03/24/2023 01:31 4.0 3.1 5.56 Yes
03/24/2023 08:31 4.0 5.1 3.57 Yes
03/24/2023 17:23 2.1 4.2 3.77 6.73 4.12 Yes
03/24/2023 22:30 1.0 4.4 3.44 Yes
03/25/2023 01:30 0.5 7.9 3.98 Yes
03/25/2023 05:00 2.5 21.7 7.10 Yes
03/25/2023 09:30 2.0 9.0 5.09 Yes
03/25/2023 14:30 3.0 19.0 5.29 Yes
03/25/2023 20:30 4.0 5.2 3.64 Yes
03/26/2023 02:30 4.6 2.2 5.09 5.91 3.35 Yes
03/26/2023 10:07 3.0 5.8 4.98 Yes
03/26/2023 15:07 2.0 8.7 5.22 Yes
03/26/2023 20:07 5.0 7.8 3.20 Yes
03/27/2023 03:07 1.6 6.6 0.79 3.60 2.73 Yes
03/27/2023 07:44 1.0 10.1 1.59 Yes
03/27/2023 10:44 3.0 15.2 1.01 Yes
03/28/2023 15:02 6.6 5.0 0.04 0.04 0.11 0.11 Yes*Detected but not accepted by METEC. Reported start before experiment start due to test center releases prior to experiment.
03/28/2023 23:39 5.6 4.7 0.01 0.01 0.09 Yes
03/29/2023 08:17 6.0 7.3 0.02 0.10 Yes
03/29/2023 16:17 5.6 5.1 0.04 0.04 0.09 Yes
03/30/2023 00:54 6.6 2.3 0.01 0.09 0.09 Yes
03/30/2023 09:31 7.6 3.9 0.01 0.01 0.09 0.09 Yes
03/30/2023 20:08 7.6 8.3 0.02 0.02 0.09 Yes
04/01/2023 17:03 4.6 5.3 0.04 0.04 0.13 No
04/02/2023 01:40 5.6 4.8 0.02 0.02 0.12 No
04/02/2023 10:17 3.5 10.9 0.06 0.14 No
04/02/2023 17:47 5.6 5.9 0.06 0.06 0.13 No
04/03/2023 02:24 4.6 7.0 0.01 0.01 0.11 No
04/03/2023 09:01 1.5 5.3 0.01 0.01 0.12 No
04/03/2023 11:03 1.1 9.6 0.03 0.03 0.11 0.11 No
04/03/2023 13:05 1.5 10.9 1.07 0.60 Yes
04/03/2023 16:35 5.6 10.4 0.10 0.14 0.19 Yes
04/04/2023 00:12 3.6 9.4 0.11 0.10 0.20 Yes
04/04/2023 05:49 6.6 9.7 0.04 0.05 0.10 No
04/04/2023 14:26 3.6 12.9 0.10 0.05 0.05 0.10 Yes
04/04/2023 20:03 5.6 4.5 0.11 0.10 0.20 Yes
04/05/2023 03:40 4.5 4.4 0.17 0.20 Yes
04/05/2023 10:10 5.6 4.5 0.10 0.14 0.19 Yes
04/05/2023 19:17 1.2 9.2 0.42 0.48 0.64 0.58 0.63 Yes
04/05/2023 23:28 0.9 2.1 0.74 0.68 0.77 Yes
04/06/2023 03:20 1.6 2.4 0.50 0.59 0.52 0.61 Yes
04/06/2023 07:57 1.1 3.1 0.62 1.03 0.92 Yes
04/06/2023 12:04 0.8 5.1 1.01 1.62 Yes
04/06/2023 15:49 1.6 6.7 0.81 0.78 0.83 Yes
04/06/2023 20:26 0.8 5.9 1.06 1.29 Yes
04/07/2023 00:11 0.9 6.4 0.43 0.48 0.64 0.58 0.64 Yes
04/07/2023 04:07 1.1 5.8 0.73 0.65 0.77 Yes
04/07/2023 10:36 2.1 5.0 0.81 0.80 0.82 Yes
04/07/2023 16:24 0.5 4.5 1.02 1.61 Yes
04/07/2023 21:54 1.1 6.9 0.50 0.58 0.53 0.62 Yes
04/08/2023 04:01 0.9 5.5 0.46 0.52 0.68 0.63 0.69 Yes
04/08/2023 09:57 1.1 2.1 0.72 0.49 0.76 Yes
04/08/2023 16:04 0.8 8.8 1.04 1.27 Yes
04/08/2023 21:49 0.6 5.3 0.87 0.85 0.91 Yes
04/09/2023 03:26 0.8 2.5 1.11 1.74 Yes
04/09/2023 09:11 0.7 2.6 0.44 0.49 0.65 0.61 0.66 Yes
04/09/2023 14:52 0.9 5.8 0.57 0.94 0.85 Yes
04/09/2023 20:44 0.5 5.3 1.08 1.31 Yes
04/10/2023 02:14 1.1 4.1 0.50 0.59 0.54 0.63 Yes
04/10/2023 20:47 0.6 5.3 1.21 1.51 1.66 Yes
04/11/2023 01:24 0.6 3.9 1.71 1.73 2.46 Yes
04/11/2023 05:01 0.6 5.0 0.86 1.53 2.48 Yes*Detected but not accepted by METEC. Reported start before experiment start due to test center releases prior to experiment.
04/11/2023 09:38 0.4 2.1 1.67 2.65 2.50 Yes
04/11/2023 13:00 0.6 8.4 1.18 1.49 1.63 Yes
04/11/2023 17:37 0.6 8.5 1.65 1.65 2.29 Yes
04/11/2023 21:14 0.6 6.1 0.87 1.52 2.39 Yes
04/12/2023 10:44 1.1 10.5 1.64 2.59 2.44 Yes
04/12/2023 13:51 1.6 11.4 1.18 1.46 1.61 Yes
04/12/2023 18:46 0.8 10.0 2.44 Yes*Detected but not accepted by METEC. Reported start before experiment start due to offsite release prior to experiment.
04/12/2023 22:31 1.0 3.1 1.21 1.45 Yes
04/13/2023 02:31 0.8 3.7 3.03 2.83 Yes
04/13/2023 06:16 0.9 4.3 1.23 1.51 1.67 Yes
04/13/2023 10:08 1.5 2.8 1.67 1.66 Yes*Detected but failed to localize
04/13/2023 14:38 0.9 4.2 2.96 1.48 Yes
04/13/2023 18:31 1.5 13.0 1.20 2.61 Yes
04/13/2023 22:01 1.0 2.7 3.16 3.05 Yes
04/14/2023 02:01 1.5 9.4 0.86 2.70 Yes
04/14/2023 05:31 2.0 6.0 1.24 1.46 Yes
04/14/2023 09:31 2.0 7.3 3.58 4.02 Yes
04/14/2023 23:00 1.1 1.6 0.86 1.38 2.24 Yes
04/15/2023 06:07 1.1 1.6 0.64 1.33 1.03 Yes
04/15/2023 13:14 1.0 12.7 1.23 1.33 Yes
04/15/2023 20:14 1.1 3.0 0.87 1.38 2.22 Yes
04/16/2023 10:28 0.8 9.0 1.23 2.58 Yes
04/16/2023 17:13 1.1 8.3 0.69 1.30 1.20 Yes
04/17/2023 00:20 0.9 4.1 0.86 1.37 2.22 Yes
04/17/2023 11:53 3.5 6.6 1.20 1.32 Yes
04/17/2023 17:23 3.1 6.0 0.87 1.99 1.32 2.32 Yes
04/17/2023 23:30 2.2 2.8 1.24 2.48 2.02 3.55 Yes
04/18/2023 15:12 0.4 10.5 1.19 2.34 1.94 3.08 Yes
04/18/2023 19:11 2.7 13.2 2.41 1.19 Yes
04/19/2023 12:29 2.7 11.8 2.52 1.20 Yes
04/19/2023 21:04 2.4 8.6 1.25 2.03 1.40 2.04 Yes
04/20/2023 14:12 2.3 20.9 2.90 3.33 Yes
04/20/2023 22:27 3.4 7.7 1.25 2.04 1.40 2.05 Yes
04/21/2023 11:14 3.3 5.8 2.63 2.88 Yes
04/21/2023 19:30 3.5 9.7 2.03 2.06 Yes
04/22/2023 04:00 3.4 5.1 2.67 1.20 Yes
04/22/2023 12:26 2.0 4.8 2.03 2.06 Yes
04/24/2023 17:09 2.5 4.8 2.59 2.58 Yes
04/25/2023 14:07 1.5 7.2 3.51 6.98 Yes
04/25/2023 20:41 1.5 8.8 3.58 5.32 Yes
04/26/2023 03:11 1.5 5.3 3.53 6.71 Yes
04/26/2023 09:38 1.0 4.1 3.48 6.96 Yes
04/26/2023 12:38 1.4 8.7 3.54 4.92 Yes
04/26/2023 17:04 1.5 9.4 3.53 5.12 Yes
04/26/2023 22:34 1.0 2.7 3.49 6.65 Yes
04/27/2023 03:35 0.5 2.7 3.17 5.41 Yes
04/27/2023 10:35 2.7 16.4 4.17 5.36 Yes
04/27/2023 18:08 1.8 15.2 3.57 5.27 Yes
04/28/2023 00:53 1.0 17.0 3.51 6.69 Yes
04/28/2023 09:30 2.5 9.6 3.07 5.34 Yes

ADED TESTING OVERVIEW

  • Evaluation of the performance of leak detection and quantification solutions
  • ADED is a controlled and blind study conducted and designed by Colorado State University at their METEC site

  • Project Canary's analysis considers collection of simultaneous releases as "events"

Contact Us

"*" indicates required fields

Name*
This field is for validation purposes and should be left unchanged.

Scroll to Top