Ray Mistry
Chief Technology Officer
Testing and participation in various studies drives the development of the insights provided through the Project Canary SENSE dashboard. Whether it’s monthly testing at METEC, where we design the experiment and test detection (i.e., we have the answers to the test ahead of time), or through participation in the ADED protocol hosted by METEC (i.e., a blind experiment where we test our capabilities blind to the test conditions), both provide means to improve.
What are we improving for?
We are looking to improve the probability and accuracy of detection. As a result, we’re pursuing a more accurate and more robust depiction of Quantification and an Emissions profile. This ties into the main success factors governed by ADED – namely, the True Positive to False Positive ratio, which we will get into later in this blog.
About the ADED protocol
In a report released by METEC for ADED 2022, Project Canary was Participant F in the ADED results. Solutions are graded based on their ability to detect True Positives (TP) while minimizing False Positives (FP). Project Canary TP matches Participant A but with only 15% FP. On the other hand, participant E did very well in TP with 88%. However, FP would lead to alert fatigue at 79%.
What to optimize for?
Leaks of shorter duration (<5min) with a sensor downwind of the leak will likely be detected (CL of >90%).
Our mean time to detection (MTTD) tunes parameters through several techniques, not just stochastic but also an optimization to the resilience of the algorithms allowing for the detection of events in progress. Through this optimization, we can also detect smaller leaks (LDL, lower detection limit) of very short durations while monitoring continuously.
Fewer False Positives
False positives often lead to alert fatigue, whether actively notified by alerts pushed to the operator or by continuously observing the Project Canary SENSE dashboard. Optimizing for fewer False Negatives (missed events) often causes greater sensitivity of False Positives (non-existent events), meaning throwing an alert for non-events through inference of factors. Therefore, Project Canary looks to optimize for the most accurate (or higher) True Positive to False Positive ratio in alerting and event detection.
ADED 2023
In a previous blog post, we discussed more testing at METEC and the intent to participate in the 2023 round of ADED as well. As sensing principles converge in their approaches to testing and validation, we find ourselves in a space where Probability of Detection, True Positive to False Positive ratios, and the ability to reason an approach to an algorithm are at the forefront of Emissions Technology success.
What is the Probably of Detection (PoD)? – Defined as the likelihood of detection of an emissions event given the lower detection limit of a technology.
For ADED 2023, Project Canary (below) detects an emission event and still has a 95%+ probability detection threshold for leaks less than 1kg for shorter and lower-level emissions while continuously monitoring. That PoD is again depicted with the recent ADED 2023 results we have received – more on this to come.
Figure 2: Probability of Detection for sub-Kg/h concentrations reach 90% based on 2023 ADED Results and even better when accounting for Wind Normalization.
Finally, complementary to detecting low-level leaks with a high probability and mitigating alert fatigue (False Positives) – we look to optimize the enumeration of simultaneous leaks. With ADED, detecting that there are 5 leaks when there are 3, or 3 leaks detected when there are 5 leaks, does not count towards your True Positive tally. We’re creating a more accurate picture of the emissions profile as we optimize Project Canary’s Quantification algorithm’s ability to “see” more leaks.