What This Game Needs Is a Goal Reports IDTechEx

BOSTON, December 21, 2022 /PRNewswire/ — What does good look like? Whether trying to predict World Cup results or evaluating the relative merits of emerging technologies, establishing a meaningful set of performance metrics is crucial but challenging. Using metrics like expected goals or possession percentages to assess soccer teams isn’t particularly accurate, as a series of shock results at this World Cup show. For emerging technologies such as quantum computing, covered in IDTechEx’s recent report, it is even more difficult to establish universally applicable benchmarks that communicate development timelines and facilitate meaningful comparisons between competing companies and technical approaches.

Possession is not everything

There are countless ways to measure the quality of a football team: possession, shots on target, expected goals, passing accuracy, aerial challenges won – the list goes on. While each of these metrics has value, none can ultimately be considered in isolation. The same is true for quantum computers. Popular and commonly used metrics include qubit number, coherence time, error rate, quantum volume, algorithmic volume, and circuit layer operations per second (see image below for extended definitions).

But, unlike goals in soccer, for quantum computing, there is no single data point that provides an indisputable measure of quality. Furthermore, many metrics used in the quantum computing industry are even more complex to understand than ‘expected goals’, especially for those without a physics background.

Despite these challenges, benchmarking the emerging quantum computing industry is essential for predicting technology trends, making investments, and starting partnerships. As such, those interested in this potentially transformative technology will benefit from a greater understanding of common benchmarking methods, the trade-offs associated with each approach, and how they can be used in combination. There is also room for hardware developers to improve in communicating how quantum computing benchmarks can be used effectively when engaging with non-expert stakeholders.

Biased referees

One of the challenges of developing benchmarks for a new technology is that there are no existing rules. As a result, some quantum computing benchmarks have been developed by companies that build quantum computers themselves. It risks the equivalent of combining football manager and referee in a single role – we don’t want teams in control of the ball deciding to use ‘possession percentage’ to decide who won the game.

While a completely independent set of benchmarks would clearly be desirable, many of those who can understand the technology and set meaningful benchmarks work for companies developing quantum computers. Furthermore, the development of internal benchmarks that are then widely adopted provides a significant strategic advantage – as the saying goes, ‘those who define the terms control the debate’.

While some internally developed benchmarks have done a good job of consolidating existing metrics into a single parameter, they run the risk of introducing even more terminology, further fragmenting the benchmark ecosystem. There is therefore a significant need for independent, broadly applicable benchmarks that facilitate comparison regarding the status of quantum computing companies and technologies with regard to commercialization.

Focus on the next game

Technology Readiness Levels (TRL) are widely used in many sectors to assess the status of emerging technologies. However, they are not well-suited for computing, as each commercially available iteration of a computing technology shows a significant performance improvement, despite always having a TRL of 10.

This consideration of ‘continuous computation’ has been missed by some existing attempts to measure quantum computers. For example, some benchmarks end at the production of a ‘universal machine’ or demonstration of ‘supremacy’. If this logic is applied to the classical computing industry, benchmarks will show no progress since the 1980s. Rather than focusing on TRL, the focus for quantum computers should be on continuously expanding capabilities to serve an ever-growing addressable market. The question should be, ‘are quantum computers ready yet’, but rather, ‘who else are they useful for now?’

It’s all about the silverware

Despite the wide variety of statistical measures, the ultimate measure of football success is winning trophies. In quantum computing, satisfying a demanding set of benchmarks is only impressive if it ultimately provides value to end users. This overarching goal can be obscured by the focus on technical metrics and ideas such as ‘supremacy’, ‘million-qubit milestones’ and ‘fault tolerance’. The key tipping point will come when quantum hardware is scaled up sufficiently to tackle quantum algorithms that can solve commercially valuable problems, such as drug development or weather forecasting, that cannot be achieved by other means. Measuring how quantum computing companies are progressing relative to this goal and the range of valuable problems that can be addressed is arguably the most important requirement for a universal benchmark.

Quantum Commercial Readiness Level

To help assess the status of quantum computing companies and technologies on this path to delivering significant societal benefits, IDTechEx has developed an unbiased ‘Quantum Commercial Readiness Level’ (QCRL) index. The QCRL ranges from 1-10, with stages 1-5 tracking use cases from educational tools to versatile machines and 6-10 assessing scalability from cloud access to personal ownership. The aspiration is that it will remain relevant as the industry progresses beyond ‘quantum dominance’, hardware agnostic and easy to understand.

Outlook

There are many challenges when it comes to benchmarking quantum computers. The terminology and science underlying them is complex and there are many competing technologies. Companies that develop benchmarks of good quantum computers also use them to judge their own technology, and well-known standards such as technology readiness level are not very useful. However, benchmarking is so essential to demonstrating progress to investors and other stakeholders that efforts to standardize it for the industry must continue. An emphasis on unbiased input and how close the technology is to adding commercial/societal value is crucial. In short, what this game needs is a goal.

More details and twenty-year market forecasts can be found in IDTechEx’s comprehensive report “Quantum Computing 2023-2043”. It includes individual forecast lines for eight different technology categories, including superconducting, photonic, trapped-ion, neutral atom, silicon spin, topological, diamond defect and annealers. There are also 60-year projections covering meta-trends for quantum computing adoption, which go beyond the horizon of a realized versatile computer and look ahead to mass market adoption.

To find out more, including downloadable sample pages, please visit www.IDTechEx.com/QuantumComputing.

About IDTechEx

IDTechEx guides your strategic business decisions through its research, subscription and consulting products, helping you take advantage of emerging technologies. For more information, contact [email protected] or visit www.IDTechEx.com.

Download images:
https://www.dropbox.com/sh/gq0ydpbk4ydscsz/AADyBOjLlAm0cmpTkl28iXSza?dl=0

Media contact:

Lucy Rogers
Sales and Marketing Administrator
[email protected]
+44(0)1223 812300

Social media links:

Twitter: www.twitter.com/IDTechEx
LinkedIn: www.linkedin.com/company/IDTechEx
Facebook: www.facebook.com/IDTechExResearch

Photo: https://mma.prnewswire.com/media/1972795/IDTechEx_QCRL.jpg
Logo: https://mma.prnewswire.com/media/478371/IDTechEx_Logo.jpg

SOURCE IDTechEx