There is Never too Much Data!

There is more data to deal with than there used to be. However, dealing with it may require a different approach than simply working harder with the same tools.

The new “Global Hydrological Monitoring Industry Trends” report confirms the rapid international adoption of continuous monitoring technologies.

One of the questions asked of over 700 respondents representing monitoring agencies from around the world was:

“Are the primary technologies used in your network (or client networks) changing? Please select all the technologies significantly used (i.e. making up 10% or more of your network) in 2002, 2012, and 2022 (forecast).”

We aggregated the individual responses into the percentages shown in table 1. The answers to this question are significant for a number of reasons as it reflects an abrupt departure from the stability of hydrometric technology in the 20th century.

Table 1. Average Percentage (%) of Stations

2002 2012 2022
Analog data retrieval (e.g. charts) 38% 26% 22%
Digital data retrieval (e.g. logger files) 46% 63% 68%
Radio (e.g. VHF, meteor burst) 16% 20% 28%
Telephone (e.g. landline, cellular) 24% 33% 40%
Satellite (e.g. GOES, Iridium) 16% 27% 41%
Web enabled sensors (e.g. IP Comms) 4% 20% 40%

Clearly there is a large investment in new technology, which infers a capital expenditure that is – in my experience – without precedent. The newer, more sophisticated, technologies require specialized training, which infers a large expenditure from operational budgets. The advanced training requirement is a challenge for individuals with limited academic experience resulting in a change in recruitment strategy, which in turn has an impact on salary costs to compete for recruits with postsecondary education.

Another important cost is the cost of managing all of this data. The question was deliberately unconstrained to capture the extent to which data redundancy is used. In 2002 the average stream gauge only had 144% technological redundancy whereas by 2022 the average gauge is forecast to have multiple technologies for data capture and transmission resulting in 240% redundancy. Whereas a chart recorder can only handle a single parameter, multi-channel data loggers handle any number of parameters. We don’t have a metric to compare the number of data points per year per gauge, but the survey results indicate a large increase, which is only getting larger as the capabilities of the technologies are being discovered and exploited.

Not only is there way more data but much of this data is being acquired in real-time to support timely decision-making. Over the years we have made incremental changes to the methods we used to rely on for manual review and quality control of our analog chart records but, fundamentally, the number of times a human touches the data in the process of data review and editing have not changed that much. There will never be a better time to give these methods a re-think than right now. There is a saying that “no matter how far you have travelled down the wrong path, it is never too late to turn around”.

My approach to data review and correction has been totally turned around as a result of getting to know Derek Forsbloom over the past several years.

On January 31st you can attend the hydrology webinar 7 Ways to Quality Control Water Data in Real-Time. Derek Forsbloom will share practical, tested, and refined controls that he has automated at Water Survey of Canada to correct the continuous water data published from 2,500 gauging stations.

For more details and to register, please follow this link: http://pages.aquaticinformatics.com/Webinar-Derek-Forsbloom.html

Or for a copy of the complete Global Hydrological Monitoring Industry Trends Report, please visit:
http://pages.aquaticinformatics.com/Water_Report.html

Resources_Webinar-WSC

WSC Webinar: 7 Ways to Quality Control Water Data in Real-Time

Derek Forsbloom shares how Water Survey of Canada systematically corrects continuous water data from 2,500 gauging stations. Learn how to elevate the quality of hydrometric data in real-time.

 

Resources_Whitepaper-Industry-Report

Survey Report: Global Hydrological Monitoring Industry Trends

700+ water professionals participated in this global hydrological monitoring study. How can you keep pace with the industry revolution? Get this report for the latest trends, standards, and practices.

One response to “There is Never too Much Data!”

  1. alejandro barzi January 28, 2013 at 9:35 am

    My personal experience over the Andeopatagonian Andes where i live and work in favour of good management of territories from lake watersheds point of view is that marriage between politicians, decisors and entrepreneurs deny to take care of facts and act in consecuence. They reject and deny, they put everything that alerts or puts limits inside a box far from being seen by the public, because is against their business – particularly that coming from ecologic sciences – So… here, althou if you collect mountains of data, will serve no one. This was my last effort over a beautiful and tiny mountain valley that is being deformed so fast, that you feel the vertigo! http://planterritorialelhoyo.blogspot.com

Join the conversation