Data rescue and digitization: tips and tricks resulting from the Dutch experience

At the end of the 1980’s KNMI digitized part of the huge amount of pre-1850 instrumental meteorological data in the Netherlands. The activity took place within the scope of a four-year EU-project. In the year 2000 KNMI renewed its efforts in the area of data rescue and digitization with a long-term activity (partly externally funded). Since then many types of data have been digitized and made available to the public. Examples of the data types that have been dealt with are: 18th and 19th century ship logs, 19th century KNMI year books, films with observer log books of the Amsterdam City Water Office, strip charts and paper rolls from pluviographs, logbooks with rainfall measurements in the 1850-1950 period, data from the colonial past and metadata archives.
Although the basic principles for digitization are the same for all data types, there may be important differences in the implementation. For instance, different techniques are available to make digital images of the raw data. At present we use three types of scanners for making these images. We have the ambition to make all digitized data freely available together (if feasible) with the digital images of the raw data.
Another example of a difference in implementation is a software program that is being developed at KNMI for automatically tracing curves in strip charts. These charts have first been transferred into high-resolution digital images. The meteorological archives around the world contain billions of strip charts from thermographs, barographs, hygrographs, pluviographs, etc. In general these charts contain much more quantitative information than extracted from them in the past. Automatically extracting this information would be of major interest.
An important aspect that has to be considered in data rescue and digitization projects is the durability of the work, how the digital data is archived and how future format changes will be tackled. KNMI is currently involved in discussing this subject with other institutions in the Netherlands.