meteologger — Meteorological logger utilities¶
This module is about meteorological loggers. Loggers are specialized
computers to which sensors are connected, and loggers log the
measurements of the sensors. The measurements logged in the logger are
eventually transferred to a computer (e.g. by GSM modems, or through
the internet, or by going to the site with your laptop, connecting it
to the logger, and copying the data) and written to a file. The
Datafile class provides functionality particularly for
inserting such data to the database.
The Datafile class is an abstract base class that is meant to be subclassed. It provides functionality that is common to all kinds of data files. However, each make of logger, or even each type of software used to unload data from a logger, provides a different kind of data file. Therefore, Datafile subclasses are specialized, each to a specific kind of file.
Datafile(base_url, cookies, datafiledict[, logger=None])¶
A Datafile instance should never be constructed; instead, a subclass should be constructed; however, the call for constructing any subclass has the same arguments: base_url is the location of the target Enhydris installation. cookies is the value returned from
enhydris_api.login(). datafiledict is a dictionary containing values for
datafile_fields, and optionally for
date_format. The logger argument has nothing to do with meteorological logger; it is a Logger object to which error, progress, and debugging information is written.
Datafilehas the following attributes and methods:
The pathname to the data file.
A string representation of error values, treated as NaN (null value). e.g. -6999
A comma-separated list of integers representing the ids of the time series to which the data file fields correspond; a zero indicates that the field is to be ignored. The first number corresponds to the first field after the date (and other fixed fields, such as the possible subset identifier; which are those fields depends on the data file format, that is, the specific
Datafilesubclass) and should be the id of the corresponding time series, or zero if the field is dummy; the second number correspondes to the second field after the fixed fields, and so on.
This is used only in the simple format; it’s an integer that represents a number of fields before the date and time that should be ignored. The default is zero. If, for example, the date and time are preceeded by a record id, set
nfields_to_ignore=1to ignore the record id.
This is used only on some
Datafilesubclasses. Some file formats mix two or more sets of measurements in the same file; for example, there may be ten-minute and hourly measurements in the same file, and for every 6 lines with ten-minute measurements there may be an additional line with hourly measurements (not necessarily the same variables). Such files have one or more additional distinguishing fields in each line, which helps to distinguish which set it is. We call these fields, which depend on the specific data file format, the subset identifiers.
Datafile(in fact its subclass) processes only one set of lines each time, and subset_identifiers specifies which subset it is. subset_identifiers is a comma-separated list of identifiers, and will cause
Datafile(in fact its subclass) to ignore lines with different subset identifiers.
Some file formats may be dependent on regional settings; these formats (i.e. these
date_formatis specified in the same way as for strftime.
For each time series specified in the
datafile_fields, retrieve the end date for the time series from the database, scan the data file, determine which is the first record of the time series not already stored in the database, and append that record and all subsequent record for the database. This is done for all time series specified in
The changes are not committed; the caller must commit them.
This is only meant to be used internally, i.e. called by subclasses whenever an error is found in a data file. The method raises an exception. line and msg are strings used in the error message.
Datafilesubclasses need to define the following methods:
Trueif line matches the
subset_identifiers. The base method always returns
True, and subclasses only need to redefine it if the file format has subsets.