casacore
TiledDataStMan.h
Go to the documentation of this file.
1 //# TiledDataStMan.h: Tiled Data Storage Manager
2 //# Copyright (C) 1995,1996,1997,1999,2001
3 //# Associated Universities, Inc. Washington DC, USA.
4 //#
5 //# This library is free software; you can redistribute it and/or modify it
6 //# under the terms of the GNU Library General Public License as published by
7 //# the Free Software Foundation; either version 2 of the License, or (at your
8 //# option) any later version.
9 //#
10 //# This library is distributed in the hope that it will be useful, but WITHOUT
11 //# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
12 //# FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public
13 //# License for more details.
14 //#
15 //# You should have received a copy of the GNU Library General Public License
16 //# along with this library; if not, write to the Free Software Foundation,
17 //# Inc., 675 Massachusetts Ave, Cambridge, MA 02139, USA.
18 //#
19 //# Correspondence concerning AIPS++ should be addressed as follows:
20 //# Internet email: aips2-request@nrao.edu.
21 //# Postal address: AIPS++ Project Office
22 //# National Radio Astronomy Observatory
23 //# 520 Edgemont Road
24 //# Charlottesville, VA 22903-2475 USA
25 //#
26 //# $Id$
27 
28 #ifndef TABLES_TILEDDATASTMAN_H
29 #define TABLES_TILEDDATASTMAN_H
30 
31 //# Includes
32 #include <casacore/casa/aips.h>
33 #include <casacore/tables/DataMan/TiledStMan.h>
34 #include <casacore/casa/Containers/Block.h>
35 #include <casacore/casa/BasicSL/String.h>
36 
37 namespace casacore { //# NAMESPACE CASACORE - BEGIN
38 
39 //# Forward Declarations
40 
41 
42 // <summary>
43 // Tiled Data Storage Manager.
44 // </summary>
45 
46 // <use visibility=export>
47 
48 // <reviewed reviewer="UNKNOWN" date="before2004/08/25" tests="">
49 // </reviewed>
50 
51 // <prerequisite>
52 //# Classes you should understand before using this one.
53 // <li> <linkto class=TiledStMan>TiledStMan</linkto>
54 // <li> <linkto class=TSMCube>TSMCube</linkto>
55 // <li> <linkto class=ROTiledStManAccessor>ROTiledStManAccessor</linkto>
56 // for a discussion of the maximum cache size
57 // <li> <linkto class=Record>Record</linkto>
58 // </prerequisite>
59 
60 // <etymology>
61 // TiledDataStMan is the Tiled Storage Manager for general
62 // data arrays.
63 // </etymology>
64 
65 // <synopsis>
66 // TiledDataStMan is a derivation from TiledStMan, the abstract
67 // tiled storage manager class. A description of the basics
68 // of tiled storage managers is given in the
69 // <linkto module=Tables:TiledStMan>Tables module</linkto> description.
70 // <p>
71 // TiledDataStMan allows the user explicit control over the
72 // definition and extension of hypercubes by means of the accessor
73 // class <linkto class=TiledDataStManAccessor>TiledDataStManAccessor</linkto>.
74 // The user can determine which row should be put in which hypercube,
75 // so it is possible to put row 0-9 in hypercube A, row 10-29 in B,
76 // row 30-39 in A again, etc.. This makes it possible to use a tiled
77 // storage manager for a data column containing data with
78 // different shapes (e.g. line and continuum data). Actually,
79 // this storage manager is developed for irregularly shaped
80 // UV-data, but can be used for any purpose.
81 // <br>
82 // Each extensible hypercube uses a file of its own. This means that there
83 // shouldn't be too many of them, otherwise the number of files may
84 // get too high.
85 // <p>
86 // The TiledDataStMan has the following (extra) properties:
87 // <ul>
88 // <li> When multiple hypercubes are used, one or more id columns have
89 // to be used to differentiate between them. The id values must
90 // be defined when the hypercube gets added; they cannot be put
91 // explicitly.
92 // <li> A hypercube can be extensible in its last dimension by setting
93 // its last dimension to zero. In that case extendHypercube can
94 // be used to extend the hypercube when needed.
95 // All fixed sized hypercubes are stored in one file, while there
96 // is one file per extensible hypercube.
97 // <li> The table must be large enough to accommodate the addition
98 // or extension of a hypercube. This means that a sufficient
99 // number of rows must be added to the table before a hypercube
100 // can be added or extended. It is the responsibility of the user
101 // to "synchronize" addition of rows and hypercubes.
102 // <li> It is possible to define coordinates for the hypercube axes
103 // in several ways:
104 // <ul>
105 // <li> Use the TiledDataStMan storage manager to hold their values
106 // and define the coordinates when adding or extending the
107 // hypercube. This is the preferred way.
108 // <li> As above, but use explicit puts to write their values.
109 // This has to be used when coordinates are defined after
110 // the hypercube has been added or extended.
111 // Note that several rows may share the same value, so
112 // overwriting a value may affect multiple rows.
113 // <li> Use another storage manager to hold their values.
114 // This is useful when their values depend on other axes,
115 // because that cannot be handled by TiledDataStMan.
116 // </ul>
117 // Note that it is possible to store one coordinate column with
118 // TiledDataStMan and another with another storage manager.
119 // </ul>
120 // </synopsis>
121 
122 // <motivation>
123 // This tiled storage manager allows one to create and extend hypercubes
124 // as needed. One has complete control over which row is stored in which
125 // hypercube.
126 // </motivation>
127 
128 // <example>
129 // The following example shows how to create a TiledDataStMan tiled
130 // storage manager using the hypercolumn as defined in the table description.
131 // Furthermore it shows how to use TiledDataStManAccessor
132 // to add a hypercube, while defining its tile shape, coordinates,
133 // and id-value.
134 // The example shows that reading the data back does not require any knowledge
135 // of the data manager. It's exactly the same if another data manager was used.
136 // <br>
137 // The table created contains the equally shaped data columns "Data" and
138 // "Weight".
139 // Each cell in those columns contains a 2D array with shape [12,20]. The
140 // coordinates of those arrays are "Pol" and "Freq".
141 // The tiled storage manager superimposes two more axes ("Baseline"and "Time")
142 // on the data resulting in a 4D hypercube with shape [12,20,30,42].
143 // The table contains 42*30 rows (which has to be equal to the number of
144 // elements in the superimposed axes).
145 // <br>
146 // The tile shape of the hypercube is (arbitrarily) set to [4,5,6,7].
147 // Of course, any tile shape could be chosen. This tile shape results
148 // in a tile size of 6720 bytes (4*5*6*7 *(4+4) bytes), which is not
149 // that large (32768 as tile size is very reasonable). The number of tiles
150 // is integral in each dimension, so no space is wasted.
151 // Finally it makes access along the various axes about equally efficient.
152 // <br>
153 // Although in this example only one hypercube is added, multiple hypercubes
154 // are possible, because an id column has been defined.
155 // <note role=caution>
156 // The example uses the global Array function indgen to fill the data
157 // and coordinate arrays with arbitrary values.
158 // </note>
159 // Note that the description of class
160 // <linkto class=ROTiledStManAccessor>ROTiledStManAccessor</linkto>
161 // contains a discussion about the effect of setting the maximum cache size.
162 //
163 // <srcblock>
164 // // Define the table description and the columns in it.
165 // TableDesc td ("", "1", TableDesc::Scratch);
166 // td.addColumn (ScalarColumnDesc<float> ("Time"));
167 // td.addColumn (ScalarColumnDesc<float> ("Baseline"));
168 // td.addColumn (ArrayColumnDesc<float> ("Pol", 1));
169 // td.addColumn (ArrayColumnDesc<float> ("Freq", 1));
170 // td.addColumn (ScalarColumnDesc<String> ("Id"));
171 // td.addColumn (ArrayColumnDesc<float> ("Data", 2));
172 // td.addColumn (ArrayColumnDesc<float> ("Weight", 2));
173 // // Define the 4-dim hypercolumn with its data, coordinate and id columns.
174 // td.defineHypercolumn ("TSMExample",
175 // 4,
176 // stringToVector ("Data,Weight"),
177 // stringToVector ("Pol,Freq,Baseline,Time"),
178 // stringToVector ("Id"));
179 //
180 // // Now create a new table from the description.
181 // SetupNewTable newtab("tTiledDataStMan_tmp.data", td, Table::New);
182 // // Create a TiledDataStMan storage manager for the hypercolumn
183 // // and bind the columns to it.
184 // TiledDataStMan sm1 ("TSMExample");
185 // newtab.bindAll (sm1);
186 // // Create the table with 42*30 rows.
187 // Table table(newtab, 42*30);
188 // // Create the accessor to be able to add a hypercube to this
189 // // storage manager.
190 // TiledDataStManAccessor accessor(table, "TSMExample");
191 // // Define the values for the coordinates of the hypercube
192 // // and put them into the record.
193 // Vector<float> timeValues(42);
194 // Vector<float> baselineValues(30);
195 // Vector<float> freqValues(20);
196 // Vector<float> polValues(12);
197 // indgen (timeValues);
198 // indgen (baselineValues, float(100));
199 // indgen (freqValues, float(200));
200 // indgen (polValues, float(300));
201 // Record hyperDef;
202 // hyperDef.define ("Time", timeValues);
203 // hyperDef.define ("Baseline", baselineValues);
204 // hyperDef.define ("Freq", freqValues);
205 // hyperDef.define ("Pol", polValues);
206 // // Define the id value as well.
207 // hyperDef.define ("Id", "");
208 // // Now add the hypercube with the given shape, tile shape,
209 // // and coordinate and id values.
210 // accessor.addHypercube (IPosition(4,12,20,30,42),
211 // IPosition(4,4,5,6,7), hyperDef);
212 // ArrayColumn<float> data (table, "Data");
213 // ArrayColumn<float> weight (table, "Weight");
214 // Matrix<float> array(IPosition(2,12,20));
215 // uInt i;
216 // indgen (array);
217 // // Write some data into the data columns.
218 // for (i=0; i<30*42; i++) {
219 // data.put (i, array);
220 // weight.put (i, array+float(100));
221 // array += float(200);
222 // }
223 // // Prepare for reading the data back.
224 // // Note that time and baseline are in fact scalar columns. They are
225 // // superimposed dimensions on the hypercube.
226 // ScalarColumn<float> time (table, "Time");
227 // ScalarColumn<float> baseline (table, "Baseline");
228 // ArrayColumn<float> freq (table, "Freq");
229 // ArrayColumn<float> pol (table, "Pol");
230 // ScalarColumn<String> id (table, "Id");
231 // float fValue;
232 // String sValue;
233 // for (i=0; i<table.nrow(); i++) {
234 // data.get (i, array);
235 // weight.get (i, array);
236 // pol.get (i, polValues);
237 // freq.get (i, freqValues);
238 // baseline.get (i, fValue);
239 // time.get (i, fValue);
240 // id.get (i, sValue);
241 // }
242 // </srcblock>
243 // Note that in this example an id column was not necessary, because
244 // there is only one hypercube.
245 // <p>
246 // The following example is more advanced. Two (extensible) hypercubes
247 // are used for line and continuum data. Writing such a data set
248 // could be done as shown. Reading it back is the same as above.
249 // <br>
250 // In this example the data columns contain line and continuum data.
251 // So there are two types of data, each with their own shape and
252 // stored in their own (extensible) hypercube. Note that the last
253 // dimension of the hypercube shape is set to zero (to make extensible),
254 // but the last tile shape dimension has been filled in,
255 // because the exact tile shape must be known.
256 // <br>
257 // Before each put of the data the appropriate hypercube is extended.
258 // Also the time has to be put, which is done (as an example) in
259 // two different ways (using an explicit put and using the extendHypercube).
260 //
261 // <srcblock>
262 // // Defining TableDesc and storage manager is same as in first example.
263 // // Create the table.
264 // Table table(newtab);
265 // // Create the accessor to be able to add the hypercubes to this
266 // // storage manager.
267 // TiledDataStManAccessor accessor(table, "TSMExample");
268 // // Fill the coordinate values.
269 // // Note that the time axis of the hypercube will have length 0 to
270 // // make it extensible. Therefore the time coordinate can only be
271 // // filled in when the hypercube is extended.
272 // Vector<float> baselineValues(30);
273 // Vector<float> freqValuesCont(1);
274 // Vector<float> freqValuesLine(20);
275 // Vector<float> polValues(4);
276 // indgen (baselineValues, float(100));
277 // indgen (freqValuesLine, float(200));
278 // indgen (freqValuesCont, float(200));
279 // indgen (polValues, float(300));
280 // Record hyperDefLine;
281 // hyperDefLine.define ("Baseline", baselineValues);
282 // hyperDefLine.define ("Pol", polValues);
283 // // Make similar record for line data.
284 // // Fill the correct id and frequency values for each type.
285 // // Add the 2 hypercubes.
286 // Record hyperDefCont (hyperDefLine);
287 // hyperDefLine.define ("Id", "L");
288 // hyperDefLine.define ("Freq", freqValuesLine);
289 // hyperDefCont.define ("Id", "C");
290 // hyperDefCont.define ("Freq", freqValuesCont);
291 // // Add the hypercubes.
292 // // Define their last dimension as zero to make them extensible.
293 // accessor.addHypercube (IPosition(4,4,20,30,0),
294 // IPosition(4,4,5,6,7), hyperDefLine);
295 // accessor.addHypercube (IPosition(4,4,1,30,0),
296 // IPosition(4,4,1,6,7), hyperDefCont);
297 // ScalarColumn<float> time (table, "Time");
298 // ScalarColumn<float> baseline (table, "Baseline");
299 // ArrayColumn<float> freq (table, "Freq");
300 // ArrayColumn<float> pol (table, "Pol");
301 // ArrayColumn<float> data (table, "Data");
302 // ArrayColumn<float> weight (table, "Weight");
303 // Matrix<float> arrayLine(IPosition(2,4,20));
304 // Matrix<float> arrayCont(IPosition(2,4,1));
305 // indgen (arrayLine);
306 // indgen (arrayCont);
307 // // Write some data into the data columns.
308 // // Alternately line and continuum is written.
309 // // Each hypercube requires 30 rows to be added (i.e. nr of baselines).
310 // // The last dimension of each hypercube is extended with 1.
311 // uInt i, j;
312 // uInt rownr = 0;
313 // for (i=0; i<42; i++) {
314 // if (i%2 == 0) {
315 // table.addRow (30);
316 // accessor.extendHypercube (1, hyperDefLine);
317 // time.put (rownr, float(i));
318 // for (j=0; j<30; j++) {
319 // data.put (rownr, arrayLine);
320 // weight.put (rownr, arrayLine);
321 // rownr++;
322 // }
323 // }else{
324 // table.addRow (30);
325 // Vector<float> timeValue(1);
326 // timeValue(0) = float(i);
327 // hyperDefCont.define ("Time", timeValue);
328 // accessor.extendHypercube (1, hyperDefCont);
329 // time.put (rownr, float(i));
330 // for (j=0; j<30; j++) {
331 // data.put (rownr, arrayCont);
332 // weight.put (rownr, arrayCont);
333 // rownr++;
334 // }
335 // }
336 // }
337 // </srcblock>
338 // Note that in this example the time is defined in 2 different ways.
339 // The first one by an explicit put, the second one as a record in
340 // the extendHypercube call. The second way if the preferred one,
341 // although it requires a bit more coding.
342 // </example>
343 
344 //# <todo asof="$DATE:$">
345 //# A List of bugs, limitations, extensions or planned refinements.
346 //# </todo>
347 
348 
350 {
352 
353 public:
354  // Create a TiledDataStMan storage manager for the hypercolumn
355  // with the given name.
356  // The hypercolumn name is also the name of the storage manager.
357  // The given maximum cache size (default is unlimited) is persistent,
358  // thus will be reused when the table is read back. Note that the class
359  // <linkto class=ROTiledStManAccessor>ROTiledStManAccessor</linkto>
360  // allows one to overwrite the maximum cache size temporarily.
361  // <br>The constructor taking a Record expects fields in the record with
362  // the name of the arguments in uppercase. If not defined, their
363  // default value is used.
364  // <group>
365  TiledDataStMan (const String& hypercolumnName,
366  uInt maximumCacheSize = 0);
367  TiledDataStMan (const String& hypercolumnName,
368  const Record& spec);
369  // </group>
370 
371  ~TiledDataStMan();
372 
373  // Clone this object.
374  // It does not clone TSMColumn objects possibly used.
375  DataManager* clone() const;
376 
377  // Get the type name of the data manager (i.e. TiledDataStMan).
378  String dataManagerType() const;
379 
380  // Make the object from the type name string.
381  // This function gets registered in the DataManager "constructor" map.
383  const Record& spec);
384 
385 private:
386  // Create a TiledDataStMan.
387  // This constructor is private, because it should only be used
388  // by makeObject.
389  TiledDataStMan();
390 
391  // Forbid copy constructor.
393 
394  // Forbid assignment.
396 
397  // Add rows to the storage manager.
398  // This will only increase the number of rows. When a hypercube is
399  // added or extended, it will be checked whether the number of rows
400  // is sufficient.
401  void addRow (uInt nrrow);
402 
403  // Add a hypercube.
404  // The number of rows in the table must be large enough to
405  // accommodate this hypercube.
406  // The possible id values must be given in the record, while
407  // coordinate values are optional. The field names in the record
408  // should match the coordinate and id column names.
409  // The last dimension in the cube shape can be zero, indicating that
410  // the hypercube is extensible.
411  void addHypercube (const IPosition& cubeShape,
412  const IPosition& tileShape,
413  const Record& values);
414 
415  // Extend the hypercube with the given number of elements in
416  // the last dimension.
417  // The record should contain the id values (to get the correct
418  // hypercube) and optionally coordinate values for the elements added.
419  void extendHypercube (uInt incrInLastDim, const Record& values);
420 
421  // Get the hypercube in which the given row is stored.
422  virtual TSMCube* getHypercube (uInt rownr);
423 
424  // Get the hypercube in which the given row is stored.
425  // It also returns the position of the row in that hypercube.
426  virtual TSMCube* getHypercube (uInt rownr, IPosition& position);
427 
428  // Flush and optionally fsync the data.
429  // It returns a True status if it had to flush (i.e. if data have changed).
430  virtual Bool flush (AipsIO&, Bool fsync);
431 
432  // Let the storage manager create files as needed for a new table.
433  // This allows a column with an indirect array to create its file.
434  virtual void create (uInt nrrow);
435 
436  // Read the header info.
437  virtual void readHeader (uInt nrrow, Bool firstTime);
438 
439  // Update the map of row numbers to cube number plus offset.
440  void updateRowMap (uInt cubeNr, uInt incrInLastDim);
441 
442  // Check if the table is large enough to hold this
443  // hypercube extension.
444  void checkNrrow (const IPosition& cubeShape,
445  uInt incrInLastDim) const;
446 
447 
448  //# Declare the data members.
449  // The map of row number to cube and position in cube.
453  // The nr of elements used in the map blocks.
455  // The row number since the last hypercube extension.
457 };
458 
459 
460 
461 
462 } //# NAMESPACE CASACORE - END
463 
464 #endif
A Vector of integers, for indexing into Array<T> objects.
Definition: IPosition.h:119
void addHypercube(const IPosition &cubeShape, const IPosition &tileShape, const Record &values)
Add a hypercube.
Tiled hypercube in a table.
Definition: TSMCube.h:105
virtual void create(uInt nrrow)
Let the storage manager create files as needed for a new table.
AipsIO is the object persistency mechanism of Casacore.
Definition: AipsIO.h:168
void checkNrrow(const IPosition &cubeShape, uInt incrInLastDim) const
Check if the table is large enough to hold this hypercube extension.
Base class for Tiled Storage Manager classes.
Definition: TiledStMan.h:107
uInt nrUsedRowMap_p
The nr of elements used in the map blocks.
void extendHypercube(uInt incrInLastDim, const Record &values)
Extend the hypercube with the given number of elements in the last dimension.
const IPosition & tileShape(uInt rownr) const
Get the tile shape of the data in the given row.
virtual TSMCube * getHypercube(uInt rownr)
Get the hypercube in which the given row is stored.
virtual void readHeader(uInt nrrow, Bool firstTime)
Read the header info.
A hierarchical collection of named fields of various types.
Definition: Record.h:180
bool Bool
Define the standard types used by Casacore.
Definition: aipstype.h:42
TiledDataStMan & operator=(const TiledDataStMan &)
Forbid assignment.
TiledDataStMan()
Create a TiledDataStMan.
Give access to some TiledDataStMan functions.
String dataManagerType() const
Get the type name of the data manager (i.e.
uInt nrrowLast_p
The row number since the last hypercube extension.
static DataManager * makeObject(const String &dataManagerType, const Record &spec)
Make the object from the type name string.
Abstract base class for a data manager.
Definition: DataManager.h:222
Tiled Data Storage Manager.
String: the storage and methods of handling collections of characters.
Definition: String.h:223
Block< uInt > rowMap_p
The map of row number to cube and position in cube.
virtual Bool flush(AipsIO &, Bool fsync)
Flush and optionally fsync the data.
this file contains all the compiler specific defines
Definition: mainpage.dox:28
DataManager * clone() const
Clone this object.
unsigned int uInt
Definition: aipstype.h:51
uInt maximumCacheSize() const
Get the current maximum cache size (in bytes).
Definition: TiledStMan.h:533
void addRow(uInt nrrow)
Add rows to the storage manager.
void updateRowMap(uInt cubeNr, uInt incrInLastDim)
Update the map of row numbers to cube number plus offset.