Alexandria 2.31.0
SDC-CH common library for the Euclid project
Loading...
Searching...
No Matches
NdArray module

Table of Contents

Introduction

The NdArray module provides a multidimensional container storing its elements contiguously in memory in row-major order. For instance, a 3D array with the shape (3,2,4) contains 24 elements.

\[ \prod_{i=0}^{n}shape[i] \]

The order in memory for that array would be:

v[0,0,0], v[0,0,1], v[0,0,2], v[0,0,3], v[0,1,0], v[0,1,1], v[0,1,2], v[0,1,3] ... v[2,1,3]
Note
The NdArray class has one template parameters: the contained type. A second template parameter, used only at construction time, is a Container class (std::vector by default), which owns the memory used by NdArray. However, NdArray uses type erasure so it can be moved around regardless of the underlying storage. This allows to switch between RAM and memory mapped files transparently, without modifying any method that expects an NdArray. The underlying container is assumed to allocate a sequential address space.

Usage

The usage of this class is fairly straight forward. You need to define it specifying at least the contained type, and pass to the constructor either a shape, or a shape and initial data.

For convenience, you can pass the shape as a vector of size_t, or as a initializer list.

// Define a multidimensional array with three dimensions of size 3, 2 and 4
NdArray::NdArray<float> nd_array1{3,2,4};
// Same, but using a vector
NdArray::NdArray<float> nd_array2(std::vector<size_t>{3,2,4});
// Pre-initialize the content with the value 42.
std::vector<float> content(3 * 2 * 4, 32.);
NdArray::NdArray<float> nd_array3(std::vector<size_t>{3,2,4}, content);
// Avoid a copy
std::vector<float> content(3 * 2 * 4, 32.);
NdArray::NdArray<float> nd_array4(std::vector<size_t>{3,2,4}, std::move(content));
// Different way of initializing the content
NdArray::NdArray<float> nd_array5(std::vector<size_t>{3,2,4});
std::fill(nd_array5.begin(), nd_array5.end(), 42.);
T fill(T... args)
T move(T... args)

The individual elements can be accessed with the method at. Similarly to the constructor, it accepts either the axes as a list of parameters, or as a vector of size_t.

nd_array.at(1, 2, 3);
nd_array.at(std::vector<size_t>{1, 2, 3});

The version that accepts a list of parameters ultimately generates a call to the second via metaprogramming, so they are fully equivalent.

To get the array shape, just use the shape method.

nd_array.shape(); // std::vector<size_t>{3,2,4};

You can also get the total size (number of elements), and a reference to the underlying container.

nd_array.size(); // 3*2*4 = 24
nd_array.data();

NdArray can be reshaped as long as the new shape matches exactly the number of elements already contained within the array.

nd_array.reshape(24); // Now nd_array is a single row with 24 elements

Last, there is an overload of the operator << for std::ostream. This can be useful for debugging, but it is also necessary for the implementation of Euclid::Table::AsciiWriter, as it relies on the existence of this operator (technically, boost::lexical_cast does). The output has the form <shape>values.

std::cout << nd_array << std::endl;
//<3,2,4>42,42,42,42,42,42
T endl(T... args)

Npy files

Alexandria 2.17 adds support for numpy array files. For using them, you need to include the header NdArray/io/Npy.h, and NdArray/io/NpyMmap.h for memory mapped files. This allows the exchange of data between Alexandria based software and Python.

Note, however, that the support is limited to primitive types - ints of different sizes, floats, doubles - with native endianness. Structured arrays are not supported.

The generated files can be read by numpy in any architecture, as the metadata is properly initialized.

An example for reading a numpy array:

auto nd_array_int = readNpy<int64_t>("/tmp/myints.npy");
auto nd_array_double = readNpy<double>("/tmp/mydoubles.npy");

For writing:

writeNpy("/tmp/output.npy", ndarray);
void writeNpy(std::ostream &out, const NdArray< T > &array)

As you can notice, the type has to be known in advance.

Alexandria also supports memory mapped files both for reading and writing, although for creation the shape has to be known in advance:

auto nd_array_int_mmap = mmapNpy<int64_t>("/tmp/myints.npy");
auto nd_array_float_mmap = createMmapNpy<float>("/tmp/mynewfloats.npy", {1000, 1000, 2});