How to create arrays of records

In Awkward Array, a “record” is a structure containing a fixed-length set of typed, possibly named fields. This is a “struct” in C or an “object” in Python (though the association of executable methods to record types is looser than the binding of methods to classes in object oriented languages).

All methods in Awkward Array are implemented as “structs of arrays,” rather than arrays of structs, so making and breaking records are inexpensive operations that you can perform frequently in data analysis.

import awkward as ak
import numpy as np

From a list of Python dicts

Records have a natural representation in JSON and Python as dicts, but only if all dicts in a series have the same set of field names. The ak.Array invokes ak.from_iter whenever presented with a list (or other non-string, non-dict iterable).

python_dicts = [
    {"x": 1, "y": 1.1, "z": "one"},
    {"x": 2, "y": 2.2, "z": "two"},
    {"x": 3, "y": 3.3, "z": "three"},
    {"x": 4, "y": 4.4, "z": "four"},
    {"x": 5, "y": 5.5, "z": "five"},
]
python_dicts
[{'x': 1, 'y': 1.1, 'z': 'one'},
 {'x': 2, 'y': 2.2, 'z': 'two'},
 {'x': 3, 'y': 3.3, 'z': 'three'},
 {'x': 4, 'y': 4.4, 'z': 'four'},
 {'x': 5, 'y': 5.5, 'z': 'five'}]
awkward_array = ak.Array(python_dicts)
awkward_array
<Array [{x: 1, y: 1.1, ... z: 'five'}] type='5 * {"x": int64, "y": float64, "z":...'>

It is important that all of the dicts in the series have the same set of field names, since Awkward Array has to identify all of the records as having a single type:

awkward_array.type
5 * {"x": int64, "y": float64, "z": string}

That is to say, an array of records is not a mapping from one type to another, such as from strings to numbers. The above record has exactly three fields: x, y, and z, and they have fixed types: int64, float64, and string. A mapping could have a variable set of keys, but the value type would have to be uniform.

If you try to mix field types in ak.from_iter (ultimately from ak.ArrayBuilder), the union of all sets of fields will be assumed, and any that aren’t filled in every item will be presumed “missing.”

array = ak.Array([
    {"a": 1, "b": 1, "c": 1},
    {"b": 2, "c": 2},
    {"c": 3, "d": 3, "e": 3, "f": 3},
    {"c": 4},
])
array
<Array [{a: 1, b: 1, c: 1, ... f: None}] type='4 * {"a": ?int64, "b": ?int64, "c...'>
array.tolist()
[{'a': 1, 'b': 1, 'c': 1, 'd': None, 'e': None, 'f': None},
 {'a': None, 'b': 2, 'c': 2, 'd': None, 'e': None, 'f': None},
 {'a': None, 'b': None, 'c': 3, 'd': 3, 'e': 3, 'f': 3},
 {'a': None, 'b': None, 'c': 4, 'd': None, 'e': None, 'f': None}]
array.type
4 * {"a": ?int64, "b": ?int64, "c": int64, "d": ?int64, "e": ?int64, "f": ?int64}

From a single dict of columns

If a single dict is passed to ak.Array, it will be interpreted as a set of columns, like Pandas’s DataFrame constructor. This is to provide a familiar interface to Pandas users.

from_columns = ak.Array({"x": [1, 2, 3, 4, 5], "y": [1.1, 2.2, 3.3, 4.4, 5.5], "z": ["one", "two", "three", "four", "five"]})
from_columns
<Array [{x: 1, y: 1.1, ... z: 'five'}] type='5 * {"x": int64, "y": float64, "z":...'>
from_columns.tolist()
[{'x': 1, 'y': 1.1, 'z': 'one'},
 {'x': 2, 'y': 2.2, 'z': 'two'},
 {'x': 3, 'y': 3.3, 'z': 'three'},
 {'x': 4, 'y': 4.4, 'z': 'four'},
 {'x': 5, 'y': 5.5, 'z': 'five'}]
from_columns.type
5 * {"x": int64, "y": float64, "z": string}

This is not the same as calling ak.from_iter on the same input, which could not be a valid ak.Array because a single dict would be an ak.Record.

from_rows = ak.from_iter({"x": [1, 2, 3, 4, 5], "y": [1.1, 2.2, 3.3, 4.4, 5.5], "z": ["one", "two", "three", "four", "five"]})
from_rows
<Record ... 'two', 'three', 'four', 'five']} type='{"x": var * int64, "y": var *...'>
from_rows.tolist()
{'x': [1, 2, 3, 4, 5],
 'y': [1.1, 2.2, 3.3, 4.4, 5.5],
 'z': ['one', 'two', 'three', 'four', 'five']}
from_rows.type
{"x": var * int64, "y": var * float64, "z": var * string}

Using ak.zip

The ak.zip function combines columns into an array of records, similar to the Pandas-style constructor described above.

from_columns = ak.zip({"x": [1, 2, 3, 4, 5], "y": [1.1, 2.2, 3.3, 4.4, 5.5], "z": ["one", "two", "three", "four", "five"]})
from_columns
<Array [{x: 1, y: 1.1, ... z: 'five'}] type='5 * {"x": int64, "y": float64, "z":...'>
from_columns.tolist()
[{'x': 1, 'y': 1.1, 'z': 'one'},
 {'x': 2, 'y': 2.2, 'z': 'two'},
 {'x': 3, 'y': 3.3, 'z': 'three'},
 {'x': 4, 'y': 4.4, 'z': 'four'},
 {'x': 5, 'y': 5.5, 'z': 'five'}]
from_columns.type
5 * {"x": int64, "y": float64, "z": string}

The difference is that ak.zip attempts to nested lists deeply, up to a depth_limit.

Given columns with nested lists:

zipped = ak.zip({"x": ak.Array([[1, 2, 3], [], [4, 5]]), "y": ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])})
zipped
<Array [[{x: 1, y: 1.1}, ... x: 5, y: 5.5}]] type='3 * var * {"x": int64, "y": f...'>
zipped.tolist()
[[{'x': 1, 'y': 1.1}, {'x': 2, 'y': 2.2}, {'x': 3, 'y': 3.3}],
 [],
 [{'x': 4, 'y': 4.4}, {'x': 5, 'y': 5.5}]]

By contrast, the same input to ak.Array’s Pandas-style constructor keeps nested lists separate.

not_zipped = ak.Array({"x": ak.Array([[1, 2, 3], [], [4, 5]]), "y": ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])})
not_zipped
<Array [{x: [1, 2, 3], ... y: [4.4, 5.5]}] type='3 * {"x": var * int64, "y": var...'>
not_zipped.tolist()
[{'x': [1, 2, 3], 'y': [1.1, 2.2, 3.3]},
 {'x': [], 'y': []},
 {'x': [4, 5], 'y': [4.4, 5.5]}]

The difference can be seen in a comparison of their types:

zipped.type, not_zipped.type
(3 * var * {"x": int64, "y": float64},
 3 * {"x": var * int64, "y": var * float64})

Also, ak.zip can build records without field names, also known as “tuples.”

tuples = ak.zip((ak.Array([[1, 2, 3], [], [4, 5]]), ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])))
tuples
<Array [[(1, 1.1), (2, 2.2, ... ), (5, 5.5)]] type='3 * var * (int64, float64)'>
tuples.tolist()
[[(1, 1.1), (2, 2.2), (3, 3.3)], [], [(4, 4.4), (5, 5.5)]]
tuples.type
3 * var * (int64, float64)

Functions that return lists of pairs, such as ak.cartesian and ak.combinations, also use the tuple type.

one = ak.Array([[1, 2, 3], [], [4, 5], [6]])
two = ak.Array([["a", "b"], ["c"], ["d"], ["e", "f"]])

ak.cartesian((one, two))
<Array [[(1, 'a'), (1, 'b', ... ), (6, 'f')]] type='4 * var * (int64, string)'>
ak.combinations(one, 2)
<Array [[(1, 2), (1, 3), (2, ... [(4, 5)], []] type='4 * var * (int64, int64)'>

Record names

In addition to optional field names, record types can also have names.

ak.Array([
    {"x": 1, "y": 1.1},
    {"x": 2, "y": 2.2},
    {"x": 3, "y": 3.3},
    {"x": 4, "y": 4.4},
    {"x": 5, "y": 5.5},
], with_name="XYZ")
<Array [{x: 1, y: 1.1}, ... {x: 5, y: 5.5}] type='5 * XYZ["x": int64, "y": float64]'>
ak.zip(
    (ak.Array([[1, 2, 3], [], [4, 5]]), ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])),
    with_name="AB",
)
<Array [[(1, 1.1), (2, 2.2, ... ), (5, 5.5)]] type='3 * var * AB[int64, float64]'>
ak.cartesian({"L": ak.Array([1, 2, 3]), "R": ak.Array(["a", "b"])}, with_name="LeftRight", axis=0)
<Array [{L: 1, R: 'a'}, ... {L: 3, R: 'b'}] type='6 * LeftRight["L": int64, "R":...'>

Names are for giving records specialized behavior through the ak.behavior registry. These are like attaching methods to a class in the sense that all records with a particular name can be given Python properties and methods.

class XYZRecord(ak.Record):
    def __repr__(self):
        return f"(X={self.x}:Y={self.y})"

class XYZArray(ak.Array):
    def diff(self):
        return abs(self.x - self.y)

ak.behavior["XYZ"] = XYZRecord
ak.behavior["*", "XYZ"] = XYZArray
ak.behavior["__typestr__", "XYZ"] = "XYZ"
ak.behavior[np.sqrt, "XYZ"] = lambda xyz: np.sqrt(xyz.x)

array = ak.Array([
    {"x": 1, "y": 1.1},
    {"x": 2, "y": 2.2},
    {"x": 3, "y": 3.3},
    {"x": 4, "y": 4.4},
    {"x": 5, "y": 5.5},
], with_name="XYZ")
array
<XYZArray [(X=1:Y=1.1), ... (X=5:Y=5.5)] type='5 * XYZ'>
array.diff()
<Array [0.1, 0.2, 0.3, 0.4, 0.5] type='5 * float64'>
array[0]
(X=1:Y=1.1)
np.sqrt(array)
<Array [1, 1.41, 1.73, 2, 2.24] type='5 * float64'>

With ArrayBuilder

ak.ArrayBuilder is described in more detail in this tutorial, but you can also construct arrays of records using the begin_record/end_record methods or the record context manager.

(This is what ak.from_iter uses internally to accumulate records.)

builder = ak.ArrayBuilder()

builder.begin_record()
builder.field("x").append(1)
builder.field("y").append(1.1)
builder.end_record()

builder.begin_record()
builder.field("x").append(2)
builder.field("y").append(2.2)
builder.end_record()

builder.begin_record()
builder.field("x").append(3)
builder.field("y").append(3.3)
builder.end_record()

array = builder.snapshot()
array
<Array [{x: 1, y: 1.1}, ... {x: 3, y: 3.3}] type='3 * {"x": int64, "y": float64}'>
builder = ak.ArrayBuilder()

with builder.record():
    builder.field("x").append(1)
    builder.field("y").append(1.1)

with builder.record():
    builder.field("x").append(2)
    builder.field("y").append(2.2)

with builder.record():
    builder.field("x").append(3)
    builder.field("y").append(3.3)

array = builder.snapshot()
array
<Array [{x: 1, y: 1.1}, ... {x: 3, y: 3.3}] type='3 * {"x": int64, "y": float64}'>

The begin_record method and record context manager can take a name, which allows you to name your records on the spot.

builder = ak.ArrayBuilder()

for i in range(3):
    with builder.record("XY"):
        builder.field("x").append(i)
        builder.field("y").append(i * 1.1)

array = builder.snapshot()
array
<Array [{x: 0, y: 0}, ... {x: 2, y: 2.2}] type='3 * XY["x": int64, "y": float64]'>

Record names also let you decide between two ways of dealing with changing sets of fields among the input records.

  • The default/generic way: all records at a given level are taken to belong to the same type and some instances of that type are missing fields.

  • Named records: different names mean different types, whether they have the same fields or not. Different record names at the same level result in a union type.

Here is an illustration of that difference.

def fill_A(builder, use_name):
    with builder.record("A" if use_name else None):
        builder.field("x").append(len(builder))
        builder.field("y").append(len(builder) * 1.1)

def fill_B(builder, use_name):
    with builder.record("B" if use_name else None):
        builder.field("y").append(len(builder) * 1.1)
        builder.field("z").append(str(len(builder)))

Without names:

builder = ak.ArrayBuilder()

fill_A(builder, use_name=False)
fill_A(builder, use_name=False)

fill_B(builder, use_name=False)
fill_B(builder, use_name=False)

fill_A(builder, use_name=False)

without_names = builder.snapshot()
without_names
<Array [{x: 0, y: 0, z: None, ... z: None}] type='5 * {"x": ?int64, "y": float64...'>
without_names.tolist()
[{'x': 0, 'y': 0.0, 'z': None},
 {'x': 1, 'y': 1.1, 'z': None},
 {'x': None, 'y': 2.2, 'z': '2'},
 {'x': None, 'y': 3.3000000000000003, 'z': '3'},
 {'x': 4, 'y': 4.4, 'z': None}]

With names:

builder = ak.ArrayBuilder()

fill_A(builder, use_name=True)
fill_A(builder, use_name=True)

fill_B(builder, use_name=True)
fill_B(builder, use_name=True)

fill_A(builder, use_name=True)

with_names = builder.snapshot()
with_names
<Array [{x: 0, y: 0}, ... {x: 4, y: 4.4}] type='5 * union[A["x": int64, "y": flo...'>
with_names.tolist()
[{'x': 0, 'y': 0.0},
 {'x': 1, 'y': 1.1},
 {'y': 2.2, 'z': '2'},
 {'y': 3.3000000000000003, 'z': '3'},
 {'x': 4, 'y': 4.4}]

The difference can be seen in the type: without_names has only one record type, but the x and z fields are optional, and with_names has a union of two record types, neither of which have optional fields.

without_names.type
5 * {"x": ?int64, "y": float64, "z": option[string]}
with_names.type
5 * union[A["x": int64, "y": float64], B["y": float64, "z": string]]

In Numba

Functions that Numba Just-In-Time (JIT) compiles can use ak.ArrayBuilder to accumulate records or columns can be accumulated in the compiled part and later combined with ak.zip. Combining columns outside of a JIT-compiled function is generally faster.

(At this time, Numba can’t use context managers, the with statement, in fully compiled code. ak.ArrayBuilder can’t be constructed or converted to an array using snapshot inside a JIT-compiled function, but can be outside the compiled context. Similarly, ak.* functions like ak.zip can’t be called inside a JIT-compiled function, but can be outside.)

import numba as nb
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/tmp/ipykernel_1931/977659880.py in <module>
----> 1 import numba as nb

~/python3.8/lib/python3.8/site-packages/numba/__init__.py in <module>
    196 
    197 _ensure_llvm()
--> 198 _ensure_critical_deps()
    199 
    200 # we know llvmlite is working as the above tests passed, import it now as SVML

~/python3.8/lib/python3.8/site-packages/numba/__init__.py in _ensure_critical_deps()
    136         raise ImportError("Numba needs NumPy 1.17 or greater")
    137     elif numpy_version > (1, 20):
--> 138         raise ImportError("Numba needs NumPy 1.20 or less")
    139 
    140     try:

ImportError: Numba needs NumPy 1.20 or less
@nb.jit
def append_record(builder, i):
    builder.begin_record()
    builder.field("x").append(i)
    builder.field("y").append(i * 1.1)
    builder.end_record()

@nb.jit
def example(builder):
    append_record(builder, 1)
    append_record(builder, 2)
    append_record(builder, 3)
    return builder

builder = example(ak.ArrayBuilder())

array = builder.snapshot()
array
@nb.jit
def faster_example():
    x = np.empty(3, np.int64)
    y = np.empty(3, np.float64)
    x[0] = 1
    y[0] = 1.1
    x[1] = 2
    y[1] = 2.2
    x[2] = 3
    y[2] = 3.3
    return x, y

array = ak.zip(dict(zip(["x", "y"], faster_example())))
array

Combining columns into arrays is a metadata-only operation, which does not scale with the size of the dataset. The second example is faster because it only requires Numba to fill NumPy arrays, which it’s designed for, without involving the machinery of ak.ArrayBuilder to identify types at runtime.