Direct constructors (fastest)

If you’re willing to think about your data in a columnar way, directly constructing layouts and wrapping them in ak.Array interfaces is the fastest way to make them. (All other methods do this at some level.)

“Thinking about data in a columnar way” is the crucial difference between this method and ArrayBuilder and LayoutBuilder. Both of the builders let you think about a data structure the way you would think about Python objects, in which all fields of a given record or elements of a list are “together” and one record or list is “separate” from another record or list. For example,

import awkward as ak
import numpy as np
builder = ak.ArrayBuilder()

with builder.list():
    with builder.record():
        builder.field("x").real(1.1)
        with builder.field("y").list():
            builder.integer(1)
    with builder.record():
        builder.field("x").real(2.2)
        with builder.field("y").list():
            builder.integer(1)
            builder.integer(2)
    with builder.record():
        builder.field("x").real(3.3)
        with builder.field("y").list():
            builder.integer(1)
            builder.integer(2)
            builder.integer(3)

with builder.list():
    pass

with builder.list():
    with builder.record():
        builder.field("x").real(4.4)
        with builder.field("y").list():
            builder.integer(3)
            builder.integer(2)
            
    with builder.record():
        builder.field("x").real(5.5)
        with builder.field("y").list():
            builder.integer(3)

array = builder.snapshot()
array
<Array [[{x: 1.1, y: [1]}, ... y: [3]}]] type='3 * var * {"x": float64, "y": var...'>
array.tolist()
[[{'x': 1.1, 'y': [1]}, {'x': 2.2, 'y': [1, 2]}, {'x': 3.3, 'y': [1, 2, 3]}],
 [],
 [{'x': 4.4, 'y': [3, 2]}, {'x': 5.5, 'y': [3]}]]

gets all of the items in the first list separately from the items in the second or third lists, and both fields of each record (x and y) are expressed near each other in the flow of the code and in the times when they’re appended.

By contrast, the physical data are laid out in columns, with all x values next to each other, regardless of which records or lists they’re in, and all the y values next to each other in another buffer.

array.layout
<ListOffsetArray64>
    <offsets><Index64 i="[0 3 3 5]" offset="0" length="4" at="0x000001ed7380"/></offsets>
    <content><RecordArray length="5">
        <field index="0" key="x">
            <NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001e8ef30"/>
        </field>
        <field index="1" key="y">
            <ListOffsetArray64>
                <offsets><Index64 i="[0 1 3 6 8 9]" offset="0" length="6" at="0x000001e8ef60"/></offsets>
                <content><NumpyArray format="l" shape="9" data="1 1 2 1 2 3 3 2 3" at="0x000001eb31a0"/></content>
            </ListOffsetArray64>
        </field>
    </RecordArray></content>
</ListOffsetArray64>

To build arrays using the layout constructors, you need to be able to write them in a form similar to the above, with the data already arranged as columns, in a tree representing the type structure of items in the array, not separate trees for each array element.

The Awkward Array library has a closed set of node types. Building the array structure you want will require you to understand the node types that you use.

The node types (with validity rules for each) are documented under ak.layout.Content, but this tutorial will walk through them explaining the situations in which you’d want to use each.

Content classes

ak.layout.Content is the abstract superclass of all node types. All Content nodes that you would create are concrete subclasses of this class. The superclass is useful for checking isinstance(some_object, ak.layout.Content), since there are some attributes that are only allowed to be Content nodes.

Sections in this document about a subclass of Content are named “Content >: XYZ” (using the “is superclass of” operator).

Parameters

Each layout node can have arbitrary metadata, called “parameters.” Some parameters have built-in meanings, which are described below, and others can be given meanings by defining functions in ak.behavior.

Index classes

ak.layout.Index instances are buffers of integers that are used to give structure to an array. For instance, the offsets in the ListOffsetArrays, above, are Indexes, but the NumpyArray of y list contents are not. ak.layout.NumpyArray is a subclass of ak.layout.Content, and ak.layout.Index is not. Indexes are more restricted than general NumPy arrays (must be one-dimensional, C-contiguous integers; dtypes are also prescribed) because they are frequently manipulated by Awkward Array operations, such as slicing.

There are five Index specializations, and each ak.layout.Content subclass has limitations on which ones it can use.

  • Index8: an Index of signed 8-bit integers

  • IndexU8: an Index of unsigned 8-bit integers

  • Index32: an Index of signed 32-bit integers

  • IndexU32: an Index of unsigned 32-bit integers

  • Index64: an Index of signed 64-bit integers

Content >: EmptyArray

ak.layout.EmptyArray is one of the two possible leaf types of a layout tree; the other is ak.layout.NumpyArray (A third, corner-case “leaf type” is a ak.layout.RecordArray with zero fields).

EmptyArray is a trivial node type: it can only represent empty arrays with unknown type.

ak.layout.EmptyArray()
<EmptyArray/>
ak.Array(ak.layout.EmptyArray())
<Array [] type='0 * unknown'>

Since this is such a simple node type, let’s use it to show examples of adding parameters.

ak.layout.EmptyArray(parameters={"name1": "value1", "name2": {"more": ["complex", "value"]}})
<EmptyArray>
    <parameters>
        <param key="name1">"value1"</param>
        <param key="name2">{"more": ["complex", "value"]}</param>
    </parameters>
</EmptyArray>

Content >: NumpyArray

ak.layout.NumpyArray is one of the two possible leaf types of a layout tree; the other is ak.layout.EmptyArray. (A third, corner-case “leaf type” is a ak.layout.RecordArray with zero fields)

NumpyArray represents data the same way as a NumPy np.ndarray. That is, it can be multidimensional, but only rectilinear arrays.

ak.layout.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5]))
<NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001ed6040"/>
ak.layout.NumpyArray(np.array([[1, 2, 3], [4, 5, 6]], np.int16))
<NumpyArray format="h" shape="2 3" data="0x 01000200 03000400 05000600" at="0x000001ddd2d0"/>
ak.layout.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])[::2])
<NumpyArray format="d" shape="3" strides="16" data="1.1 3.3 5.5" at="0x000001ec1d40"/>
ak.layout.NumpyArray(np.array([[1, 2, 3], [4, 5, 6]], np.int16)[:, 1:])
<NumpyArray format="h" shape="2 2" strides="6 2" data="0x 02000300 04000500 0600" at="0x000001a1bd22"/>

In most array structures, the NumpyArrays only need to be 1-dimensional, since regular-length dimensions can be represented by ak.layout.RegularArray and variable-length dimensions can be represented by ak.layout.ListArray or ak.layout.ListOffsetArray.

The ak.from_numpy function has a regulararray argument to choose between putting multiple dimensions into the NumpyArray node or nesting a 1-dimensional NumpyArray in RegularArray nodes.

ak.from_numpy(np.array([[1, 2, 3], [4, 5, 6]], np.int16), regulararray=False, highlevel=False)
<NumpyArray format="h" shape="2 3" data="0x 01000200 03000400 05000600" at="0x000001ed6150"/>
ak.from_numpy(np.array([[1, 2, 3], [4, 5, 6]], np.int16), regulararray=True, highlevel=False)
<RegularArray size="3">
    <content><NumpyArray format="h" shape="6" data="1 2 3 4 5 6" at="0x000001a03fd0"/></content>
</RegularArray>

All of these representations look the same in an ak.Array (high-level view).

ak.Array(ak.layout.NumpyArray(np.array([[1, 2, 3], [4, 5, 6]])))
<Array [[1, 2, 3], [4, 5, 6]] type='2 * 3 * int64'>
ak.Array(ak.layout.RegularArray(ak.layout.NumpyArray(np.array([1, 2, 3, 4, 5, 6])), 3))
<Array [[1, 2, 3], [4, 5, 6]] type='2 * 3 * int64'>

If you are producing arrays, you can pick any representation that is convenient. If you are consuming arrays, you need to be aware of the different representations.

Content >: RegularArray

ak.layout.RegularArray represents regular-length lists (lists with all the same length). This was shown above as being equivalent to dimensions in a ak.layout.NumpyArray, but it can also contain irregular data.

layout = ak.layout.RegularArray(
    ak.from_iter([1, 2, 3, 4, 5, 6], highlevel=False),
    3,
)
layout
<RegularArray size="3">
    <content><NumpyArray format="l" shape="6" data="1 2 3 4 5 6" at="0x000001e4a040"/></content>
</RegularArray>
ak.Array(layout)
<Array [[1, 2, 3], [4, 5, 6]] type='2 * 3 * int64'>
layout = ak.layout.RegularArray(
    ak.from_iter([[], [1], [1, 2], [1, 2, 3], [1, 2, 3, 4], [1, 2, 3, 4, 5]], highlevel=False),
    3,
)
layout
<RegularArray size="3">
    <content><ListOffsetArray64>
        <offsets><Index64 i="[0 0 1 3 6 10 15]" offset="0" length="7" at="0x000001edeba0"/></offsets>
        <content><NumpyArray format="l" shape="15" data="1 1 2 1 2 ... 1 2 3 4 5" at="0x000001e49f30"/></content>
    </ListOffsetArray64></content>
</RegularArray>
ak.Array(layout)
<Array [[[], [1], [1, 2, ... [1, 2, 3, 4, 5]]] type='2 * 3 * var * int64'>

The data type for a RegularArray is ak.types.RegularType, printed above as the “3 *” in the type above. (The “2 *” describes the length of the array itself, which is always “regular” in the sense that there’s only one of them, equal to itself.)

The “var *” is the type of variable-length lists, nested inside of the RegularArray.

RegularArray is the first array type that can have unreachable data: the length of its nested content might not evenly divide the RegularArray’s regular size.

ak.Array(
    ak.layout.RegularArray(
        ak.layout.NumpyArray(np.array([1, 2, 3, 4, 5, 6, 7])),
        3,
    )
)
<Array [[1, 2, 3], [4, 5, 6]] type='2 * 3 * int64'>

In the high-level array, we only see [[1, 2, 3], [4, 5, 6]] and not 7. Since the 7 items in the nested NumpyArray can’t be subdivided into lists of length 3. This 7 exists in the underlying physical data, but in the high-level view, it is as though it did not.

Content >: ListArray

ak.layout.ListArray and ak.layout.ListOffsetArray are the two node types that describe variable-length lists (ak.types.ListType, represented in type strings as “var *”). ak.layout.ListArray is the most general. It takes two Indexes, starts and stops, which indicate where each nested list starts and stops.

layout = ak.layout.ListArray64(
    ak.layout.Index64(np.array([0, 3, 3])),
    ak.layout.Index64(np.array([3, 3, 5])),
    ak.layout.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])),
)
layout
<ListArray64>
    <starts><Index64 i="[0 3 3]" offset="0" length="3" at="0x000001ef8fc0"/></starts>
    <stops><Index64 i="[3 3 5]" offset="0" length="3" at="0x000001ef2ab0"/></stops>
    <content><NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001ef2950"/></content>
</ListArray64>
ak.Array(layout)
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

The nested content, [1.1, 2.2, 3.3, 4.4, 5.5] is divided into three lists, [1.1, 2.2, 3.3], [], [4.4, 5.5] by starts=[0, 3, 3] and stops=[3, 3, 5]. That is to say, the first list is drawn from indexes 0 through 3 of the content, the second is empty (from 3 to 3), and the third is drawn from indexes 3 through 5.

Content >: ListOffsetArray

ak.layout.ListOffsetArray and ak.layout.ListArray are the two node types that describe variable-length lists (ak.types.ListType, represented in type strings as “var *”). ak.layout.ListOffsetArray is an important special case, in which

starts = offsets[:-1]
stops  = offsets[1:]

for a single array offsets. If we were only representing arrays and not doing computations on them, we would always use ListOffsetArrays, because they are the most compact. Knowing that a node is a ListOffsetArray can also simplify the implementation of some operations. In a sense, operations that produce ListArrays (previous section) can be thought of as delaying the part of the operation that would propagate down into the content, as an optimization.

layout = ak.layout.ListOffsetArray64(
    ak.layout.Index64(np.array([0, 3, 3, 5])),
    ak.layout.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])),
)
layout
<ListOffsetArray64>
    <offsets><Index64 i="[0 3 3 5]" offset="0" length="4" at="0x000001ef95c0"/></offsets>
    <content><NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001e8de40"/></content>
</ListOffsetArray64>
ak.Array(layout)
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

The length of the offsets array is one larger than the length of the array itself; an empty array has an offsets of length 1.

However, the offsets does not need to start at 0 or stop at len(content).

ak.Array(
    ak.layout.ListOffsetArray64(
        ak.layout.Index64(np.array([1, 3, 3, 4])),
        ak.layout.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5])),
    )
)
<Array [[2.2, 3.3], [], [4.4]] type='3 * var * float64'>

In the above, offsets[0] == 1 means that the 1.1 in the content is unreachable, and offsets[-1] == 4 means that the 5.5 is unreachable. Just as in ListArrays, unreachable data in ListOffsetArrays usually comes about because we don’t want computations to always propagate all the way down large trees.

Nested lists

As with all non-leaf Content nodes, arbitrarily deep nested lists can be built by nesting RegularArrays, ListArrays, and ListOffsetArrays. In each case, the starts, stops, or offsets only index the next level down in structure.

For example, here is an array of 5 lists, whose length is approximately 20 each.

layout = ak.layout.ListOffsetArray64(
    ak.layout.Index64(np.array([0, 18, 42, 59, 83, 100])),
    ak.layout.NumpyArray(np.arange(100))
)
array = ak.Array(layout)
array[0], array[1], array[2], array[3], array[4]
(<Array [0, 1, 2, 3, 4, ... 13, 14, 15, 16, 17] type='18 * int64'>,
 <Array [18, 19, 20, 21, 22, ... 38, 39, 40, 41] type='24 * int64'>,
 <Array [42, 43, 44, 45, 46, ... 55, 56, 57, 58] type='17 * int64'>,
 <Array [59, 60, 61, 62, 63, ... 79, 80, 81, 82] type='24 * int64'>,
 <Array [83, 84, 85, 86, 87, ... 96, 97, 98, 99] type='17 * int64'>)

Making an array of 3 lists that contains these 5 lists requires us to make indexes that go up to 5, not 100.

array = ak.Array(
    ak.layout.ListOffsetArray64(
        ak.layout.Index64(np.array([0, 3, 3, 5])),
        layout,
    )
)
array[0], array[1], array[2]
(<Array [[0, 1, 2, 3, 4, ... 55, 56, 57, 58]] type='3 * var * int64'>,
 <Array [] type='0 * var * int64'>,
 <Array [[59, 60, 61, 62, ... 96, 97, 98, 99]] type='2 * var * int64'>)

Strings and bytestrings

As described above, any Content node can have any parameters, but some have special meanings. Most parameters are intended to associate runtime behaviors with data structures, the way that methods add computational abilities to a class. Unicode strings and raw bytestrings are an example of this.

Awkward Array has no “StringArray” type because such a thing would be stored, sliced, and operated upon in most circumatances as a list-type array (RegularArray, ListArray, ListOffsetArray) of bytes. A parameter named “__array__” adds behaviors, such as the string interpretation, to arrays.

The ak.layout.NumpyArray must be directly nested within the list-type array, which can be checked with ak.is_valid and ak.validity_error.

Here is an example of a raw bytestring:

ak.Array(
    ak.layout.ListOffsetArray64(
        ak.layout.Index64(np.array([0, 3, 8, 11, 15])),
        ak.layout.NumpyArray(
            np.array([104, 101, 121, 116, 104, 101, 114, 101, 121, 111, 117, 103, 117, 121, 115], np.uint8),
            parameters={"__array__": "byte"}
        ),
        parameters={"__array__": "bytestring"}
    )
)
<Array [b'hey', b'there', b'you', b'guys'] type='4 * bytes'>

And here is an example of a Unicode-encoded string (UTF-8):

ak.Array(
    ak.layout.ListOffsetArray64(
        ak.layout.Index64(np.array([0, 3, 12, 15, 19])),
        ak.layout.NumpyArray(
            np.array([104, 101, 121, 226, 128, 148, 226, 128, 148, 226, 128, 148, 121, 111, 117, 103, 117, 121, 115], np.uint8),
            parameters={"__array__": "char"}
        ),
        parameters={"__array__": "string"}
    )
)
<Array ['hey', '———', 'you', 'guys'] type='4 * string'>

As with any other lists, strings can be nested within lists. Only the ak.layout.RegularArray/ak.layout.ListArray/ak.layout.ListOffsetArray corresponding to the strings should have the "__array__": "string" or "bytestring" parameter.

ak.Array(
    ak.layout.ListOffsetArray64(
        ak.layout.Index64([0, 2, 4]),
        ak.layout.ListOffsetArray64(
            ak.layout.Index64(np.array([0, 3, 12, 15, 19])),
            ak.layout.NumpyArray(
                np.array([104, 101, 121, 226, 128, 148, 226, 128, 148, 226, 128, 148, 121, 111, 117, 103, 117, 121, 115], np.uint8),
                parameters={"__array__": "char"}
            ),
            parameters={"__array__": "string"}
        )
    )
)
<Array [['hey', '———'], ['you', 'guys']] type='2 * var * string'>

Content >: RecordArray

ak.layout.RecordArray and ak.layout.UnionArray are the only two node types that have multiple contents, not just a single content (and the property is pluralized to reflect this fact). RecordArrays represent a “product type,” data containing records with fields x, y, and z have x’s type AND y’s type AND z’s type, whereas UnionArrays represent a “sum type,” data that are x’s type OR y’s type OR z’s type.

RecordArrays have no ak.layout.Index-valued properties; they may be thought of as metadata-only groupings of Content nodes. Since the RecordArray node holds an array for each field, it is a “struct of arrays,” rather than an “array of structs.”

RecordArray fields are ordered and provided as an ordered list of contents and field names (the recordlookup).

layout = ak.layout.RecordArray(
    [
        ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
        ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
    ],
    [
        "x",
        "y",
    ],
)
layout
<RecordArray length="5">
    <field index="0" key="x">
        <NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001f01b30"/>
    </field>
    <field index="1" key="y">
        <ListOffsetArray64>
            <offsets><Index64 i="[0 1 3 6 8 9]" offset="0" length="6" at="0x000001f01d70"/></offsets>
            <content><NumpyArray format="l" shape="9" data="1 1 2 1 2 3 3 2 3" at="0x000001ef29b0"/></content>
        </ListOffsetArray64>
    </field>
</RecordArray>
ak.Array(layout)
<Array [{x: 1.1, y: [1]}, ... x: 5.5, y: [3]}] type='5 * {"x": float64, "y": var...'>
ak.to_list(layout)
[{'x': 1.1, 'y': [1]},
 {'x': 2.2, 'y': [1, 2]},
 {'x': 3.3, 'y': [1, 2, 3]},
 {'x': 4.4, 'y': [3, 2]},
 {'x': 5.5, 'y': [3]}]

RecordArray fields do not need to have names. If the recordlookup is None, the RecordArray is interpreted as an array of tuples. (The word “tuple,” in statically typed environments, usually means a fixed-length type in which each element may be a different type.)

layout = ak.layout.RecordArray(
    [
        ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
        ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
    ],
    None,
)
layout
<RecordArray length="5">
    <field index="0">
        <NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001ef11b0"/>
    </field>
    <field index="1">
        <ListOffsetArray64>
            <offsets><Index64 i="[0 1 3 6 8 9]" offset="0" length="6" at="0x000001effcb0"/></offsets>
            <content><NumpyArray format="l" shape="9" data="1 1 2 1 2 3 3 2 3" at="0x000001efbcb0"/></content>
        </ListOffsetArray64>
    </field>
</RecordArray>
ak.Array(layout)
<Array [(1.1, [1]), (2.2, ... ), (5.5, [3])] type='5 * (float64, var * int64)'>
ak.to_list(layout)
[(1.1, [1]), (2.2, [1, 2]), (3.3, [1, 2, 3]), (4.4, [3, 2]), (5.5, [3])]

Since the RecordArray node holds an array for each of its fields, it is possible for these arrays to have different lengths. In such a case, the length of the RecordArray can be given explicitly or it is taken to be the length of the shortest field-array.

content0 = ak.layout.NumpyArray(np.array([1, 2, 3, 4, 5, 6, 7, 8]))
content1 = ak.layout.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5]))
content2 = ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2, 1], [3, 2], [3]], highlevel=False)
print(f"{len(content0) = }, {len(content1) = }, {len(content2) = }")

layout = ak.layout.RecordArray([content0, content1, content2], ["x", "y", "z"])
print(f"{len(layout) = }")
len(content0) = 8, len(content1) = 5, len(content2) = 6
len(layout) = 5
layout = ak.layout.RecordArray([content0, content1, content2], ["x", "y", "z"], length=3)
print(f"{len(layout) = }")
len(layout) = 3

RecordArrays are also allowed to have zero fields. This is an unusual case, but it is one that allows a RecordArray to be a leaf node (like ak.layout.EmptyArray and ak.layout.NumpyArray). If a RecordArray has no fields, a length must be given.

ak.Array(ak.layout.RecordArray([], [], length=5))
<Array [{}, {}, {}, {}, {}] type='5 * {}'>
ak.Array(ak.layout.RecordArray([], None, length=5))
<Array [(), (), (), (), ()] type='5 * ()'>

Scalar Records

An ak.layout.RecordArray is an array of records. Just as you can extract a scalar number from an array of numbers, you can extract a scalar record. Unlike numbers, records may still be sliced in some ways like Awkward Arrays:

array = ak.Array(
    ak.layout.RecordArray(
        [
            ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
            ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
        ],
        [
            "x",
            "y",
        ],
    )
)
record = array[2]
record
<Record {x: 3.3, y: [1, 2, 3]} type='{"x": float64, "y": var * int64}'>
record["y", -1]
3

Therefore, we need an ak.layout.Record type, but this Record is not an array, so it is not a subclass of ak.layout.Content.

Due to the columnar orientation of Awkward Array, a RecordArray does not contain Records, a Record contains a RecordArray.

record.layout
<Record at="2">
    <RecordArray length="5">
        <field index="0" key="x">
            <NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001f08c50"/>
        </field>
        <field index="1" key="y">
            <ListOffsetArray64>
                <offsets><Index64 i="[0 1 3 6 8 9]" offset="0" length="6" at="0x000001effe20"/></offsets>
                <content><NumpyArray format="l" shape="9" data="1 1 2 1 2 3 3 2 3" at="0x000001ef8d70"/></content>
            </ListOffsetArray64>
        </field>
    </RecordArray>
</Record>

It can be built by passing a RecordArray as its first argument and the item of interest in its second argument.

layout = ak.layout.Record(
    ak.layout.RecordArray(
        [
            ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
            ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
        ],
        [
            "x",
            "y",
        ],
    ),
    2
)
record = ak.Record(layout)   # note the high-level ak.Record, rather than ak.Array
record
<Record {x: 3.3, y: [1, 2, 3]} type='{"x": float64, "y": var * int64}'>

Naming record types

The records discussed so far are generic. Naming a record not only makes it easier to read type strings, it’s also how ak.behavior overloads functions and adds methods to records as though they were classes in object-oriented programming.

A name is given to an ak.layout.RecordArray node through its “__record__” parameter.

layout = ak.layout.RecordArray(
    [
        ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
        ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
    ],
    [
        "x",
        "y",
    ],
    parameters={"__record__": "Special"}
)
layout
<RecordArray length="5">
    <parameters>
        <param key="__record__">"Special"</param>
    </parameters>
    <field index="0" key="x">
        <NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001eef1a0"/>
    </field>
    <field index="1" key="y">
        <ListOffsetArray64>
            <offsets><Index64 i="[0 1 3 6 8 9]" offset="0" length="6" at="0x000001ef2890"/></offsets>
            <content><NumpyArray format="l" shape="9" data="1 1 2 1 2 3 3 2 3" at="0x000001f0e220"/></content>
        </ListOffsetArray64>
    </field>
</RecordArray>
ak.Array(layout)
<Array [{x: 1.1, y: [1]}, ... x: 5.5, y: [3]}] type='5 * Special["x": float64, "...'>
ak.type(layout)
Special["x": float64, "y": var * int64]

Behavioral overloads are presented in more depth in ak.behavior, but here are three examples:

ak.behavior[np.sqrt, "Special"] = lambda special: np.sqrt(special.x)

np.sqrt(ak.Array(layout))
<Array [1.05, 1.48, 1.82, 2.1, 2.35] type='5 * float64'>
class SpecialRecord(ak.Record):
    def len_y(self):
        return len(self.y)

ak.behavior["Special"] = SpecialRecord

ak.Record(layout[2]).len_y()
3
class SpecialArray(ak.Array):
    def len_y(self):
        return ak.num(self.y)

ak.behavior["*", "Special"] = SpecialArray

ak.Array(layout).len_y()
<Array [1, 2, 3, 2, 1] type='5 * int64'>

Content >: IndexedArray

ak.layout.IndexedArray is the only Content node that has exactly the same type as its content. An IndexedArray rearranges the elements of its content, as though it were a lazily applied array-slice.

layout = ak.layout.IndexedArray64(
    ak.layout.Index64(np.array([2, 0, 0, 1, 2])),
    ak.layout.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3])),
)
layout
<IndexedArray64>
    <index><Index64 i="[2 0 0 1 2]" offset="0" length="5" at="0x000001f10cf0"/></index>
    <content><NumpyArray format="d" shape="4" data="0 1.1 2.2 3.3" at="0x000001ef7e60"/></content>
</IndexedArray64>
ak.Array(layout)
<Array [2.2, 0, 0, 1.1, 2.2] type='5 * float64'>

As such, IndexedArrays can be used to (lazily) remove items, permute items, and/or duplicate items.

IndexedArrays are used as an optimization when performing some operations on ak.layout.RecordArray, to avoid propagating them down to every field. This is more relevant for RecordArrays than other nodes because RecordArrays can contain multiple contents (and UnionArrays, which also have multiple contents, have an index to merge an array-slice into).

For example, slicing the following recordarray by [3, 2, 4, 4, 1, 0, 3] does not affect any of the RecordArray’s fields.

recordarray = ak.layout.RecordArray(
    [
        ak.from_iter([1.1, 2.2, 3.3, 4.4, 5.5], highlevel=False),
        ak.from_iter([[1], [1, 2], [1, 2, 3], [3, 2], [3]], highlevel=False),
    ],
    None,
)
recordarray
<RecordArray length="5">
    <field index="0">
        <NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001f10c60"/>
    </field>
    <field index="1">
        <ListOffsetArray64>
            <offsets><Index64 i="[0 1 3 6 8 9]" offset="0" length="6" at="0x000001efb530"/></offsets>
            <content><NumpyArray format="l" shape="9" data="1 1 2 1 2 3 3 2 3" at="0x000001f06f80"/></content>
        </ListOffsetArray64>
    </field>
</RecordArray>
recordarray[[3, 2, 4, 4, 1, 0, 3]]
<IndexedArray64>
    <index><Index64 i="[3 2 4 4 1 0 3]" offset="0" length="7" at="0x000001ef1520"/></index>
    <content><RecordArray length="5">
        <field index="0">
            <NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001f10c60"/>
        </field>
        <field index="1">
            <ListOffsetArray64>
                <offsets><Index64 i="[0 1 3 6 8 9]" offset="0" length="6" at="0x000001efb530"/></offsets>
                <content><NumpyArray format="l" shape="9" data="1 1 2 1 2 3 3 2 3" at="0x000001f06f80"/></content>
            </ListOffsetArray64>
        </field>
    </RecordArray></content>
</IndexedArray64>

Categorical data

IndexedArrays, with the "__array__": "categorical" parameter, can represent categorical data (sometimes called dictionary-encoding).

layout = ak.layout.IndexedArray64(
    ak.layout.Index64(np.array([2, 2, 1, 4, 0, 5, 3, 3, 0, 1])),
    ak.from_iter(["zero", "one", "two", "three", "four", "five"], highlevel=False),
    parameters={"__array__": "categorical"}
)
ak.to_list(layout)
['two', 'two', 'one', 'four', 'zero', 'five', 'three', 'three', 'zero', 'one']

The above has only one copy of strings from "zero" to "five", but they are effectively replicated 10 times in the array.

Any IndexedArray can perform this lazy replication, but labeling it as "__array__": "categorical" is a promise that the content contains only unique values.

Functions like ak.to_arrow and ak.to_parquet will project (eagerly evaluate) an IndexedArray that is not labeled as "__array__": "categorical" and dictionary-encode an IndexedArray that is. This distinguishes between the use of IndexedArrays as invisible optimizations and intentional, user-visible ones.

Content >: IndexedOptionArray

ak.layout.IndexedOptionArray is the most general of the four nodes that allow for missing data (ak.layout.IndexedOptionArray, ak.layout.ByteMaskedArray, ak.layout.BitMaskedArray, and ak.layout.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).

ak.layout.IndexedOptionArray is an ak.layout.IndexedArray in which negative values in the index are interpreted as missing. Since it has an index, the content does not need to have “dummy values” at each index of a missing value. It is therefore more compact for record-type data with many missing values, but ak.layout.ByteMaskedArray and especially ak.layout.BitMaskedArray are more compact for numerical data or data without many missing values.

layout = ak.layout.IndexedOptionArray64(
    ak.layout.Index64(np.array([2, -1, 0, -1, -1, 1, 2])),
    ak.layout.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3])),
)
layout
<IndexedOptionArray64>
    <index><Index64 i="[2 -1 0 -1 -1 1 2]" offset="0" length="7" at="0x000001f07ee0"/></index>
    <content><NumpyArray format="d" shape="4" data="0 1.1 2.2 3.3" at="0x000001efa910"/></content>
</IndexedOptionArray64>
ak.Array(layout)
<Array [2.2, None, 0, None, None, 1.1, 2.2] type='7 * ?float64'>

Because of its flexibility, most operations that output potentially missing values use an IndexedOptionArray. ak.packed and conversions to/from Arrow or Parquet convert it back into a more compact type.

Content >: ByteMaskedArray

ak.layout.ByteMaskedArray is the simplest of the four nodes that allow for missing data (ak.layout.IndexedOptionArray, ak.layout.ByteMaskedArray, ak.layout.BitMaskedArray, and ak.layout.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).

ak.layout.ByteMaskedArray is most similar to NumPy’s masked arrays, except that Awkward ByteMaskedArrays can contain any data type and variable-length structures. The valid_when parameter lets you choose whether True means an element is valid (not masked/not None) or False means an element is valid.

layout = ak.layout.ByteMaskedArray(
    ak.layout.Index8(np.array([False, False, True, True, False, True, False], np.int8)),
    ak.layout.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6])),
    valid_when=False,
)
layout
<ByteMaskedArray valid_when="false">
    <mask><Index8 i="[0 0 1 1 0 1 0]" offset="0" length="7" at="0x000001f14040"/></mask>
    <content><NumpyArray format="d" shape="7" data="0 1.1 2.2 3.3 4.4 5.5 6.6" at="0x000001f10b60"/></content>
</ByteMaskedArray>
ak.Array(layout)
<Array [0, 1.1, None, None, 4.4, None, 6.6] type='7 * ?float64'>

Content >: BitMaskedArray

ak.layout.BitMaskedArray is the most compact of the four nodes that allow for missing data (ak.layout.IndexedOptionArray, ak.layout.ByteMaskedArray, ak.layout.BitMaskedArray, and ak.layout.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).

ak.layout.BitMaskedArray is just like ak.layout.ByteMaskedArray except that the booleans are bit-packed. It is motivated primarily by Apache Arrow, which uses bit-packed masks; supporting bit-packed masks in Awkward Array allows us to represent Arrow data directly. Most operations immediately convert BitMaskedArrays into ByteMaskedArrays.

Since bits always come in groups of at least 8, an explicit length must be supplied to the constructor. Also, lsb_order=True or False determines whether the bytes are interpreted least-significant bit first or most-significant bit first, respectively.

layout = ak.layout.BitMaskedArray(
    ak.layout.IndexU8(np.packbits(np.array([False, False, True, True, False, True, False], np.uint8))),
    ak.layout.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6])),
    valid_when=False,
    length=7,
    lsb_order=True,
)
layout
<BitMaskedArray valid_when="false" length="7" lsb_order="true">
    <mask><IndexU8 i="[52]" offset="0" length="1" at="0x000001c7b310"/></mask>
    <content><NumpyArray format="d" shape="7" data="0 1.1 2.2 3.3 4.4 5.5 6.6" at="0x000001efa8d0"/></content>
</BitMaskedArray>
ak.Array(layout)
<Array [0, 1.1, None, 3.3, None, None, 6.6] type='7 * ?float64'>

Changing only the lsb_order changes the interpretation in important ways!

ak.Array(
    ak.layout.BitMaskedArray(
        layout.mask,
        layout.content,
        layout.valid_when,
        len(layout),
        lsb_order=False,
    )
)
<Array [0, 1.1, None, None, 4.4, None, 6.6] type='7 * ?float64'>

Content >: UnmaskedArray

ak.layout.UnmaskedArray describes formally missing data, but in a case in which no data are actually missing. It is a corner case of the four nodes that allow for missing data (ak.layout.IndexedOptionArray, ak.layout.ByteMaskedArray, ak.layout.BitMaskedArray, and ak.layout.UnmaskedArray). Missing data is also known as “option type” (denoted by a question mark ? or the word option in type strings).

layout = ak.layout.UnmaskedArray(
    ak.layout.NumpyArray(np.array([1.1, 2.2, 3.3, 4.4, 5.5]))
)
layout
<UnmaskedArray>
    <content><NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001f13fc0"/></content>
</UnmaskedArray>
ak.Array(layout)
<Array [1.1, 2.2, 3.3, 4.4, 5.5] type='5 * ?float64'>

The only distinguishing feature of an UnmaskedArray is the question mark ? or option in its type.

ak.type(layout)
?float64

Content >: UnionArray

ak.layout.UnionArray and ak.layout.RecordArray are the only two node types that have multiple contents, not just a single content (and the property is pluralized to reflect this fact). RecordArrays represent a “product type,” data containing records with fields x, y, and z have x’s type AND y’s type AND z’s type, whereas UnionArrays represent a “sum type,” data that are x’s type OR y’s type OR z’s type.

In addition, ak.layout.UnionArray has two ak.layout.Index-typed attributes, tags and index; it is the most complex node type. The tags specify which content array to draw each array element from, and the index specifies which element from that content.

The UnionArray element at index i is therefore:

contents[tags[i]][index[i]]

Although the ability to make arrays with mixed data type is very expressive, not all operations support union type (including iteration in Numba). If you intend to make union-type data for an application, be sure to verify that it will work by generating some test data using ak.from_iter.

Awkward Array’s UnionArray is equivalent to Apache Arrow’s dense union. Awkward Array has no counterpart for Apache Arrow’s sparse union (which has no index). ak.from_arrow generates an index on demand when reading sparse union from Arrow.

layout = ak.layout.UnionArray8_64(
    ak.layout.Index8( np.array([0, 1, 2, 0, 0, 1, 1, 2, 2, 0], np.int8)),
    ak.layout.Index64(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])),
    [
        ak.layout.NumpyArray(np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9])),
        ak.from_iter([[], [1], [1, 2], [1, 2, 3], [1, 2, 3, 4], [1, 2, 3, 4, 5], [6], [6, 7], [6, 7, 8], [6, 7, 8, 9]], highlevel=False),
        ak.from_iter(["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine"], highlevel=False)
    ]
)
layout
<UnionArray8_64>
    <tags><Index8 i="[0 1 2 0 0 1 1 2 2 0]" offset="0" length="10" at="0x000001f15210"/></tags>
    <index><Index64 i="[0 1 2 3 4 5 6 7 8 9]" offset="0" length="10" at="0x000001eef110"/></index>
    <content tag="0">
        <NumpyArray format="d" shape="10" data="0 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 9.9" at="0x000001f05a70"/>
    </content>
    <content tag="1">
        <ListOffsetArray64>
            <offsets><Index64 i="[0 0 1 3 6 10 15 16 18 21 25]" offset="0" length="11" at="0x000001ed6070"/></offsets>
            <content><NumpyArray format="l" shape="25" data="1 1 2 1 2 ... 8 6 7 8 9" at="0x000001f15af0"/></content>
        </ListOffsetArray64>
    </content>
    <content tag="2">
        <ListOffsetArray64>
            <parameters>
                <param key="__array__">"string"</param>
            </parameters>
            <offsets><Index64 i="[0 4 7 10 15 19 23 26 31 36 40]" offset="0" length="11" at="0x000001f03700"/></offsets>
            <content><NumpyArray format="B" shape="40" data="122 101 114 111 111 ... 116 110 105 110 101" at="0x000001f15990">
                <parameters>
                    <param key="__array__">"char"</param>
                </parameters>
            </NumpyArray></content>
        </ListOffsetArray64>
    </content>
</UnionArray8_64>
ak.to_list(layout)
[0.0, [1], 'two', 3.3, 4.4, [1, 2, 3, 4, 5], [6], 'seven', 'eight', 9.9]

The index can be used to prevent the need to set up “dummy values” for all contents other than the one specified by a given tag. The above example could thus be more compact with the following (no unreachable data):

layout = ak.layout.UnionArray8_64(
    ak.layout.Index8( np.array([0, 1, 2, 0, 0, 1, 1, 2, 2, 0], np.int8)),
    ak.layout.Index64(np.array([0, 0, 0, 1, 2, 1, 2, 1, 2, 3])),
    [
        ak.layout.NumpyArray(np.array([0.0, 3.3, 4.4, 9.9])),
        ak.from_iter([[1], [1, 2, 3, 4, 5], [6]], highlevel=False),
        ak.from_iter(["two", "seven", "eight"], highlevel=False)
    ]
)
layout
<UnionArray8_64>
    <tags><Index8 i="[0 1 2 0 0 1 1 2 2 0]" offset="0" length="10" at="0x000001f1a480"/></tags>
    <index><Index64 i="[0 0 0 1 2 1 2 1 2 3]" offset="0" length="10" at="0x000001f037d0"/></index>
    <content tag="0">
        <NumpyArray format="d" shape="4" data="0 3.3 4.4 9.9" at="0x000001f03830"/>
    </content>
    <content tag="1">
        <ListOffsetArray64>
            <offsets><Index64 i="[0 1 6 7]" offset="0" length="4" at="0x000001f14450"/></offsets>
            <content><NumpyArray format="l" shape="7" data="1 1 2 3 4 5 6" at="0x000001f10dd0"/></content>
        </ListOffsetArray64>
    </content>
    <content tag="2">
        <ListOffsetArray64>
            <parameters>
                <param key="__array__">"string"</param>
            </parameters>
            <offsets><Index64 i="[0 3 8 13]" offset="0" length="4" at="0x000001f1e680"/></offsets>
            <content><NumpyArray format="B" shape="13" data="116 119 111 115 101 ... 101 105 103 104 116" at="0x000001f1e710">
                <parameters>
                    <param key="__array__">"char"</param>
                </parameters>
            </NumpyArray></content>
        </ListOffsetArray64>
    </content>
</UnionArray8_64>
ak.to_list(layout)
[0.0, [1], 'two', 3.3, 4.4, [1, 2, 3, 4, 5], [6], 'seven', 'eight', 9.9]

ak.from_iter is by far the easiest way to create UnionArrays for small tests.

Content >: VirtualArray

ak.layout.VirtualArray represents data to be generated on demand. It takes a Python function, an optional cache, and optional information about the data to be generated as arguments. Since the Python function is bound to the Python process and might invoke objects such as file handles, network connections, or data without a generic serialization, VirtualArrays can’t be serialized in files or byte streams, but they are frequently used to lazily load data from a file.

# The generator function can also take arguments, see the documentation.
def generate():
    print("generating")
    return ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])

layout = ak.layout.VirtualArray(ak.layout.ArrayGenerator(generate))
layout
<VirtualArray cache_key="ak0">
    <ArrayGenerator f="<function generate at 0x7f7981a92670>"/>
</VirtualArray>
ak.Array(layout)[2]
generating
generating
generating
<Array [4.4, 5.5] type='2 * float64'>
ak.Array(layout)
generating
generating
generating
generating
generating
generating
generating
generating
generating
generating
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

This VirtualArray has no hints and no cache, so it must call generate every time it is accessed for any information, such as “What is your length?” or “What is element 2?” (which can happen many times when determining how much to print in a string repr). To reduce the number of calls to generate, we can

  • provide more information about the array that will be generated, such as its length and its form,

  • provide a cache, so that the array will not need to be generated more than once (until it gets evicted from cache).

layout = ak.layout.VirtualArray(
    ak.layout.ArrayGenerator(generate, length=3)
)
layout
<VirtualArray cache_key="ak1">
    <ArrayGenerator f="<function generate at 0x7f7981a92670>">
        <length>3</length>
    </ArrayGenerator>
</VirtualArray>
ak.Array(layout)[2]
generating
generating
<Array [4.4, 5.5] type='2 * float64'>
ak.Array(layout)
generating
generating
generating
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>
import json

form = ak.forms.Form.fromjson(json.dumps({
    "class": "ListOffsetArray64",
    "offsets": "i64",
    "content": "float64",
}))

layout = ak.layout.VirtualArray(
    ak.layout.ArrayGenerator(generate, length=3, form=form)
)
layout
<VirtualArray cache_key="ak2">
    <ArrayGenerator f="<function generate at 0x7f7981a92670>">
        <length>3</length>
        <form>
            {
                "class": "ListOffsetArray64",
                "offsets": "i64",
                "content": "float64"
            }
        </form>
    </ArrayGenerator>
</VirtualArray>
ak.Array(layout)[2]
generating
<Array [4.4, 5.5] type='2 * float64'>
ak.Array(layout)
generating
generating
generating
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

The cache can be any Python object that satisfies Python’s MutableMapping protocol, but the data are only weakly referenced by the VirtualArray. The weak reference ensures that that VirtualArray cannot reference itself through this cache, since a cyclic reference would not be caught by the Python garbage collector (since the Awkward Array nodes are implemented in C++, not Python). As another technicality, Python dict objects can’t be weakly referenced.

Strong references to the cache are held by the high-level ak.Array objects, so just be sure to keep a reference to the desired cache until the ak.Array is constructed (i.e. in the same function scope).

import cachetools

cache = cachetools.LRUCache(np.inf)   # infinite-lifetime cache, alternative to dict

layout = ak.layout.VirtualArray(
    ak.layout.ArrayGenerator(generate),
    cache=ak.layout.ArrayCache(cache),
)

array = ak.Array(layout)

# only now is it safe for "cache" to go out of scope
layout
<VirtualArray cache_key="ak3">
    <ArrayGenerator f="<function generate at 0x7f7981a92670>"/>
    <ArrayCache mapping="LRUCache([], maxsize=inf, currsize=0)"/>
</VirtualArray>
array
generating
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>
array
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

VirtualArrays are admittedly tricky to set up properly, but most of these technical issues will be eliminated by Awkward Array 2.0, which will replace the C++ layer with Python.

The high-level ak.virtual function is an easier way to create a VirtualArray, but if the VirtualArray is to be embedded within another structure, the cache must not go out of scope before the final ak.Array wrapper is built.

VirtualArrays are frequently used as fields of a large RecordArray, so that only the fields that are accessed need to be read. Another common use-case is to put a VirtualArray in each partition of a PartitionedArray (see below), so that partitions load on demand.

VirtualArrays can be used in Numba-compiled function and they are read on demand, but are not thread-safe (because they need to acquire a Python context to call the generate function).

PartitionedArray and IrregularlyPartitionedArray

ak.partition.PartitionedArray’s subclass, ak.partition.IrregularlyPartitionedArray, is not a ak.layout.Content subclass, deliberately preventing it from being used within a layout tree. It can, however, be the root of a layout tree within an ak.Array. This reflects our view that partitioning is only useful (and easiest to get right) for whole arrays, not internal nodes of a columnar structure. The name “IrregularlyPartitionedArray” for the only concrete class was chosen to allow for regularly partitioned arrays in the future (which would not need a set of stops).

Each partition is an individually contiguous array of the same type. The whole sequence is presented as though it were a single array. (PartitionedArray is lazy concatenation.)

layout = ak.partition.IrregularlyPartitionedArray(
    [
        ak.layout.NumpyArray(np.array([0.0, 1.1, 2.2])),
        ak.layout.NumpyArray(np.array([3.3])),
        ak.layout.NumpyArray(np.array([], np.float64)),
        ak.layout.NumpyArray(np.array([4.4, 5.5, 6.6, 7.7])),
        ak.layout.NumpyArray(np.array([8.8, 9.9])),
    ],
    [
        3, 4, 4, 8, 10
    ],
)
layout
<IrregularlyPartitionedArray>
    <partition start="0" stop="3">
        <NumpyArray format="d" shape="3" data="0 1.1 2.2" at="0x000001f036a0"/>
    </partition>
    <partition start="3" stop="4">
        <NumpyArray format="d" shape="1" data="3.3" at="0x000001ca1cd0"/>
    </partition>
    <partition start="4" stop="8">
        <NumpyArray format="d" shape="4" data="4.4 5.5 6.6 7.7" at="0x000001f02720"/>
    </partition>
    <partition start="8" stop="10">
        <NumpyArray format="d" shape="2" data="8.8 9.9" at="0x000001c9fd80"/>
    </partition>
</IrregularlyPartitionedArray>
ak.Array(layout)
<Array [0, 1.1, 2.2, 3.3, ... 7.7, 8.8, 9.9] type='10 * float64'>

The second argument of the constructor is the stops, the index at which each partition stops. It has the same meaning as ak.layout.ListArray’s stops, but a corresponding starts is not needed because these counts are strictly monatonic and the first start is assumed to be 0.

Note that the stops are managed in Python: generally, the partitions must be large to be efficient. (stops can also be inferred if not given.)

Partitions can also be constructed with the high-level ak.partitioned function, and since partitions are always at the root of an array tree, the high-level function would always be sufficient. An unpartitioned array can also be turned into a partitioned one with ak.repartition.

The following example combines PartitionedArrays with VirtualArrays to make a lazy array:

form = ak.forms.Form.from_numpy(np.dtype("int64"))

array = ak.partitioned([
    ak.virtual(lambda count: ak.Array(np.arange(count * 3)), args=(i,), length=i * 3, form=form)
    for i in range(4)
])

array.layout
<IrregularlyPartitionedArray>
    <partition start="0" stop="3">
        <VirtualArray cache_key="ak5">
            <ArrayGenerator f="<function <listcomp>.<lambda> at 0x7f7981a928b0>" args="(1,)">
                <length>3</length>
                <form>
                    {
                        "class": "NumpyArray",
                        "itemsize": 8,
                        "format": "l",
                        "primitive": "int64"
                    }
                </form>
            </ArrayGenerator>
            <ArrayCache mapping="{}"/>
        </VirtualArray>
    </partition>
    <partition start="3" stop="9">
        <VirtualArray cache_key="ak6">
            <ArrayGenerator f="<function <listcomp>.<lambda> at 0x7f7981a92940>" args="(2,)">
                <length>6</length>
                <form>
                    {
                        "class": "NumpyArray",
                        "itemsize": 8,
                        "format": "l",
                        "primitive": "int64"
                    }
                </form>
            </ArrayGenerator>
            <ArrayCache mapping="{}"/>
        </VirtualArray>
    </partition>
    <partition start="9" stop="18">
        <VirtualArray cache_key="ak7">
            <ArrayGenerator f="<function <listcomp>.<lambda> at 0x7f7981a92dc0>" args="(3,)">
                <length>9</length>
                <form>
                    {
                        "class": "NumpyArray",
                        "itemsize": 8,
                        "format": "l",
                        "primitive": "int64"
                    }
                </form>
            </ArrayGenerator>
            <ArrayCache mapping="{}"/>
        </VirtualArray>
    </partition>
</IrregularlyPartitionedArray>
array.tolist()
[0, 1, 2, 0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 6, 7, 8]

Relationship to ak.from_buffers

The generic buffers tutorial describes a function, ak.from_buffers that builds an array from one-dimensional buffers and an ak.forms.Form. Forms describe the complete tree structure of an array without the array data or lengths, and the array data are in the buffers. The ak.from_buffers function was designed to operate on data produced by ak.to_buffers, but you can also prepare its form, length, and buffers manually.

The ak.from_buffers builds arrays using the above constructors, but the interface allows these structures to be built as data, rather than function calls. (Forms have a JSON representation.) If you are always building the same type of array, directly calling the constructors is likely easier. If you’re generating different data types programmatically, preparing data for ak.from_buffers may be easier than generating and evaluating Python code that call these constructors.

Every ak.layout.Content subclass has a corresponding ak.forms.Form, and you can see a layout’s Form through its form property.

array = ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
array.layout
<ListOffsetArray64>
    <offsets><Index64 i="[0 3 3 5]" offset="0" length="4" at="0x000001f188a0"/></offsets>
    <content><NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000001f1e730"/></content>
</ListOffsetArray64>
# Abbreviated JSON representation
array.layout.form
{
    "class": "ListOffsetArray64",
    "offsets": "i64",
    "content": "float64"
}
# Full JSON representation
print(array.layout.form.tojson(pretty=True))
{
    "class": "ListOffsetArray64",
    "offsets": "i64",
    "content": {
        "class": "NumpyArray",
        "inner_shape": [],
        "itemsize": 8,
        "format": "d",
        "primitive": "float64",
        "has_identities": false,
        "parameters": {},
        "form_key": null
    },
    "has_identities": false,
    "parameters": {},
    "form_key": null
}

In this way, you can figure out how to generate Forms corresponding to the Content nodes you want ak.from_buffers to make.