How to convert to/from Python objects

Builtin Python objects like dicts and lists can be converted into Awkward Arrays, and all Awkward Arrays can be converted into Python objects. Awkward type information, such as the distinction between fixed-size and variable-length lists, is lost in the transformation to Python objects.

import awkward as ak
import numpy as np
import pandas as pd

From Python to Awkward

The function for Python → Awkward conversion is ak.from_iter.

py_objects = [[1.1, 2.2, 3.3], [], [4.4, 5.5]]
py_objects
[[1.1, 2.2, 3.3], [], [4.4, 5.5]]
ak_array = ak.from_iter(py_objects)
ak_array
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

See the sections below for how Python types are mapped to Awkward types.

Note that this should be considered a slow, memory-intensive function: not only does it need to iterate over Python data, but it needs to discover the type of the data progressively. Internally, this function uses an ak.ArrayBuilder to accumulate data and discover types simultaneously. Don’t, for instance, convert a large, numerical dataset from NumPy or Arrow into Python objects just to use ak.from_iter. There are specialized functions for that: see their tutorials (left-bar or ≡ button on mobile).

This is also the fallback operation of the ak.Array and ak.Record constructors. Usually, small examples are built by passing Python objects directly to these constructors.

ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>
ak.Record({"x": 1, "y": [1.1, 2.2]})
<Record {x: 1, y: [1.1, 2.2]} type='{"x": int64, "y": var * float64}'>

From Awkward to Python

The function for Awkward → Python conversion is ak.to_list.

ak_array = ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
ak_array
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>
ak.to_list(ak_array)
[[1.1, 2.2, 3.3], [], [4.4, 5.5]]
ak_record = ak.Record({"x": 1, "y": [1.1, 2.2]})
ak_record
<Record {x: 1, y: [1.1, 2.2]} type='{"x": int64, "y": var * float64}'>
ak.to_list(ak_record)
{'x': 1, 'y': [1.1, 2.2]}

Note that this should be considered a slow, memory-intensive function, like ak.from_iter. Don’t, for instance, convert a large, numerical dataset with ak.to_list just to convert those lists into NumPy or Arrow. There are specialized functions for that: see their tutorials (left-bar or ≡ button on mobile).

Awkward Arrays and Records have a tolist method (note: no underscore), which is an analogy of NumPy’s tolist. For small datasets (or a small slice of a dataset), this is a convenient way to get a quick view.

x = ak.Array(np.arange(1000))
y = ak.Array(np.tile(np.array([0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9]), 100))
ak_array = ak.zip({"x": x, "y": y})
ak_array
<Array [{x: 0, y: 0}, ... {x: 999, y: 9.9}] type='1000 * {"x": int64, "y": float64}'>
ak_array[100].tolist()
{'x': 100, 'y': 0.0}
ak_array[100:110].tolist()
[{'x': 100, 'y': 0.0},
 {'x': 101, 'y': 1.1},
 {'x': 102, 'y': 2.2},
 {'x': 103, 'y': 3.3},
 {'x': 104, 'y': 4.4},
 {'x': 105, 'y': 5.5},
 {'x': 106, 'y': 6.6},
 {'x': 107, 'y': 7.7},
 {'x': 108, 'y': 8.8},
 {'x': 109, 'y': 9.9}]

Pandas-style constructor

As we have seen, the ak.Array constructor interprets an iterable argument as the data that it is meant to represent, as in:

py_objects1 = [[1.1, 2.2, 3.3], [], [4.4, 5.5]]
py_objects1
[[1.1, 2.2, 3.3], [], [4.4, 5.5]]
ak.Array(py_objects1)
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

But sometimes, you have several iterables that you want to use as columns of a table. The Pandas DataFrame constructor interprets a dict of iterables as columns:

py_objects2 = ["one", "two", "three"]
py_objects2
['one', 'two', 'three']
pd.DataFrame({"x": py_objects1, "y": py_objects2})
x y
0 [1.1, 2.2, 3.3] one
1 [] two
2 [4.4, 5.5] three

And so does the ak.Array constructor:

ak_array = ak.Array({"x": py_objects1, "y": py_objects2})
ak_array
<Array [{x: [1.1, 2.2, 3.3, ... y: 'three'}] type='3 * {"x": var * float64, "y":...'>
ak.type(ak_array)
3 * {"x": var * float64, "y": string}
ak.to_list(ak_array)
[{'x': [1.1, 2.2, 3.3], 'y': 'one'},
 {'x': [], 'y': 'two'},
 {'x': [4.4, 5.5], 'y': 'three'}]

Note that this is the transpose of the way the data would be interpreted if it were in a list, rather than a dict. The "x" and "y" values are interpreted as being interleaved in each record. There is no potential for conflict between the ak.from_iter-style and Pandas-style constructors because ak.from_iter applied to a dict would always return an ak.Record, rather than an ak.Array.

ak_record = ak.from_iter({"x": py_objects1, "y": py_objects2})
ak_record
<Record ... 5.5]], y: ['one', 'two', 'three']} type='{"x": var * var * float64, ...'>
ak.type(ak_record)
{"x": var * var * float64, "y": var * string}
ak.to_list(ak_record)
{'x': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'y': ['one', 'two', 'three']}

The ak.from_iter function applied to a dict is also equivalent to the ak.Record constructor.

ak.Record({"x": py_objects1, "y": py_objects2})
<Record ... 5.5]], y: ['one', 'two', 'three']} type='{"x": var * var * float64, ...'>

Conversion of numbers and booleans

Python float, int, and bool (so-called “primitive” types) are converted to float64, int64, and bool types in Awkward Arrays.

All floating-point Awkward types are converted to Python’s float, all integral Awkward types are converted to Python’s int, and Awkward’s boolean type is converted to Python’s bool.

ak.Array([1.1, 2.2, 3.3])
<Array [1.1, 2.2, 3.3] type='3 * float64'>
ak.Array([1.1, 2.2, 3.3]).tolist()
[1.1, 2.2, 3.3]
ak.Array([1, 2, 3, 4, 5])
<Array [1, 2, 3, 4, 5] type='5 * int64'>
ak.Array([1, 2, 3, 4, 5]).tolist()
[1, 2, 3, 4, 5]
ak.Array([True, False, True, False, False])
<Array [True, False, True, False, False] type='5 * bool'>
ak.Array([True, False, True, False, False]).tolist()
[True, False, True, False, False]

Conversion of lists

Python lists, as well as iterables other than dict, tuple, str, and bytes, are converted to Awkward’s variable-length lists. It is not possible to construct fixed-size lists with ak.from_iter. (One way to do that is by converting a NumPy array with ak.from_numpy.)

Awkward’s variable-length and fixed-size lists are converted into Python lists with ak.to_list.

ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>
ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]]).tolist()
[[1.1, 2.2, 3.3], [], [4.4, 5.5]]
ak.Array([[1, 2, 3], [4, 5, 6]])
<Array [[1, 2, 3], [4, 5, 6]] type='2 * var * int64'>
ak.Array([[1, 2, 3], [4, 5, 6]]).tolist()
[[1, 2, 3], [4, 5, 6]]

Note

Advanced topic: the rest of this section may be skipped if you don’t care about the distinction between fixed-size and variable-length lists.

Note that a NumPy array is an iterable, so ak.from_iter iterates over it, constructing variable-length Awkward lists. By contrast, ak.from_numpy casts the data (without iteration) into fixed-size Awkward lists.

np_array = np.array([[100, 200], [101, 201], [103, 203]])
np_array
array([[100, 200],
       [101, 201],
       [103, 203]])
ak.from_iter(np_array)
<Array [[100, 200], [101, 201], [103, 203]] type='3 * var * int64'>
ak.from_numpy(np_array)
<Array [[100, 200], [101, 201], [103, 203]] type='3 * 2 * int64'>

Note that the types differ: var * int64 vs 2 * int64. The ak.Array constructor uses ak.from_numpy if given a NumPy array (with dtype != "O") and ak.from_iter if given an iterable that it does not recognize.

This can be particularly subtle when NumPy arrays are nested within iterables.

np_array = np.array([[100, 200], [101, 201], [103, 203]])
np_array
array([[100, 200],
       [101, 201],
       [103, 203]])
# This is a NumPy array: constructor uses ak.from_numpy to get an array of fixed-size lists.
ak.Array(np_array)
<Array [[100, 200], [101, 201], [103, 203]] type='3 * 2 * int64'>
py_objects = [np.array([100, 200]), np.array([101, 201]), np.array([103, 203])]
py_objects
[array([100, 200]), array([101, 201]), array([103, 203])]
# This is a list that contains NumPy arrays: constructor uses ak.from_iter to get an array of variable-length lists.
ak.Array(py_objects)
<Array [[100, 200], [101, 201], [103, 203]] type='3 * var * int64'>
np_array_dtype_O = np.array([[100, 200], [101, 201], [103, 203]], dtype="O")
np_array_dtype_O
array([[100, 200],
       [101, 201],
       [103, 203]], dtype=object)
# This NumPy array has dtype="O", so it cannot be cast without iteration: constructor uses ak.from_iter.
ak.Array(np_array_dtype_O)
<Array [[100, 200], [101, 201], [103, 203]] type='3 * var * int64'>

The logic behind this policy is that only NumPy arrays with dtype != "O" are guaranteed to have fixed-size contents. Other cases must have var type lists.

py_objects = [np.array([1.1, 2.2, 3.3]), np.array([]), np.array([4.4, 5.5])]
py_objects
[array([1.1, 2.2, 3.3]), array([], dtype=float64), array([4.4, 5.5])]
ak.Array(py_objects)
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>
np_array_dtype_O = np.array([[1.1, 2.2, 3.3], [], [4.4, 5.5]], dtype="O")
np_array_dtype_O
array([list([1.1, 2.2, 3.3]), list([]), list([4.4, 5.5])], dtype=object)
ak.Array(np_array_dtype_O)
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>

Conversion of strings and bytestrings

Python strings (type str) are converted to and from Awkward’s UTF-8 encoded strings and Python bytestrings (type bytes) are converted to and from Awkward’s unencoded bytestrings.

ak.Array(["one", "two", "three", "four"])
<Array ['one', 'two', 'three', 'four'] type='4 * string'>
ak.Array(["one", "two", "three", "four"]).tolist()
['one', 'two', 'three', 'four']
ak.Array([b"one", b"two", b"three", b"four"])
<Array [b'one', b'two', b'three', b'four'] type='4 * bytes'>
ak.Array([b"one", b"two", b"three", b"four"]).tolist()
[b'one', b'two', b'three', b'four']

Note

Advanced topic: the rest of this section may be skipped if you don’t care about internal representations.

Awkward’s strings and bytestrings are not distinct types, but specializations of variable-length lists. Whereas a list might be internally represented by a ListArray or a ListOffsetArray,

ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]]).layout
<ListOffsetArray64>
    <offsets><Index64 i="[0 3 3 5]" offset="0" length="4" at="0x000002941a30"/></offsets>
    <content><NumpyArray format="d" shape="5" data="1.1 2.2 3.3 4.4 5.5" at="0x000002943a40"/></content>
</ListOffsetArray64>

Strings and bytestrings are just ListArrays and ListOffsetArrays of one-byte integers with special parameters:

ak.Array(["one", "two", "three", "four"]).layout
<ListOffsetArray64>
    <parameters>
        <param key="__array__">"string"</param>
    </parameters>
    <offsets><Index64 i="[0 3 6 11 15]" offset="0" length="5" at="0x000002945a80"/></offsets>
    <content><NumpyArray format="B" shape="15" data="111 110 101 116 119 ... 101 102 111 117 114" at="0x000002947a90">
        <parameters>
            <param key="__array__">"char"</param>
        </parameters>
    </NumpyArray></content>
</ListOffsetArray64>

These parameters indicate that the arrays of strings should have special behaviors, such as equality-per-string, rather than equality-per-character.

ak.Array([[1.1, 2.2], [], [3.3]]) == ak.Array([[1.1, 200], [], [3.3]])
<Array [[True, False], [], [True]] type='3 * var * bool'>
ak.Array(["one", "two", "three", "four"]) == ak.Array(["one", "TWO", "thirty three", "four"])
<Array [True, False, False, True] type='4 * bool'>

(Without this overloaded behavior, the string comparison would yield [True, True, True] for "one" == "one" and would fail to broadcast "three" and "thirty three".)

Special behaviors for strings are implemented using the same ak.behavior mechanism that you might use to give special behaviors to Arrays and Records.

ak.behavior["string"]
awkward.behaviors.string.StringBehavior
ak.behavior["bytestring"]
awkward.behaviors.string.ByteStringBehavior
ak.behavior[np.equal, "string", "string"]
<function awkward.behaviors.string._string_equal(one, two)>

The fact that strings are really just variable-length lists is worth keeping in mind, since they might behave in unexpectedly list-like ways. If you notice any behavior that ought to be overloded for strings, recommend it as a feature request.

Conversion of dicts and tuples

Python dicts with string-valued keys are converted to and from Awkward’s record type with named fields. The data associated with different fields can have different types, but you generally want data associated with all instances of the same field to have the same type. Python dicts with non-string valued keys have no equivalent in Awkward Array (records are very different from mappings).

Python tuples are converted to and from Awkward’s record type with unnamed fields. Note that Awkward views Python’s lists and tuples in very different ways: lists are expected to be variable-length with all elements having the same type, while tuples are expected to be fixed-size with elements having potentially different types, just like a record.

In the following example, the "x" field has type int64 and the "y" field has type var * int64.

ak_array_rec = ak.Array([{"x": 1, "y": [1, 2]}, {"x": 2, "y": []}])
ak_array_rec
<Array [{x: 1, y: [1, 2]}, {x: 2, y: []}] type='2 * {"x": int64, "y": var * int64}'>
ak_array_rec.tolist()
[{'x': 1, 'y': [1, 2]}, {'x': 2, 'y': []}]

Here is the corresponding example with tuples:

ak_array_tup = ak.Array([(1, [1, 2]), (2, [])])
ak_array_tup
<Array [(1, [1, 2]), (2, [])] type='2 * (int64, var * int64)'>
ak_array_tup.tolist()
[(1, [1, 2]), (2, [])]

Both of these Awkward types, {"x": int64, "y": var * int64} and (int64, var * int64), have two fields, but the first one has names for those fields.

Both can be extracted using strings between square brackets, though the strings must be "0" and "1" for the tuple.

ak_array_rec["y"]
<Array [[1, 2], []] type='2 * var * int64'>
ak_array_rec["y", 1]
<Array [] type='0 * int64'>
ak_array_tup["1"]
<Array [[1, 2], []] type='2 * var * int64'>
ak_array_tup["1", 1]
<Array [] type='0 * int64'>

Note the difference in meaning between the "1" and the 1 in the above example. For safety, you may want to use the slot[0-9] methods with tuples:

ak_array_tup.slot1
<Array [[1, 2], []] type='2 * var * int64'>
ak_array_tup.slot1[1]
<Array [] type='0 * int64'>

Or possibly ak.unzip:

x, y = ak.unzip(ak_array_rec)
y
<Array [[1, 2], []] type='2 * var * int64'>
slot0, slot1 = ak.unzip(ak_array_tup)
slot1
<Array [[1, 2], []] type='2 * var * int64'>

That way, you can name the variables anything you like.

If fields are missing from some records, the missing values are filled in with None (option type: more on that below).

ak.Array([{"x": 1, "y": [1, 2]}, {"x": 2}])
<Array [{x: 1, y: [1, 2]}, {x: 2, y: None}] type='2 * {"x": int64, "y": option[v...'>

If some tuples have different lengths, the resulting Awkward Array is taken to be heterogeneous (union type: more on that below).

ak.Array([(1, [1, 2]), (2,)])
<Array [(1, [1, 2]), (2)] type='2 * union[(int64, var * int64), (int64)]'>

An Awkward Record is a scalar drawn from a record array, so an ak.Record can be built from a single dict with string-valued keys.

ak.Record({"x": 1, "y": [1, 2], "z": 3.3})
<Record {x: 1, y: [1, 2], z: 3.3} type='{"x": int64, "y": var * int64, "z": floa...'>

The same is not true for tuples. The ak.Record constructor expects named fields.

# ak.Record((1, [1, 2], 3.3))   would raise TypeError

Missing values: Python None

Python’s None can appear anywhere in the structure parsed by ak.from_iter. It makes all data at that level of nesting have option type and is represented in ak.to_list as None.

ak.Array([1.1, 2.2, None, 3.3, None, 4.4])
<Array [1.1, 2.2, None, 3.3, None, 4.4] type='6 * ?float64'>
ak.Array([1.1, 2.2, None, 3.3, None, 4.4]).tolist()
[1.1, 2.2, None, 3.3, None, 4.4]

Note

Advanced topic: the rest of this section describes the equivalence of missing record fields and record fields with None values, which is only relevant to datasets with missing fields.

As described above, fields that are absent from some records but not others are filled in with None. As a consequence, conversions from Python to Awkward Array back to Python don’t necessarily result in the original expression:

ak.Array([
    {"x": 1.1, "y": [1]                    },
    {"x": 2.2,                 "z": "two"  },
    {"x": 3.3, "y": [1, 2, 3], "z": "three"}
]).tolist()
[{'x': 1.1, 'y': [1], 'z': None},
 {'x': 2.2, 'y': None, 'z': 'two'},
 {'x': 3.3, 'y': [1, 2, 3], 'z': 'three'}]

This is a deliberate choice. It would have been possible to convert records with missing fields into arrays with union type (more on that below), for which ak.to_list would result in the original expression,

ak.concatenate([
    ak.Array([{"x": 1.1, "y": [1]                    }]),
    ak.Array([{"x": 2.2,                 "z": "two"  }]),
    ak.Array([{"x": 3.3, "y": [1, 2, 3], "z": "three"}]),
]).tolist()
[{'x': 1.1, 'y': [1]},
 {'x': 2.2, 'z': 'two'},
 {'x': 3.3, 'y': [1, 2, 3], 'z': 'three'}]

But typical datasets of records with different sets of fields represent missing fields, rather than entirely different types of objects. (Even in particle physics applications that mix “electron objects” with “photon objects,” both types of objects have the same trajectory fields "x", "y", "z" and differ in fields that exist for one and not the other, such as "charge" for electrons but not photons.)

The memory use of union arrays scales with the number of different types, up to \(2^n\) for records with \(n\) potentially missing fields. Option types of completely disjoint records with \(n_1\) and \(n_2\) fields use a memory footprint that scales as \(n_1 + n_2\). Assuming that disjoint records are a single record type with missing fields is a recoverable mistake, but assuming that a single record type with missing fields are distinct for every combination of missing fields is potentially disastrous.

Tuples of different lengths, on the other hand, are assumed to be different types because mistaking slot \(i\) for slot \(i + 1\) would create unions anyway.

ak.Array([
    (1.1, [1]               ),
    (2.2,            "two"  ),
    (3.3, [1, 2, 3], "three"),
]).tolist()
[(1.1, [1]), (2.2, 'two'), (3.3, [1, 2, 3], 'three')]

Union types: heterogeneous data

If the data in a Python iterable have different types at the same level of nesting (“heterogeneous”), the Awkward Arrays produced by ak.from_iter have union types.

Most Awkward operations are defined on union typed Arrays, but they’re not generally not as efficient as the same operations on simply typed Arrays.

The following example mixes numbers (float64) with lists (var * int64).

ak.Array([1.1, 2.2, [], [1], [1, 2], 3.3])
<Array [1.1, 2.2, [], [1], [1, 2], 3.3] type='6 * union[float64, var * int64]'>

The ak.to_list function converts it back into a heterogeneous Python list.

ak.Array([1.1, 2.2, [], [1], [1, 2], 3.3]).tolist()
[1.1, 2.2, [], [1], [1, 2], 3.3]

Any types may be mixed: numbers and lists, lists and records, missing data, etc.

ak.Array([[1, 2, 3], {"x": 1, "y": 2}, None])
<Array [[1, 2, 3], {x: 1, y: 2}, None] type='3 * ?union[var * int64, {"x": int64...'>

One exception is that numerical data are merged without creating a union type: integers are expanded to floating point numbers.

ak.Array([1, 2, 3, 4, 5.5, 6.6, 7.7, 8, 9])
<Array [1, 2, 3, 4, 5.5, 6.6, 7.7, 8, 9] type='9 * float64'>

But booleans are not merged with integers.

ak.Array([1, 2, 3, True, True, False, 4, 5])
<Array [1, 2, 3, True, True, False, 4, 5] type='8 * union[int64, bool]'>

As described above, records with different sets of fields are presumed to be a single record type with missing values.

ak.type(ak.Array([
    {"x": 1.1, "y": [1]                    },
    {"x": 2.2,                 "z": "two"  },
    {"x": 3.3, "y": [1, 2, 3], "z": "three"}
]))
3 * {"x": float64, "y": option[var * int64], "z": option[string]}

But tuples with different lengths are presumed to be distinct types.

ak.type(ak.Array([
    (1.1, [1]               ),
    (2.2,            "two"  ),
    (3.3, [1, 2, 3], "three"),
]))
3 * union[(float64, union[var * int64, string]), (float64, var * int64, string)]

More control over conversions

The conversions described above are applied by ak.from_iter when it maps data into an ak.ArrayBuilder. For more control over the conversion process (e.g. to make unions of records), use ak.ArrayBuilder directly.