Here is a short introduction to table formats which represents the main point of contention at the hackathon. We are looking at a single image with 3 channels and 1 label image (aka object type) containing 2 nuclei. Say we measure mean and sum intensity in each label region for all channels. Since there is only one site and object type we can reduce the object ID from what we used earlier for unique IDs across the full dataset (roi, o, label) to just being the integer ID in the label image (label). There are four things to keep track of: The ID of the label object, the channel ID, the intensity aggregation (feature) that was computed and finally, the values that resulted from the aggregations.

In [1]:

import polars as pl
import numpy as np
from itertools import product

rng = np.random.default_rng(seed=42)

labels = list(range(1, 3))
channels = [f"ch{i}" for i in range(3)]
features = ["Mean", "Sum"]
values = rng.normal(size=(len(labels), len(channels), len(features)))

The main two camps at the hackathon were arguing in favor of tables tidy in terms of objects, which means the columns of the table are derived from the Cartesian product of channels and features, for many data frame libraries this means they need to be joined as column names usually can only be strings like so:

In [2]:

df_wide = pl.DataFrame(
    {
        "label": labels,
        **{
            f"c={ch}_f={f}": vals
            for (ch, f), vals in zip(
                product(channels, features), values.reshape(len(labels), -1).T
            )
        },
    }
)
df_wide

shape: (2, 7)

label	c=ch0_f=Mean	c=ch0_f=Sum	c=ch1_f=Mean	c=ch1_f=Sum	c=ch2_f=Mean	c=ch2_f=Sum
i64	f64	f64	f64	f64	f64	f64
1	0.304717	-1.039984	0.750451	0.940565	-1.951035	-1.30218
2	0.12784	-0.316243	-0.016801	-0.853044	0.879398	0.777792

The main argument of its advocates was that this format maps well to the analysis done with such tables, where we often want to reason about nuclei. I would therefore call this format tidy with respect to label objects. This is where you need to go if you want to compute a UMAP embedding or train a regression model using one of them as a target. The disadvantage is that column names contain both the channel and aggregation function so a selection of only ch0 will involve processing of column names and string parsing. Achieving this robustly, requires knowledge on how the column names were constructed.

The second format that was proposed looked like this:

In [3]:

df_tall = pl.DataFrame(
    {
        "label": [p[0] for p in product(labels, channels)],
        "c": [p[1] for p in product(labels, channels)],
        **{
            f"f={f}": vals
            for f, vals in zip(
                features, values.reshape(len(labels) * len(channels), len(features)).T
            )
        },
    }
)
df_tall

shape: (6, 4)

label	c	f=Mean	f=Sum
i64	str	f64	f64
1	"ch0"	0.304717	-1.039984
1	"ch1"	0.750451	0.940565
1	"ch2"	-1.951035	-1.30218
2	"ch0"	0.12784	-0.316243
2	"ch1"	-0.016801	-0.853044
2	"ch2"	0.879398	0.777792

The argument that was made in favor of this format went along the lines of flexibility. What I like about it personally is that there is no string parsing needed for accessing each value, and it is possible to write a table schema without having to know the channel names beforehand. The latter point is the main reason I chose this format in the end. We want people to use descriptive channel names, meaning that tall tables are the only way we can specify a table schema—a prerequisite of having tables be part of robust downstream processing—while allowing user defined channel names.

Here are two more options, that were not seriously proposed by anyone, just in case there are some wide/tall table extremists out there… The first one is in my view an interesting extreme, essentially a single linear buffer of f64 values with all the ‘metadata’ stored as categorical coordinates:

In [4]:

df_tallest = pl.DataFrame(
    {
        "label": [p[0] for p in product(labels, channels, features)],
        "c": [p[1] for p in product(labels, channels, features)],
        "f": [p[2] for p in product(labels, channels, features)],
        "val": values.flatten(),
    }
)
df_tallest

shape: (12, 4)

label	c	f	val
i64	str	str	f64
1	"ch0"	"Mean"	0.304717
1	"ch0"	"Sum"	-1.039984
1	"ch1"	"Mean"	0.750451
1	"ch1"	"Sum"	0.940565
1	"ch2"	"Mean"	-1.951035
…	…	…	…
2	"ch0"	"Sum"	-0.316243
2	"ch1"	"Mean"	-0.016801
2	"ch1"	"Sum"	-0.853044
2	"ch2"	"Mean"	0.879398
2	"ch2"	"Sum"	0.777792

I am slightly intrigued by this one, but it is impractical since we would lock ourselves into a single data type for val. I guess df_tall is the sweet spot.

Finally, what looks like another linear buffer (i.e. df_tallest.transposed()), but from a columnar data processing perspective is just plain silly, and with all metadata crammed into a string. A format that—if nothing else—I hope we can all collectively disagree on:

In [5]:

df_widest = pl.DataFrame(
    {
        f"label={lbl}_c={c}_f={f}": val
        for (lbl, c, f), val in zip(
            product(labels, channels, features), values.flatten()
        )
    }
)
df_widest

shape: (1, 12)

label=1_c=ch0_f=Mean	label=1_c=ch0_f=Sum	label=1_c=ch1_f=Mean	label=1_c=ch1_f=Sum	label=1_c=ch2_f=Mean	label=1_c=ch2_f=Sum	label=2_c=ch0_f=Mean	label=2_c=ch0_f=Sum	label=2_c=ch1_f=Mean	label=2_c=ch1_f=Sum	label=2_c=ch2_f=Mean	label=2_c=ch2_f=Sum
f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64
0.304717	-1.039984	0.750451	0.940565	-1.951035	-1.30218	0.12784	-0.316243	-0.016801	-0.853044	0.879398	0.777792