categorical variable detection in new SCORCH dataset metadata

[carlocolantuoni]

brian recently uploaded 4 new SCORCH datasets to NeMO Analytics, named:

SCORCH_PFC_Excite_SKS
SCORCH_PFC_Inhib_SKS
SCORCH_PFC_Glia_SKS
SCORCH_PFC_NNC_SKS

They contain very similar metadata columns (brian can you tell us how similar? are they identical?).

However, only in the "SCORCH_PFC_NNC_SKS" dataset has nemo analytics detected categorical variables (such as "status" that contains, ) so that we can use them in violin plots etc.

i do think there are some empty and NA values in the columns of interest - could this throw the auto detect off?

[brianherb]

Columns should be identical, there are three columns that refer to cell types:

'subtype','subclass','class'

and one that has drug/HIV information - 'Status'. The values are 'CTR' (control), 'OUD' (opioid use disorder) 'HIV', 'HIVOUD' - if there are any NA's these can be changed to 'CTR' if this is causing the column to not be considered a categorical variable.

[brianherb]

Is there a way to add a new metadata column to an existing dataset?

[adkinsrs]

I'll poke around on the datasets on the server and see what's up.

Related tickets:
#229 would resolve this before upload
#596 same "upload metadata post-upload" request from @carlocolantuoni

[adkinsrs]

Looked at the "Excite" dataset and saw the following dtype assignments

nCount_RNA          float64
nFeature_RNA          int64
percent.mt          float64
subtype              object
subclass             object
class                object
study                object
sample               object
Status               object
seurat_clusters    category
UMAP_1              float64
UMAP_2              float64
dtype: object

For the subtype and subclass categories, I see patterns of "#-#" in some of the group names, which the uploader I believe is interpreting as an equation. So it cannot resolve the entire category of data as either a string nor a number type so it makes it a mixed object dtype. For the class, I only see one group (ExN), and so not entirely sure why it thinks that's an object dtype. I'm sure the other 3 datasets are behaving the same way.

What I can do is go through the 4 datasets and just force them to be categorical, write to a new file, and move that back to the original file path.

[adkinsrs]

Found out only 2 of the 4 datasets had the "object" dtype instead of "category for those columns I listed above. I corrected those two and updated the h5ad object.

[adkinsrs]

Going to close this as I believe the original issue is resolved. Feel free to reopen if you still cannot curate violin plots.