MemoryError: Unable to allocate 163. GiB for an array with shape (1092172, 20002) and data type float64

[wuxiangning]

Hello!When I was running a large amount of data interaction, I had such a problem, how can I solve this problem?
This is my code:
cpdb_file_path = 'cellphonedb.zip'
metadata = pd.read_csv(meta_file_path, sep = '\t')
metadata.head(3)
counts_file_path ="F:/x/Rdata/Selected_data/normolized/241016_allzzm_endo.h5ad"
pdb_results = cpdb_statistical_analysis_method.call(
cpdb_file_path = cpdb_file_path,
meta_file_path = meta_file_path,
counts_file_path =counts_file_path,
counts_data = 'hgnc_symbol',
output_path = out_path)

Reading user files...
MemoryError: Unable to allocate 163. GiB for an array with shape (1092172, 20002) and data type float64

Help!!!

[ktroule]

Hi.

You can either downsample the adata yourself or use the option subsampling which will automatically perform the downsampling. More information here.

Regards

[wuxiangning]

I have tried adjusting the subsampling to subset my large dataset, but I still encounter the same error. Out of frustration, I can only subset the data in R first and then call it into Python for computation. This seems redundant, and I’m not sure if there is an issue with the subsampling.
Thank you.

[ktroule]

If "MemoryError" error still persists I would suggest to downsample the datasets even more as the limiting factor seems to be the machine you are using.

To subsample the dataset in scanpy you can use scanpy.pp.subsample(data, fraction=None, *, n_obs=None, random_state=0, copy=False)

Regards