Automated quantification of DNA demethylation effects in cells via 3D mapping of nuclear signatures and population homogeneity assessment.

Today's advanced microscopic imaging applies to the preclinical stages of drug discovery that employ high-throughput and high-content three-dimensional (3D) analysis of cells to more efficiently screen candidate compounds. Drug efficacy can be assessed by measuring response homogeneity to treatment within a cell population. In this study, topologically quantified nuclear patterns of methylated cytosine and global nuclear DNA are utilized as signatures of cellular response to the treatment of cultured cells with the demethylating anti-cancer agents: 5-azacytidine (5-AZA) and octreotide (OCT). Mouse pituitary folliculostellate TtT-GF cells treated with 5-AZA and OCT for 48 hours, and untreated populations, were studied by immunofluorescence with a specific antibody against 5-methylcytosine (MeC), and 4,6-diamidino-2-phenylindole (DAPI) for delineation of methylated sites and global DNA in nuclei (n = 163). Cell images were processed utilizing an automated 3D analysis software that we developed by combining seeded watershed segmentation to extract nuclear shells with measurements of Kullback-Leibler's (K-L) divergence to analyze cell population homogeneity in the relative nuclear distribution patterns of MeC versus DAPI stained sites. Each cell was assigned to one of the four classes: similar, likely similar, unlikely similar, and dissimilar. Evaluation of the different cell groups revealed a significantly higher number of cells with similar or likely similar MeC/DAPI patterns among untreated cells (approximately 100%), 5-AZA-treated cells (90%), and a lower degree of same type of cells (64%) in the OCT-treated population. The latter group contained (28%) of unlikely similar or dissimilar (7%) cells. Our approach was successful in the assessment of cellular behavior relevant to the biological impact of the applied drugs, i.e., the reorganization of MeC/DAPI distribution by demethylation. In a comparison with other metrics, K-L divergence has proven to be a more valuable and robust tool for categorization of individual cells within a population, with potential applications in epigenetic drug screening.