Purpose The purpose of this study was to provide an analysis of imaging performance in digital mammography, using indirect detector instrumentation, by combining the Linear Cascaded Systems (LCS) theory and the Signal Detection Theory (SDT). Observer performance was assessed, by examining frequently employed detectors, consisting of phosphor-based X-ray converters (granular Gd2O2S:Tb and structural CsI:Tl), coupled with the recently introduced complementary metal-oxide-semiconductor (CMOS) sensor. By applying combinations of various irradiation conditions (filter-target and exposure levels at 28 kV) on imaging detectors, our study aimed to find the optimum system set-up for digital mammography. For this purpose, the signal to noise transfer properties of the medical imaging detectors were examined for breast carcinoma detectability. Methods An analytical model was applied to calculate X-ray interactions within software breast phantoms and detective media. Modeling involved: (a) three X-ray spectra used in digital mammography: 28 kV Mo/Mo (Mo: 0.030 mm), 28 kV Rh/Rh (Rh: 0.025 mm) and 28 kV W/Rh (Rh: 0.060 mm) at different entrance surface air kerma (ESAK) of 3 mGy and 5 mGy, (b) a 5 cm thick Perspex software phantom incorporating a small Ca lesion of varying size (0.1–1 cm), and (c) two 200 μm thick phosphor-based X-ray converters (Gd2O2S:Tb, CsI:Tl), coupled to a CMOS based detector of 22.5 μm pixel size. Results Best (lowest) contrast threshold (CT) values were obtained with the combination: (i) W/Rh target-filter, (ii) 5 mGy (ESAK), and (iii) CsI:Tl-CMOS detector. For lesion diameter 0.5 cm the CT was found improved, in comparison to other anode/filter combinations, approximately 42% than Rh/Rh and 55% than Mo/Mo, for small sized carcinoma (0.1 cm) and approximately 50% than Rh/Rh and 125% than Mo/Mo, for big sized carcinoma (1 cm), considering 5 mGy X-ray beam. By decreasing lesion diameter and thickness, a limiting CT (100%) was occurred for size values less than 0.2 cm. Conclusion CT was found to be affected by the selection of target/filter and exposure combination. It was found that the optimum thickness of CsI:Tl was approximately 190 μm and for Gd2O2S:Tb 120 μm for the studied energy and ESAK range.