Invasive lobular breast carcinoma (ILC) shows specific stromal features, T lymphocyte infiltration (TIL) being associated with poor prognosis. Here, we reveal the involved mechanism by performing single cell RNA sequencing, combined immunohistochemistry, deconvolution of bulk RNA sequencing from large retrospective ILC series, and functional assays using primary cells. We show that ILC accumulates FAP+ inflammatory cancer-associated fibroblasts (iCAF) through a previously undescribed mechanism mediated by E-cadherin/CDH1 on CAF plasticity. Indeed, CDH1 inactivation in ILC cancer cells prevents differentiation of iCAF into myofibroblastic CAF (myCAF), leading to iCAF accumulation. In turn, iCAF attract TILs into the tumor center and shape their spatial organization in ILC. Subsequently, CDH1-inactivated ILC cancer cells promote immune escape by lack of retention and activation of ITGAE-expressing resident memory CD8+ T lymphocytes (TRM). Hence, our study uncovers reciprocal interactions between CDH1-inactivated cancer cells, iCAF and CD8+ TRM, gaining insight into the underlying mechanism associated with the distinctive stromal reaction observed in ILC and revealing why and how TILs have a poor prognosis in ILC patients, a mechanism that can be generalized to other CDH1-inactivated cancer types.