A subset of Cancer-Associated Fibroblasts (CAF-S1) mediates immunosuppression in breast cancers (BC), but its heterogeneity and its impact on immunotherapy response remain unknown. Here, we identify 8 CAF-S1 clusters by analyzing more than 18000 single CAF-S1 fibroblasts from BC. We validate the 5 most abundant clusters by flow cytometry and in silico analyses in other cancer types, highlighting their relevance. Myofibroblasts from clusters 0 and 3, characterized by extra-cellular matrix proteins and TGF signaling respectively, are indicative of primary resistance to immunotherapies. Cluster 0 up-regulates PD-1 and CTLA-4 protein levels in regulatory T lymphocytes (Tregs), which in turn increases CAF-S1 cluster 3 cellular content. Thus, our study highlights a positive feedback loop between specific CAF-S1 clusters and Tregs and uncovers their role in immunotherapy resistance.