In biological microscopy, light scattering represents the main limitation to image at depth. Recently, a set of wavefront shaping techniques has been developed in order to manipulate coherent light in strongly disordered materials. The Transmission Matrix approach has shown its capability to inverse the effect of scattering and efficiently focus light. In practice, the matrix is usually measured using an invasive detector or low-resolution acoustic guide stars. Here, we introduce a non-invasive and all-optical strategy based on linear fluorescence to reconstruct the transmission matrices, to and from a fluorescent object placed inside a scattering medium. It consists in demixing the incoherent patterns emitted by the object using low-rank factorizations and phase retrieval algorithms. We experimentally demonstrate the efficiency of this method through robust and selective focusing. Additionally, from the same measurements, it is possible to exploit memory effect correlations to image and reconstruct extended objects. This approach opens up a new route towards imaging in scattering media with linear or non-linear contrast mechanisms.