The Arctic regions experience strong seasonality and are largely affected by increasing temperature. This is particularly evident in the Kongsfjorden, which is surrounded by glaciers and is affected by seasonal and annual changes in temperature. It is largely influenced by glacial meltwater bringing in fluvial inputs such as total suspended matter (TSM) and warm Atlantic waters, which could alter the phytoplankton community. Seven stations in the Kongsfjorden representing glacier-influenced head (KF7, KF6, KF5), mid (KF4, KF3), and open region of the fjord (KF2 and KF1) were considered to evaluate the effect of TSM on phytoplankton community structure, abundance, chlorophyll a (Chl a), diversity index, evenness, and richness during summer 2011 (June) and 2018 (August) and related with atmospheric and hydrological parameters. The annual average atmospheric temperature (AAAT) over Ny-Ålesund showed an increase in temperature by a degree from −3.52 °C in 2011 to −2.44 °C in 2018, while the summer average atmospheric temperature (SAAT) over the same period increased from 5.80 to 6.16 °C. Increased freshening of the fjord led to an increase in TSM during 2018 which coincided with a decrease in Chl a by an order of magnitude. Although sea surface temperature (SST) was warmer in 2011, TSM was higher in 2018. The number of phytoplankton groups identified decreased from 11 in 2011 to 4 in 2018. A distinct alteration in phytoplankton community structure was observed from head fjord to open fjord with higher diversity observed during 2011 compared to 2018. This work highlights the effect of TSM on the phytoplankton community in the Kongsfjorden.