The Cassini-Huygens mission has been observing Titan since October 2004, resulting in over 70 targeted flybys. Titan's thermosphere is sampled by the Ion and Neutral Mass Spectrometer (INMS) during several of these flybys. The measured upper atmospheric density varies significantly from pass to pass. In order to quantify the processes controlling this variability, we calculate the nitrogen scale height for a variety of parameters related to the solar and plasma environments and, from these, we infer an effective upper atmospheric temperature. In particular, we investigate how these calculated scale heights and temperatures correlate with the plasma environment. Measured densities and inferred temperatures are found to be reduced when INMS samples Titan within Saturn's magnetospheric lobe regions, while they are enhanced when INMS samples Titan in Saturn's plasma sheet. Finally the data analysis is supplemented with Navier-Stokes model calculations using the Titan Global Ionosphere Thermosphere Model. Our analysis indicates that, during the solar minimum conditions prevailing during the Cassini tour, the plasma interaction plays a significant role in determining the thermal structure of the upper atmosphere and, in certain cases, may override the expected solar-driven diurnal variation in temperatures in the upper atmosphere.