The huge volumes of currently generated agricultural waste pose a challenge to the economy of the 21st century. One of the directions for their reuse may be as fertilizer. Spent mushroom substrate (SMS) could become an alternative to manure (M). A three-year field experiment was carried out, in which the purpose was to test and compare the effect of SMS alone, as well as in multiple variants with mineral fertilization, and in manure with a variety of soil quality indices—such as enzymatic activity, soil phytotoxicity, and greenhouse gas emissions, i.e., CO2. The use of SMS resulted in significant stimulation of respiratory and dehydrogenase activity. Inhibition of acid phosphatase and arylsulfatase activity via SMS was recorded. SMS showed varying effects on soil phytotoxicity, dependent on time. A positive effect was noted for the growth index (GI), while inhibition of root growth was observed in the first two years of the experiment. The effect of M on soil respiratory and dehydrogenase activity was significantly weaker compared to SMS. Therefore, M is a safer fertilizer as it does not cause a significant persistent increase in CO2 emissions. Changes in the phytotoxicity parameters of the soil fertilized with manure, however, showed a similar trend as in the soil fertilized with SMS.