Deer antlers, usually harvested annually on a farm, are an accessible material used to determine the exposition to potentially toxic elements, PTEs, during growth. Moreover, the study of antlers from animals of different ages allows the assessment of long-term exposition to these elements. The aim of the study was to analyze the concentration of eight potentially toxic elements (Cd, Pb, As, Ba, Ni, Sr, La, Ce) in individual positions of the antlers (first, second, and third position, corresponding to the stages of development and life of these animals) and in the food that the animals consumed during the growth of individual antler fragments, depending on the age of the farmed fallow deer (Dama dama). The mineral composition of samples was analyzed using inductively coupled plasma mass spectrometry (ICP-MS). The analysis included 31 male deer aged 2–8 years old. The average concentration of Pb, Ba, and Ni was higher in the second position of the antler, and As, La, and Ce in the third position. In addition, the oldest individuals showed a higher Cd, Pb, and As concentration in the third position. A significant positive relationship was found between the age of animals and accumulation of As (r = 0.582, p < 0.05), as well as Ba and Sr (r = −0.534, r = −0.644 at p < 0.05, respectively). The average content of Ba and Sr also significantly negatively depended on body mass and antler mass stags (r = −0.436, r = −0.515 at p < 0.05, respectively). Cd concentration in feed was significantly higher in June compared to winter, spring, and later summer (p < 0.05). On the other hand, the concentration of Ba in food was significantly higher in spring and winter than in early and later summer (p < 0.05). An increase in the PTEs in the pasture determined the concentration of these components in fallow deer antlers.