Changes in the rate and fidelity of mitochondrial protein synthesis impact the metabolic and physiological roles of mitochondria. Here we explored how environmental stress in the form of a high fat diet modulates mitochondrial translation using mutant mice with error-prone (Mrps12ep/ep) or hyper-accurate (Mrps12ha/ha) mitochondrial ribosomes. We find that while, metabolically both mutations are beneficial in reducing body weight, decreasing circulating insulin and increasing glucose tolerance they cause tissue specific defects when placed on a high fat diet. In contrast to the effect of the mutations on a normal diet the Mrps12ha/ha mice show more pronounced phenotypic and molecular defects as a result of the slow, accurate nature of their protein synthesis. While the Mrps12ep/ep mice accumulate more fat, exhibit larger vacuoles in the liver and lose glucose tolerance, the Mrps12ha/ha mice develop severe hypertrophic cardiomyopathy and hypoxia due to the stress of the diet, showing that benefit or detriment of error-prone and hyper-accurate protein synthesis in mitochondria is dependent on tissue and environmental conditions.