LRRK2 serine/threonine kinase is associated with inherited Parkinson’s disease. LRRK2 phosphorylates a subset of Rab GTPases within their switch 2 motif to control their interactions with effectors. Recent work has shown that the metal-dependent protein phosphatase PPM1H counteracts LRRK2 by dephosphorylating Rabs. PPM1H is highly selective and closely related PPM1J/M exhibit virtually no activity toward substrates such as Rab8a phosphorylated at T72 (pT72). Here we have identified the structural determinant of PPM1H specificity for Rabs. The crystal structure of PPM1H reveals a conserved catalytic domain that adopts a β-sandwich fold. The striking difference is that PPM1H has evolved a 110-residue flap domain that punctuates the catalytic domain. The flap domain distantly resembles tudor domains that interact with histones in the context of epigenetics. Cellular assays and 3-D modelling suggest that the flap domain encodes the docking motif for phosphorylated Rabs. Consistent with this hypothesis, a PPM1J chimera with the PPM1H flap domain dephosphorylates pT72 of Rab8a with a higher specific activity than PPM1H. Therefore, PPM1H has acquired a Rab-specific interaction domain within a conserved phosphatase fold.