Updated project metadata. Characterizing the human leukocyte antigen (HLA) bound ligandome by mass spectrometry (MS) holds great promise for developing vaccines and drugs for immune oncology. Still, the identification of such non-tryptic peptides presents substantial computational challenges. To address these, we synthesized >300,000 peptides within the ProteomeTools project representing HLA class I & II ligands and products of the proteases AspN and LysN and analyzed these by multi-modal LC-MS/MS. The resulting data enabled training of a single model using the deep learning framework Prosit that shows outstanding prediction accuracy of fragment ion spectra for tryptic and non-tryptic peptides. Applying Prosit demonstrates that the identification of HLA peptides can be improved by 50-300% on average, that proteasomal HLA peptide splicing may not exist and that additional neo-epitopes that elicit an immune response can be identified from patient tumors. Together, the provided peptides, spectra and computational tools substantially expand the scope of immunopeptidomics workflows.