Seems like a perfectly fine house rule. Alternatively, pick a background with a Lore you only want to stay Trained in, and then use Additional Lore for the Lore you really want.
Lores from Downtime sound great. Languages through downtime would be awesome too. Those systems in Society would be amazing and fix a few of my gripes with how 2E feels in Society. Only 2 years for them to approve it...