
BellyBeard |

I would say it's allowed, and wouldn't drop into combat unless they crit the save and are very unhappy, but the player would need to be careful. The target may not realize you are casting unless they crit, but other observers can always tell if you don't use Conceal Spell, and if they inform the target they aren't likely to be helpful with their spells after that.
This all assumes you have means of convincing them to come to your workshop where you're crafting, since that would be a task in itself for your average spellcaster who is no doubt very busy.
Either way, even if they refuse to help, I don't think it ends the downtime activity, in that I don't think previous crafting days are wasted or anything like that. Downtime is more fluid and abstract.