There was little surprise when young Asao tested as being able to communicate with the kami. With his lineage it was expected of every child, of course. What was unexpected was the apparent communication he was able to have with his great-great-grandfather, Isawa Atsuji, a former Elemental Master of Fire. He seemed to be guiding the young Asao to form better relationships with the kami, to help develop him into a potentially very powerful shugenja. His family taught Asao to hide his communication with his ancestor, as such active communication is far from common, thus far from traditional, and that would make it a question on his capability.
As he got older, his instructors realized that Asao was directly connecting to the Void itself, and being Isawa, they understood the potential of his ability better than most, and his lessons were altered in an attempt to develop him even further beyond most of the Isawa student capabilities. The new lesson focus had an unexpected alteration to Asao's development, though, as he began connecting better to the water kami than the fire kami as he had been. Once it became clear that Asao was settling into an affinity to water, Atsuji became enraged that his chosen descendent would turn away from Fire so easily and quickly. His ancestral spirit became confrontational instead of supportive, and during a training opportunity working with the kami of fire, the kami flared out of control and badly burned Asao's face, scarring him badly. Atsuji has made it clear multiple times that Asao's treachery in turning away from the kami of fire deserved far more punishment than he has received.
The issue is not that Asao is unfaithful to his ancestor. The issue is not that Asao has unusual troubles as a shugenja. The issue is that his ancestor considers fire affinity a mandatory part of following in his footsteps, instead of accepting that other affinities could prove just as useful to the clan and family. The final affinity to water strongly suggests that the early indications of an affinity to fire were likely due to the direct involvement of Atsuji's spirit, and not reflective of Asao at all.
