Interesting question. I'd say relates to the relatively unidimensional, fixed, intrapersonal nature of writing, as opposed to the dynamic, multidimensional, interpersonal nature of speech.
In speech you can draw on tone, pitch, intensity and timbre of voice, quickness and variations in clarity of elocution, facial expressions, body movements and posture, hand and arm gestures, plus shared personal history with the speaker, shared environment (the place you're talking in), a common situation, heck even smells, all that to convey your point accross. You can rephrase something you've badly phrased, you can ask and answer to questions, and you can adapt all those elements on the fly depending on the reaction of your audience.
All that is very fine-tuned and yet mostly subconscious, because as social animals we've spent the better part of the last million years talking to one another and it has become ingrained in us to the point we don't notice most of what's happening in a conversation, even though all of it affects us. Add how essential communication has been to the survival of our species, and how much of our life we spent talking (sometimes several hours a day) and you see why we're so good at it.
Heck we're so good at communication we don't even need words in some cases. Friends and lovers understand each other with a glare, with a smile, sometimes less. I had two friends in hs who often played basketball together, they once told me if one of them threw the ball behind him, without even looking, he'd know the other would catch it. They had enough shared history to understand each other (to some extent) without words and at a distance. And that kind of stuff is ubiquitous in daily life.
Words are almost overkill at this point.
Now compare that to writing. You pretty much only have the words, with their natural ambiguity, their context-dependence, their evolving meaning, their relating to a world and an intent that might be unclear or unappropriately grasped by both reader and writer. You have to use them to convey something to someone you don't know, who will perhaps not be born in your lifetime, who might not share a language or a continent with you, who cannot give you feedback. Thus you have to establilsh a standard of conversation, with a unknown, at times unknownable interlocutor, accross unpredictable distances and differences, and with only one (the latest, the most disembodied, perhaps the flimsiest) of the means of conversation. That's akin to trying to complete a triathlon with two hands tied behind your back and blindfolded.
No wonder it's a skill. Even conversation is a skill. Even speaking in more 'general' conditions (in public instead of private for instance) is a skill. The more general the conditions, the less tied to a narrow situation you can immediately grasp, the more skill required to maintain a proper level of attention, engagement and understanding. In writing you face the most general condition of all.
