The Man in the Yellow Hat

Somebody help me.

Struck by sudden inspiration, I spent twelve hours on Wednesday writing a story featuring a villain called "the man in the yellow hat." I was immensely proud of it. Last night I was riding home from work with Girlfriend, and said something like,

"Yes, I just finished it. So I'll tell you about it. And now I can pay attention to you again, now that my mind isn't entirely occupied"--here I filled my voice with a tone of cosmic terror far surpassing its customary level of menace-- "by the man in the yellow hat!"

I looked at her expectantly.

"Um, Curious George?" said she.

"FUCK," said I.

***

The man in the yellow hat was just an image I thought was unassuming but memorable. So ... now I don't know what to do. Recolor his hat? But what color? The man in the orange hat? No, orange is too loud a color. The man in the silver hat? Well, maybe, he is (after all) a robot.

So...

Purple? Black? White? Blue? Red? Grey? I don't know! Help!