The flag was my invention yes, but it fit into the other scenes we saw of drawing where he came from, and sights of his home that he knew he'd never get back to.
I never heard any reference to the Silent One getting up and leaving. The implication was that he died in prison. As to why he wouldn't have gotten up and walked out, I think being traumatized by the knowledge that he's so far from home with no comprehensible means of getting home dissuaded him from walking out.
Each theory is as good as another of course.
