We are probably using the term "text file" in a slightly different way.
First of all, a PDF file is *NOT* a text file.
A "simple text" file in a Windows environment typically would have a TXT extension, would typically be encoded as ASCII text (although UTF-8 and other international encodings are becoming more common), and typically would be opened with a very simple text editor such as Notepad. Such "simple text" files usually have no formatting codes or other meta data. And I certainly agree that PDF files are not "simple text" files.
Files such as RTF, Microsoft Word, WordPerfect, HTML, and (yes) PDF are "rich text" files. Such "rich text" files really are text files rather than graphics files, but they typically include formatting codes and other meta data, and they often include embedded or linked graphics files.
To this end, we can observe that RTF and HTML files can be opened with Notepad just fine. As such, you can see the formatting codes and other meta data as text, but to see the formatting codes and other meta data rendered properly you would have to open an RTF file in a word processor and you would have to open an HTML file in a Web browser or other program that can render HTML.
Microsoft Word documents really can't be opened with Notepad in a way that makes any sense because the formatting codes and meta data are binary, but that doesn't mean that Microsoft Word documents aren't "rich text" files, because they are. Whether PDF's can be opened with Notepad in a way that makes sense depends a little bit on the specific PDF. The text in a PDF is fine in Notepad, as are the formatting codes and other meta data. The only issue you have with opening a PDF file in simple text editor such as Notepad is that embedded images are binary and Notepad can't display binary data in hexadecimal or other format that makes sense. But a slightly more sophisticated text editor than Notepad - and I mean a fairly simple text editor, not a graphics editor - can certainly deal with the binary meta data in a Microsoft Word file or binary graphics images in a PDF just fine, so long only as the binary data can be displayed in hexadecimal.
Way back at the beginning of time, I had to support some applications that created PDF files, and as such I had to learn how to hand code PDF files using a tool just as primitive as Notepad. And to this day, I hand code HTML files on my personal Web page using a tool just as primitive as Notepad. So I'm confident that these are text files rather than graphics files, albeit "rich text" files which include formatting codes and meta data and maybe embedded or linked graphics.