You, the document poster, control the privacy settings with regards to that document. Privacy Settings on the communication that the document is attached to determine who can see the document that you upload. For example, if you attach a document to a public wall post to a colleague, then anyone can search for and find your document. However, if you attach a document to a private Project wall post then only you and the other members of the private project can see the document.
Similarly, privacy settings can be made on Teams such that no communication or document can be viewed by non-members. “Direct Messages” can be set to private and these privacy settings apply to document attachments as well.