Based on a persons upbringing, ofcourse they have reason to make assumptions about you, wrongly or rightly.

They could think you didnt know, or based on their way of life, tradition and all, it would show lack of respect. Its all boiling down to cultural differences.

English folks would most definitely frown upon someone not taking of the cap in a tee party for instance.