The person X who asked "were you in the building at the time?" was the same person X who asked "did you shoot the president?" immediately before. To be clear, this person X asked this exact same question as another person before him. So if Oswald was directing answers at this person X, let's look at the first response: Question from person X "did you shoot the president? Response "I work in that building". That doesn't make sense right? It appears Oswald was not responding to this person X. Next we have the "were you in the building at the time?" question from person X. Oswald responds with a sarcastic "naturally" as if he felt that his previous answer was sufficient, an answer to someone else. not this person X. In a nutshell, we don't know the context of Oswald's response when he said "I work in that building", and therefore we don't know what the answer "naturally, if I work in that building, yes sir" actually relates to.