Uh.. I think Empire? How do you tell? When I began the game I followed Hadvar out of the starting area. So I don't know.
Your decision at the game beginning doesn’t mean anything in fact - you’ll see it. You still can join either of them :) I just thought you already joined, that’s why asked. What quests are you doing now?