It is infact used as someone's doing like ' karam kiye ja phal ki iccha mat kar ye insaan jaise karam karega waisa phal dega bhagwan'. So this means its used like Karma or ones doing which in no way can mean fate.
Or i have heard it being used as 'Bhagan ka karam hum pe bana rahe' means hope god gives blesses us with his generosity.
Or Karamoon ki saza like what we have done or might have done in past or present is not right so we get punished. Even here it is used as our doings which makes our fate.