To defend your city you need to open your gates! Click on your rally point and click on the open gates checkbox. This will allow your troops to sally forth and defend your castle. Otherwise only your defenses (archer towers, traps, abatis, etc.) will defend you. If your gates are closed your troops will not fight to defend you.
Scouts also do not defend unless your gates are open. Your enemies can scout you without worrying about your defenses unless your gates are open!
