Reminder that the Winter Soldier only operated for 50 years of the over 70 years Bucky was captive.
It took over 20 years for them to break Bucky down enough to use him, and even then, not only did he need to be convinced what he was doing was good, but he constantly needed to be wiped to be kept complacent.
It only took seeing Steve and hearing 'I'm with you to the end of the line' to bring that all crashing down.