Let's look at Christ's atonement a little differently. In Psalm 51, David tells us that God isn't really interested in a sacrifice for David's sins. God wants "a broken heart and a contrite spirit." These are the sacrifices that God is interested in.
Later, in Acts 10, Peter tells Cornelius, "In any nation, the one that fears God and works righteousness is accepted by him."
God was already merciful, even before Christ's atonement. He would forgive the repentant, and he would accept those who chose righteousness.
More than anything, Jesus' death was for us. He brought about an awesome new covenant. David was a man who had the Spirit of God under the old covenant, but very few others did. Some priests, some prophets, some kings, men like Moses, Saul, and the judges. However, almost all normal people were not offered the Spirit of God.
Under the new covenant, though, the promise is that the Spirit of God would be poured out on old men, young men, maidservants, everyone. "The promise is to you and to your children and to all who are afar off," Peter told the Jews (Acts 2).
So here's what Jesus' death did for us. We were people who didn't have the power to stop sinning. (Paul talks about this in Romans 3 and Romans 7.) The Law didn't really help us because we didn't have the power to obey it. So God came up with a different answer. Paul writes about it like this:
So Jesus' death broke the power of sin and gave us access to the Spirit of God. Thus, we are far better off than those under the Law who had to struggle and find out that their very nature is to disobey the Law.
However, to get back to your question, those under the old covenant could be forgiven as soon as they repented. They didn't have to wait for Jesus to die to be forgiven. Mercy is in the very nature of God; he's always been forgiving to the repentant even with only a broken heart and contrite spirit as atonement.