Having done a little research and talked to my wife, the former banker, the picture is just a bit more complex than that.
The original version of the Community Reinvestment Act had very few teeth. It was revised in the mid-1990s and teeth were put into it. If your bank wanted to expand or merge with another bank, it had to have a good CRA record. One of Gretchen's good friends was the compliance officer for the bank that she worked for, so she heard a good bit about the difficulty of complying and the amount of paperwork required.
The problem was that the banks were trying really hard to stay in compliance with the CRA, but finding borrowers who actually qualified for the kind of mortgage loans that Fannie Mae and Freddie Mac were allowed to buy was difficult and sometimes impossible. This meant that if the banks made mortgage loans to less-creditworthy customers in order to stay in compliance with the CRA, they were stuck with them.
The banks, quite understandably, didn't want all of this potentially bad paper on their books. Thus Congress loosened up the regulations on Fannie Mae and Freddie Mac and told them they could buy this sort of crappy loan, because Congress wanted the banks to make the crappy loans to people in the CRA areas.
What they didn't say to Fannie and Freddie was that the only crappy loans you can buy are ones that are made for the sake of CRA compliance. That's not too much of a surprise, I suppose, since doing anything else would have appeared to be blatantly discriminatory.
Well, once Fannie and Freddie stood ready to buy crappy mortgage loans, there was a sudden boom of people who were ready to originate crappy mortgage loans, because they had a place to get rid of them to. They could get the profits from originating the loan without having to carry the long-term risk.
As usual, you get the behavior that you create the incentives for.
So the story starts with the CRA, but doesn't end there.