The risky lending business has been booming — but now its problems are becoming increasingly visible on Wall Street and beyond.