In addition to being an attractive target for hackers, storing these SSO credentials presents an interesting trust problem from the perspective of more privacy-conscious users.
We are just committing to the user that we are not storing their information. We are not providing strong guarantees, cryptographic or otherwise, that we will not use this information for our own gain.
Especially for more powerful SSO integrations, such as bank accounts, it might be hard to convince folks to trust us.