Banks moving on from OFX to Open Banking? FDX?

[redstreet]

Though not directly related to the code in this repo, this is the best place I could think of for asking this question: banks and institutions seem to be moving away from ofx and direct connect, into "Open Banking", in the US, the Financial Data Exchange standards.

Does anyone know how to access these via high level open source tools similar to ofxtools? Ideally, something like a drop in replacement for ofxtools? Here is an example list of US banks that have moved to Open Banking.

[PrplHaz4]

Good to see they are actually using it - last I checked FDX didn't have much traction. It does sound like it's established as the successor to OFX. Unless the free ("Observer") tier actually enables you to get data, it doesn't sound very favorable to the open source community...

Sorry, no actual answer to your question but will be following along...

[csingley]

When I got in contact with them, they pretty much told me to piss up a rope. The world is pretty far removed from the dot-com ideals of bringing open source, Jim Clark, IETF etc. to revolutionize hidebound industries like finance & healthcare. Now it's just "father knows best" money center banks, financial-processing rentiers, and VC-funded fintech douchebags driving the bus.

The only reason FDX is even a thing is as an industry response to show "we're doing something" and head off the threat of legislation/regulation. To quote that wikii:

Financial Data Exchange (FDX) organization was formed in 2018 as a non-profit consortium that started to sign on members from the fintech and banking communities in late 2018.[30] The group consists of the largest financial institutions as well as aggregators and fintechs. Founders sought to create a common technical standard to enable secure, consumer-permissioned data sharing for financial data, effectively sounding the first signal that the US was going to pursue something like Open Banking. FDX aims to establish a common, shared standard for Open Banking through a market-driven approach, the idea being to engage with the different market players and use a consortium approach similar to Bluetooth.

Emphasis on "market-driven".

What they're getting to is public-key cryptographic methods to ensure that all access is authorized by a root of trust depending from a consortium member. It isn't going to be "I can log in with my password", it's gonna be "My service provider can do this for me (and charge me a fee) if they're clicked up with a consortium member". And the consortium ain't gonna be open access, for sure. The way they were talking to me was very much pay-to-play.... coz there's plenty of VCs who will pay.

A pox on all their houses.

Really though, it doesn't matter all that much to me. Banks will probably offer OFX-format data downloads (that I can parse with ofxtools) on their websites for a long time. I don't really use the OFXClient anyway

[redstreet]

With the little bit of reading I did, I got the same impression -- that FDX is only for commercial interests, and not for FOSS folks. I was trying not to be cynical, but here we are, sigh.

I use OFXClient a lot in my workflow (see this article), and that part of my workflow is down to literally under 8 seconds to download ~25 accounts. I'm not looking forward to logging in to websites to download ofxs. Assuming ofxs are still provided (Schwab quit providing ofx at all).

Bummer, but thanks for the entertaining post, @csingley!

[djedi]

@redstreet I've played around with Playwright and was able to get OFX files off of bank/ccs that no longer support ofx direct connect. That might be an option for you for automation.

[redstreet]

Cool, thank you for the suggestion, I'll check it out, @djedi. I'm annoyed enough at going from flawless, single-command 1-second downloads and imports to a hundreds of clicks and windows.

Any recommendations on articles/videos on the fastest way to get myself setup to login to a bank account and download?

Also, how do folks store passwords securely when using these types of browser navigation? How do folks handle 2FA?

[amandabot]

@redstreet I used to use Protractor for accounts where I couldn't use direct OFX, and I'm now starting to use Playwright with success. The documentation is fairly easy to follow, but here's a couple tips related to automating logins and your other questions:

• Some banks/financial institutions can detect browsers running commands via a webdriver, so you need to start the instance of the browser yourself and connect to it with Playwright. Otherwise, the site might not authenticate.
• For passwords, I keep them in a config file which I inject into my script and reference them as code (bank.password)
• For MFA, you'll need to still interact with the browser. It's possible to write a few lines to detect the input of the MFA textbox, poll it for completion, and click the submit the button programmatically. It's slower overall compared to a single command, but still pretty minimal interaction.

[redstreet]

That's great, thanks for the second recommendation, @amandabot. Forgive my ignorance here of web technologies, but:

• Playwright is an end to end tester for web apps. I assume this means that it provides automation of an ofx download would masquerade as a test, which presumably can simulate clicks on a webpage and such? Given this, is there a guide I can follow that will most closely resemble how I would setup an ofx download script (Playwright and web technologies are a rabbit hole that I I'm trying to avoid getting lost in)

• I run WSL and Windows, and am wondering if running Playwright is best done via WSL, or by avoiding WSL. There seems to be a site with instructions for headed mode for WSL, but it looks complex. Any experience with this? Will I need headed mode? I installed Playwright in WSL, and the basic tests pass in headless mode

Thanks in advance!

[djedi]

@redstreet Here is a script I run regularly that logs in to AMEX and downloads a transaction file, then parses the file to import transactions into YNAB. I got tired of Yodlee's stupic sync issues all the time. This has been more reliable for me. https://gist.github.com/djedi/46a9ede2002fce07dea3cfe5d9f93a83

@amandabot I've read some tutorials on connecting an open browser to playwright, but so far I haven't had success. Do you have any tips?

[amandabot]

@redstreet

Yeah, Playwright is intended for end-to-end testing, but it can be leveraged for automation outside of the testing framework.

Check out the gist below for a mini-tutorial on Playwright setup. It does a Google search and clicks into the Playwright docs. That's more or less the extent of what you'll need to do to get OFX docs from a site. As long as you know some basic HTML selector syntax, you should be good. The Playwright docs are pretty good as well.

I run it on plain Windows. You could probably do WSL as well.

I would stick with headed mode since you mentioned MFA; you'll need to see the screen to know when your input is required. If your site doesn't have MFA, headless is fine too, but personally, I like to watch the automation zip around the site at superhuman speeds 😅. It's also nice in case something breaks or changes that I can be aware of what the issue is.

@djedi
Here's a bare-bones gist in TypeScript with the setup I use: https://gist.github.com/amandabot/ff1dbeedf877f12d1f99c3ba7cac5664
The main difference between mine and yours is mine depends on the browser being started manually. You could also attach to the one created by Playwright (browser.launch()), but that comes with the webdriver flag set to true and is subject to the limitations I mentioned earlier. This is the magic line which attaches to the browser instance: const browser = await chromium.connectOverCDP('http://localhost:9222');

[redstreet]

These examples and pointers are awesome @amandabot and @djedi. Writing my automation now. Thanks a ton!