Communion Of Dreams

Perhaps it is the drugs, or the economic degree talking, but a curious thought occurred in consideration of a stock-market piece in the Atlantic: are we seeing the first real indication of some kind of self-aware Artificial Intelligence in the millisecond-to-millisecond world of automated stock trading?

OK, probably not. The cynic in me says that someone has just figured out a way to game the system to their advantage, throwing out a lot of confusing chaff to slow down the computer systems of other traders. Here’s an introductory paragraph to explain what this is all about:

It’s thanks to Nanex, the data services firm, that we know what their handiwork looks like at all. In the aftermath of the May 6 “flash crash,” which saw the Dow plunge nearly 1,000 points in just a few minutes, the company spent weeks digging into their market recordings, replaying the day’s trades and trying to understand what happened. Most stock charts show, at best, detail down to the one-minute scale, but Nanex’s data shows much finer slices of time. The company’s software engineer Jeffrey Donovan stared and stared at the data. He began to think that he could see odd patterns emerge from the numbers. He had a hunch that if he plotted the action around a stock sequentially at the millisecond range, he’d find something. When he tried it, he was blown away by the pattern. He called it “The Knife.” This is what he saw:

Followed by a graph showing a clear pattern. Then here’s the bit that tells how this could be an advantage to another trader:

Donovan thinks that the odd algorithms are just a way of introducing noise into the works. Other firms have to deal with that noise, but the originating entity can easily filter it out because they know what they did. Perhaps that gives them an advantage of some milliseconds. In the highly competitive and fast HFT world, where even one’s physical proximity to a stock exchange matters, market players could be looking for any advantage.

But think about this. What a delightful SF explanation it would be to have one of these powerful automated systems (they have to be some of the most powerful and complex computer/software systems on the planet) starting to “wake up” and experiment in manipulating its environment: the world of stock trading. Here’s a bit from the MeFi thread where I came across this:

All of the serious HFT firms these days use “natural language processing”, which means using artificial intelligence to extract profitable information from news streams. People think of this as just headlines but really it’s anything that might contain useful information – these computers have all of the cable news channels supplied to them digitally and use everything they can scrape. Some of the firms even use facial recognition software to determine whether the speakers believe what they’re saying. My friends joke about how Cramer is a goldmine for their algorithms but that the profitable trades rarely match up with his advice.

One of the facts about ‘hard’ AI, as is required for profitable NLP, is that the coders who developed it don’t even understand completely how it works. If they did, it would just be a regular program. What’s even stranger is that they can’t use regular tools, like a debugger, to observe the algorithms’ behavior, because it interferes with the processing and causes different trades to be emitted. In a very real sense, they can’t explain why their robots send the orders they do. They can tell you what data they “trained” it with, and what sorts of data they “feed” it, but they’re inherently unpredictable.

As a result, a lot of programmers at HFT firms spend most of their time trying to keep the software from running away. They create elaborate safeguard systems to form a walled garden around the traders but, exactly like a human trader, the programs know that they make money by being novel, doing things that other traders haven’t thought of. These gatekeeper programs are therefore under constant, hectic development as new algorithms are rolled out. The development pace necessitates that they implement only the most important safeguards, which means that certain types of algorithmic behavior can easily pass through. As has been pointed out by others, these were “quotes” not “trades”, and they were far away from the inside price – therefore not something the risk management software would be necessarily be looking for.

Even better, perhaps such an AI entity was aware enough to realize its position in the larger world stage, and also realize that one way to bring down humanity would be through the kind of economic crash we recently just avoided – something even worse than the Great Depression. How to do it? Well . . .

But already since the May event, Nanex’s monitoring turned up another potentially disastrous situation. On July 16 in a quiet hour before the market opened, suddenly they saw a huge spike in bandwidth. When they looked at the data, they found that 84,000 quotes for each of 300 stocks had been made in under 20 seconds.

“This all happened pre-market when volume is low, but if this kind of burst had come in at a time when we were getting hit hardest, I guarantee it would have caused delays in the [central quotation system],” Donovan said. That, in turn, could have become one of those dominoes that always seem to present themselves whenever there is a catastrophic failure of a complex system.

Think about it.

Curious, indeed.

Jim Downey