|Feb 18, 2005 03:52 PM
|Aug 3, 2006 02:12 PM
|Aug 3, 2006 04:12 PM
jazzfusionb has contributed to 14 posts out of 21154 total posts
(0.07%) in 6,943 days (0.00 posts per day).
20 Most recent posts:
My opinion is that you should always use a consistent date format. I think that YYYY is the only way to go, so I would like to see you consistently use it everywhere; just make sure that your documentation is up to par.
This is slightly off topic, but I argue that the only intelligent way to write dates is like YYYY/MM/DD:
--its logically consistent, unlike the usual defective american way of MM/DD/YY(YY)
--it correctly lists the date fields top down, from most significant to least, unlike the defective european standard of DD/MM/YYYY.
But maybe its too late to change all of your formats this radically now.
I just spoke to Ed Lenz at your firm, and he said that I am using the DTN feed.
I am also following up with sending him an email describing some other issues that I have seen today.
>Is this your own software that you are seeing this in?
Well, I use my own software to log the exact messages that DTN sends me. So the new YYYY date format that I saw in some of today's messages is really what DTN is now sending down the wire.
Another bizarre thing is that the Last Trade Date field is only supposed to be filled in for OTC or BB stocks, but I was seeing it get filled in for decidedly different stocks (e.g. mainstrem NYSE or Nasdaq stocks).
>Please let me know if it is developer feed or DTN IQ.
How do I know which feed I am using? I am a developer, but I am also sybscribing to 1300 stock symbols, which someone yesterday told me means that I am using DTN's feed and not IQFeed--is that correct?
The IQ online documentation for update (Q) messages
clearly states that the date format for both the
Last Trade Date
fields is of the form
But I just happened to notice when pouring thru some log files today that, at least for the Last Trade Date field, they seem to have updated the format to
(I have no idea about the Expiration date field, since I do not trade options or futures at the moment.)
Anyone else notice this, or is this just a fluke?
If they have, in fact noticed this change, then they need to update their documentation as it is now out of synch.
Mind you, I am all in favor of this change, as anyone who still uses 2 digit years after the y2k fiasco should be most severely punished! I just need to know that I can count on it...
For purposes of this discussion, assume that just US equities are under consideration.
I am assuming the information should be embedded in IQFeed's update messages somewhere. The documentation for these messages is here:
--the message Type is Q (update)
--the "Time" field ends with 't' or 'T' (i.e. a trade occurred)
then where in the message is the exchange info stored?
Note that the "Exchange ID" codes seem to be worthless: for US Equities, the main values that should be returned are either
But the above exchange code list has no values for the various ECNs, and the ECNs in aggregate are a big source of liquidity!
Similarly, the "Market Center" field seems like it might store this info, but its possible values
again say nothing about ECNs, mainly just regional US exchanges.
Am I missing something, or is this a huge limitation with IQFeed (no ECN info)?
>I'm looking for developers that:
>- use DTN/IQFeed as a data feed
>- use IB as a broker
>- be developing applications for automated trading using APIs from the above
>- develpop in Java (well mostly)
I hit all of the above.
>I was wondering if you guys would be interested in a separate discussion forum. We could use either IB or DTNs forums or start our own
>elsewhere. Any suggestions are welcome.
>The purpose would be to assist each other with development problems, etc as IB and IQFeed (in particular) are not always helpful.
>Often there's workrounds to bugs that we can all benefit from.
I would love to speak to you guys in person over the phone--ho can I reach you?
I have already contacted Ned Solot, and he is a nice guy. He develops in C++ by the way (but I think that Java is a vastly better choice!).
>My partner and I have been developing a fully automated trading platform for the past 2 years but mostly the past 14 months.
>We went live 2 weeks ago.
I have been working solo (altho full time) for the last year on my trading system. I started trading seriously with real money from October thru February, but my system was NOT fully automated. It is now, since I stopped trading in March and spent full time working on my technology. I can't wait to start testing my full auto system with real orders, perhaps sometime this coming week.
I learned a lot doing the orders and baby sitting positions manually, but I should have cut this phase short a few months ago, because I have now run out of a time limit that I agreed on with my wife and I have to at least look for a day job. If I am lucky, I will get a short term Java consulting gig, but those are now really hard to find in New York City where I live.
So how is your full auto system doing? Making money? Too soon to tell?
>Our system is totally Java based and we do all development under Linux.
I am totally Java based, but develop under Windows.
In the future, as Microsoft turns more and more into The Beast, I may very well move onto Linux, Solaris, or OSX. What Linux distro do you like? I know very little here, but want something that is well supported and actively developed and reasonable for a non-sys admin like me to maintain. I have been told to experiment with Knoppix, and then probably use something like SuSe professional.
>As we are in Australia
Do you have VOIP so that you could call me here in New York City cheaply?
>and only trade on NASDAQ,
Why is that?
>With a UPS and (coming soon) generator, we should be able to run smoothly most nights.
I have just a UPS. I figure that if NYC's power goes out like it did in the summer of 2003, then all is lost anyways.
>We are still looking for another data feed to compliment IQFeed but the only one that seems okay is realtick.
>We are also looking at using a co-lo in the US.
If you are doing enough volume, then you should contact
Vice President of Sales
Nexa Technologies, Inc.
They have a lot of stuff that they sell to hedge funds, ranging from real time and historical data (they just bought Tick Data, who I use for my model development--expensive, but the best) to FIX engines that route directly to the exchange to co-lo services (they claim latencies in the few 10s of milliseconds).
David & Ned,
I too am seeing corrupted data messages (e.g. letters in the last field, like Ned originally reports, among many other errors).
IQ, if you are reading this thread, please correct this!
I disagree with you, David, however, that going to UDP is a good idea. It is absolutely essential that I get messages in time sequential order as they occurred in the market, and so if you do no use TCP then somone (either IQ in their client, or even worse, you in your custom code) would need to write all that code which is a pain and bug prone. You may as well stick with TCP which does that for you.
But what I do not understand is the apparent current behavior with IQ's client code.
Why, if the iq client is having probelms keeping up with the server's data, does it send me corrupted messages? My code cannot proceed if it fails to parse it, so I have to drop the message anyways, so why does their client not drop it in the first place?
As near as I can tell, IQ currently does an extremely bad job of doing any data integrity and syntax checks on the messages that they send out. They need too. And they should drop messages that are corrupt due to being partially filled.
Here is something more about IB's smart routing that they just sent me in an email today:
Smart Routing Gets Smarter
We are pleased to announce yet another brokerage industry first, IB SmartRouting Auto-RecoverySM which will automatically re-route your order to a new market center should a market center with your order experience a technology malfunction. Unlike other brokers that must wait for an exchange to recover in order to ascertain whether or not your order has been executed, IB will automatically re-submit your order to a functioning exchange and undertake the risk of a double fill on the malfunctioning exchange. Rather than having to experience the uncertainty of a pending order in limbo, you will now know the exact status of your order. Auto-Recovery is yet another reason to choose IB and SmartRouting.
>IB does not report every trade and that is intentional on their part.
I just found a good discussion of this here:
>IB documents a limit of 50 messages per second, but it seems that sometimes I get more.
Let me know if you know of a solid link discussing this; someone just posted on this topic today:
>I do see some errors when debug logging is set over 0, but all seem to work fine when it is set to 0, however, the message parser provided in the watchlist C++ sample caused me all sorts of problems so I rewrote it.
I am not sure what you are refering to about debug = 0 logging in the context of IQFeed. IB does have such a capability with their api, see for example http://chuckcaplan.com/twsapi/index.php/void%20setServerLogLevel%28%29, but that is not what we were talking about.
The errors in my log file are generated by my own program when various consistency tests that I have devised fail. IQ themselves almost never reports an error. (Their server software should be written better so that problems are caught before ever being sent out.)
>>But it does not solve the problem of missing ECN liquidity from IQFeed
>I'm confused. ... Which ECN's are missing?
All of them are missing--look at Jay's first reply to my initial post.
>What I would really like to see IQ implement is depth of mkt info similar to the book viewer avaliable at http://www.archipelago.com/
Yup, that would be nice too.
>That makes 2 of us, and I think there's another, Skunk, and maybe more. I'm working on an ATS using both APIs under VC++.
You live in the NYC area? Ever wanna talk shop?
>With IQ, enable regional quotes. Check the API docs on how to do this.
Thanks--I had actually not encountered that or realized the significance of it when reading their docs myself.
They describe it at
(for the installation on my computer, at least).
But it does not solve the problem of missing ECN liquidity from IQFeed, it just allows you to precisely see the breakdown of liquidity from exchanges that they do support.
>IB does not report every trade and that is intentional on their part.
Oh really? That is news to me. Does IB document that anywhere?
>IQ does report every trade (or at least tries to) and gives you the market center where the trade executed.
I have set up a lot of consistency checks (e.g. compare the total volume of trades that IQFeed reports versus a sum of all the individual ones that they have sent me), and every day I have megabytes of log file errors generated by problems that I see with IQFeed.
I am not sure what part of this is because the exchanges themselves are really bad in the data that they provide, or whether IQFeed is messing up. I suspect some of both, but am sure that IQFeed is at least partly responsible as I notice lots of other errors (e.g. Q messages that do not follow their own internal format and so are unparseable) which indicate that they do not have high quality software.
>One question for you. Are you using the API from IB and/or IQ? or just the TWS from IB and DTNIQ software from IQ?
I am using the Java APIs for both, since I am writing a fully automatic trading system in Java.
With IB's Java api, this is NOT a socket tap into their remote servers. Instead, you need to be running a copy of their client software, TWS (Trader WorkStation), and you make a socket connection to that, which in turn internally communicates with IB's remote servers. I am in the middle of writing the software to auto execute my model's orders; it is more of a pain than it should be, because IB could have made their api a lot better.
With IQ/DTN's java api, you are essentially doing the same thing: opening a socket connection to some local software (in this case a windoze dll--bad choice on DTNs part, since now you are stuck with using windoze as your OS, wish they would have written the whole thing in Java, but thats another topic...).
>I'm not convinced there is a "right" vs "wrong" way to report this.
I cannot imagine any trader who would NOT want to see all the market liquidity that is available on the market.
In principle, I understand that DTN currently finds it too expensive to include the ECNs (but why does IB not find it so expensive?), but I cannot imagine it ever being good for you as a trader to not have the info available.
The only change that you might want from what IB does is to have even more information (e.g. a breakdown, by exchange, of the liquidity at the BBO).
If you still disagree, please post a counter example and educate me.
In saying this, I am assuming that your broker offers you access to all the exchanges; if your broker, say limits you to NYSE or NASDAQ then perhaps you might only want to see the quote size available there. Actually, even for this case, you probably still want to see the total (since that can strongly hint at the real buying or selling pressure) as well as the size at the exchanges at which you can execute (since that is all the size that you will be able to get off). So this case argues strongly for having even more information than what IB provides.
>Using your example, you might want to ask IB if you placed an order for 1500 shares using SMART routing,
>if they would split your order and sent 1000 shares to ARCA and 500 shares to ISLAND.
That is my understanding of what they do with SMART routing. The whole point of SMART routing at IB is that you do not have to send in multiple orders, they will do all that for you behind the scenes using logic on their servers.
I asked the people at IB why you would ever not want to use SMART order routing, since it is so much better in general. They said that the only people for whom it does not make sense are the people who are doing exchange arbitrage, so they need precise control over what exchange gets their order.
>I suspect most brokerages would send all 1500 shares to a single exchange for execution.
Then they don't have SMART routing. (I have no idea if the concept is unique to IB, but I would be very surprised if so, since it is such an obvious thing that you would want to have. Then again most brokerages have really crappy technology, so maybe they have been cheap and just not invested in this capability.)
>My hunch is IB will tell say you would have to manually send 2 orders.
They have never done that to me.
Ignoring fast markets, where there are complex timing issues (i.e. what you see on your screen is a delayed and inaccurate representation of what might actually be available), I have ALWAYS been able to put in, say, a single limit order at the best bid and pick up all the shares that were offered. Occaisionally, the filling of my shares comes in stages; I have never thought of this before, but maybe it is IB getting reports back from different exchanges at different times.
Who do you use as your broker? I recommend IB actually. They aren't perfect, but are decent.
>We provide the BBO from the exchange, which is what the exchange requires us to show.
So you just show the NYSE quotes for listed stocks, and NASDAQ quotes for OTC stocks?
>What you may be seeing from IB is an aggregate of ECN quotes as well.
I am sure that it is.
>We don't carry the ECN feeds on IQFeed.
Are there any plans by you guys to do so in the future?
There is now quite a bit of liquidity information that you are missing if you do not carry them, right?
Is the issue here simply that the ECNs want to charge you too much money, or is it just the current state of your technology?
I use IQFeed for my real time data, and Interactive Brokers as my broker.
Today, while trading a stock, I noticed that there were consistent large discrepancies between the quote data provided by IQFeed and IB throughout the day.
The discrepancies were not so much in the bid & offer prices but in their sizes: IQFeed was regularly displaying sizes 1/2 or 1/3 of what IB was reporting.
I called both IQFeed's and IB's technical support today to try and track down this discrepancy.
IB told me that they form the sizes in the intelligent way that anyone doing trading would wish, namely, they aggregate all the sources (if there are multiple ones) at the best bid and offer to show the total liquidity available at those prices.
IQFeed, in contrast, told me that they do NOT aggregate: they just report the size of just one of the sources of the best quote. For example, if the best bid price is $10 and both ARCA and ISLAND have shares at that price, say 500 and 1000 shares respectively, IQFeed will only report a bid size which comes from one of them, say the largest (1000), and does not sum them up (1500) like IB does.
Would someone from IQFeed please respond and confirm whether or not what I was told by your technical support guy is correct?
And if it is correct, why do you do this? This makes your data greatly inferior for trading. Is there any possible way that you could change your system to report accurate information for traders?
And if it is not correct (i.e. you do aggregate sources), then we need to start an investigation to find out why your data was so far off the mark.