This is not a wide-spread problem, and may be isolated to 2.6.9. So, it makes sense that others who run O_DIRECT with EXT3 do not see the issue. I will use this post for future updates.
Ok here is the research that I did, and found the cause of my O_DIRECT problem
|RHEL Bug ID||Description|
|161985||O_DIRECT on RHEL v4 may not return correct number of bytes when concurrent I/O|
|178084||Last AIO read of a file opened with O_DIRECT returns wrong length|
|178720||O_DIRECT bug when reading last block of sparse file|
|191736||CVE-2004-2660 O_DIRECT write sometimes leaks memory|
So, I can't use O_DIRECT