The subject was discussed a long time ago, but maybe someone reads my post and provide a solution.
The problem I’m facing is with regard to Oracle database(10.2.0.4) installed entirely on a SSD disk, on CentOS 6.
From time to time, every one month or so, the SSD is put into offline mode, no I/O can be made and Oracle hangs. Usually this is resolved with a reboot, but had also some data corruption in one case.

Here is a small fragment of the OS logs:

Feb 16 05:11:21 oraclexx kernel: ata6.00: exception Emask 0×0 SAct 0xffff
SErr 0×0 action 0×6 frozen
Feb 16 05:11:21 oraclexx kernel: ata6.00: failed command: WRITE FPDMA QUEUED
Feb 16 05:11:21 oraclexx kernel: ata6.00: cmd
61/08:00:38:15:43/00:00:03:00:00/40 tag 0 ncq 4096 out
Feb 16 05:11:21 oraclexx kernel: res
40/00:00:00:00:00/00:01:00:00:00/a0 Emask 0×4 (timeout)
Feb 16 05:11:21 oraclexx kernel: ata6.00: status: { DRDY }

Feb 16 05:11:32 oraclexx kernel: ata6: hard resetting link
Feb 16 05:11:32 oraclexx kernel: ata6: SATA link down (SStatus 0 SControl
310)
Feb 16 05:11:32 oraclexx kernel: ata6.00: disabled
Feb 16 05:11:32 oraclexx kernel: ata6.00: device reported invalid CHS
sector 0

Feb 16 05:11:32 oraclexx kernel: end_request: I/O error, dev sdb, sector
54727992
Feb 16 05:11:32 oraclexx kernel: Buffer I/O error on device sdb1, logical
block 6840743
Feb 16 05:11:32 oraclexx kernel: lost page write due to I/O error on sdb1
Feb 16 05:11:32 oraclexx kernel: sd 6:0:0:0: [sdb] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE

Feb 16 05:11:32 oraclexx kernel: end_request: I/O error, dev sdb, sector
34656896
Feb 16 05:11:32 oraclexx kernel: sd 6:0:0:0: rejecting I/O to offline device
Feb 16 05:11:32 oraclexx kernel: end_request: I/O error, dev sdb, sector 0
Feb 16 05:11:32 oraclexx kernel: Buffer I/O error on device sdb1, logical
block 4331856
Feb 16 05:11:32 oraclexx kernel: lost page write due to I/O error on sdb1

Feb 16 05:11:32 oraclexx kernel: end_request: I/O error, dev sdb, sector
55003384
Feb 16 05:11:32 oraclexx kernel: ata6: EH complete
Feb 16 05:11:32 oraclexx kernel: ata6.00: detaching (SCSI 6:0:0:0)
Feb 16 05:11:32 oraclexx kernel: Aborting journal on device sdb1-8.
Feb 16 05:11:32 oraclexx kernel: EXT4-fs error (device sdb1):
ext4_journal_start_sb:
Feb 16 05:11:32 oraclexx kernel: JBD2: I/O error detected when updating
journal superblock for sdb1-8.
Feb 16 05:11:32 oraclexx kernel: EXT4-fs error (device sdb1) in
ext4_reserve_inode_write: Journal has aborted

Best regards and hope someone answers soon :)

I am glad to see that this post has blossomed quite a bit since my post. One quick side note – the disk in your Laptop is battery backed (by the laptop battery :) ).

I am glad that you found the setting to disable the write cache on the disk as I think it helps explain more clearly that when you are testing memory access on any device there is no easy way to predict what the performance will be without testing.

A quick note on Alex’s comment on RamSan performance, our write time is the sum of RamSan access time + HBA driver/Asic access Latency + OS/application access latency. For a few years back on an HPUX platform I would guess that a 2 Gbps HBA was used in the server. I end up doing a lot of FC performance testing and there have been dramatic advances in the access latency of 4 Gbps HBAs vs 2 Gbps HBAs. The 2 Gbps HBAs tended to add 0.2-0.3 ms to an access and the 4 Gpbs HBAs only add 0.03 to 0.04 ms to an access (assuming interrupt Coalescing is disabled). Our access time was 0.02 ms with our 2 Gbps controllers and 0.015 with our 4 Gbps controllers + transfer time (this varies by the block access size and link bandwidth for 13k and a 2 Gbps link it would be 13k/~200,000KB/s = 0.065 ms). Adding the OS and oracle overhead and I would expect the 0.5 ms time. There have been a lot of enhancements since then (largely concerning equipment that we do not control) that has allowed much lower write times.

For the insert and commit loop the LGWR was definitely writing less than 1k per write.

Here is the some more data from the statspack report, I would be happy to send the whole report if you want it (our email format is firstname.lastinitial@ramsan.com). We actually had Linux multipathing running and we were using linux mirroring on the RamSan volume, I may see what it looks like if we pull these component off and just duplex the logs.

Also if you are going to Oracle world the week after next I would be happy to chat there.

Thanks,
Jamon Bowen


WORKLOAD REPOSITORY report for

Statistic                                   Total     per Second   per Trans
-------------------------------- ---------------- -------------- -----------
physical write bytes                  956,456,960    1,593,699.2       157.8
physical write total IO requests        4,021,945        6,701.6         0.7
physical write total bytes          4,987,928,576    8,311,150.4       822.7
physical write total multi block        3,423,986        5,705.2         0.6
physical writes                           116,755          194.5         0.0
physical writes direct                        415            0.7         0.0
physical writes direct (lob)                    0            0.0         0.0
physical writes direct temporary                0            0.0         0.0
physical writes from cache                116,340          193.9         0.0
physical writes non checkpoint            114,196          190.3         0.0
pinned buffers inspected                        0            0.0         0.0
prefetch warmup blocks aged out                 0            0.0         0.0
prefetched blocks aged out befor                0            0.0         0.0
process last non-idle time                      0            0.0         0.0
recursive calls                        18,194,819       30,317.2         3.0
recursive cpu usage                        43,813           73.0         0.0
redo blocks written                     7,726,344       12,874.0         1.3
redo buffer allocation retries                146            0.2         0.0
redo entries                            8,606,700       14,340.9         1.4
redo log space requests                       323            0.5         0.0
redo log space wait time                    7,366           12.3         0.0
redo ordering marks                        38,362           63.9         0.0
redo size                           3,102,037,856    5,168,779.5       511.7
redo synch time                                 0            0.0         0.0
redo synch writes                             166            0.3         0.0
redo wastage                          727,552,932    1,212,287.2       120.0
redo write time                            46,272           77.1         0.0
redo writer latching time                     370            0.6         0.0
redo writes                             4,012,526        6,685.9         0.7


I previously thought that the LGWR process would *always* guarantee my redo log changes were written directly to disk once I had committed. In other words, once LGWR had made its I/O request and returned to my session, my changes were safe.

However, after investigation, apparently LGWR writes to disk (under Linux) using an I/O feature called DIRECT IO. After checking with Oracle and Redhat, DIRECT IO only bypasses the Redhat O/S I/O caching mechanism. It does not instruct the hardware to bypass any subsequent I/O controller caching or Physical disk caching.

Therefore to be really sure that your changes are actually written to disk, you need to investigate the caching aspect of each hardware component between your server and physical storage!

Stewart

From that project, we could achieve 0.5 ms write for about 13K write for LGWR (8K block read if interesting was about the same 0.4-0.5 ms). Response time wasn’t impacted by number of IOs but had some correlation with IO size. It was HP-UX PA-RISC platform and we used Oracle 9i with Veritas storage stack (and it was couple years ago as well). We used host based mirroring across two RamSan boxes and without it – same response time.

To be fair, we allocated redo logs from 4 way striped volume from EMC and we had about 0.7 ms write time but (and here goes a big BUT) we observed some instability and as far as I could recall it was due to cache saturation in some circumstances.

In more recent experience, during one RAC load testing project, I used RAM disk mounted by RAC nodes via NFS and few kilobytes writes and reads were stable in the range of 0.3 ms. I was quite pleased with such NFS performance. To be fair, it wasn’t a regular 1 GB Ethernet – proprietary high-speed “fabric” as vendor calls it.

I couldn’t get 0.1 ms for LGWR write but I think it’s because my writes were relatively large. In Jamon’s somewhat artificial case, LGWR write size was probably in the range of 512 bytes to 1K. There is no statspack report time-frame but I assume that one commit caused about half to one LGWR write since it’s performance was dazzling fast. If Jamon is reading this comment, it would be nice of him to provide LGWR write size from that case.

Val, I was assuming that the dump would be a screamingly fast sequential dump – so I was expecting the cache to be irrelevant after the first couple of seconds even if it were on. But a small lag for compression would explain why my hibernate results were consistent with your log writer results.

Given the time I spend getting on and off trains, planes and whatever the third one was, I think I’ll have the cache ON most of the time, and remember to switch it off for critical collision tests.

Jonathan, I was surprised initially since an hibernate dump of the system RAM should provide a continuous stream of sequential bytes, and sector interleave should cater for the small time it should take Windows to initiate the next sequential write. But I have researched a bit and discovered that XP makes compressed writes, so probably the time it takes to compress the next write is enough to make the disk spin over the next interleaved sector.

It matches the simple benchmarking I’ve just run. On my laptop, with WCE (write cache enabled) I get 20MB/s throughput for 64K OS-cache-bypass writes; without WCE I get only 3.6 MB/s.

I wonder how Linux running on a laptop handles WCE.

That’s a brave decision, I hope your users will have mercy on you after they have experience potentially severe performance problems ;)