Oracle Scratchpad

February 22, 2018

Huge Pages

Filed under: Oracle,RAC,Troubleshooting,Tuning — Jonathan Lewis @ 9:03 am GMT Feb 22,2018

A useful quick summary from Neil Chandler replying to a thread on Oracle-L:

Topic: RAC install on Linux

You should always be using Hugepages.

They give a minor performance improvement and a significant memory saving in terms of the amount of memory needed to handle the pages – less Transaction Lookaside Buffers, which also means less TLB misses (which are expensive).

You are handling the memory chopped up into 2MB pieces instead of 4K. But you also have a single shared memory TLB for Hugepages.

The kernel has less work to do, bookkeeping fewer pointers in the TLB.

You also have contiguous memory allocation and it can’t be swapped.

If you are having problems with Hugepages, you have probably overallocated them (I’ve seen this several times at clients so it’s not uncommon). Hugepages can *only* be used for your SGA’s. All of your SGA’s should fit into the Hugepages and that should generally be no more than about 60% of the total server memory (but there are exceptions), leaving plenty of “normal” memory (small pages) for PGA , O/S and other stuff like monitoring agendas.

As an added bonus, AMM can’t use Hugepages, so your are forced to use ASMM. AMM doesn’t work well and has been kind-of deprecated by oracle anyway – dbca won’t let you setup AMM if the server has more than 4GB of memory.

There are a few follow-up emails after Neil’s; particularly helpful are two from Stefan Koehler, here and here.

For reference here’s a link to the Linux v6 guide to setting up huge pages for Oracle (if I remember to keep this up to date I’ll add further links for newer versions of Linux – unless someone conveniently supplies them in the comments). And a link to Tim Hall’s pages on the subject that add some further bits and pieces that are likely to be relevant in some environments.

Update Nov 2020

I’ve been talking to a client recently about huge pages on a Linux based system, and thought I’d add a couple of notes here catching up a couple of details that I’d missed in the past.

Starting from 11.2.0.3, Oracle will start up and use a mixture of huge/large pages and standard pages if you haven’t allocated enough huge pages. Prior to 11.2.0.3 the instance would refuse to start so it was easy to discover that you need to do some reconfiguring.

Since you don’t get the automatic warning that you’ve got it wrong, the alert log now tells you about your memory configuration (apparently only if it’s wrong) with an output like the following (which came from a 19c instance with a 9G SGA max size).

****************************************************
 Sys-V shared memory will be used for creating SGA
 ****************************************************
2020-11-10T12:56:14.183333+00:00
**********************************************************************
2020-11-10T12:56:14.183380+00:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)

2020-11-10T12:56:14.183464+00:00
 Per process system memlock (soft) limit = 128G
2020-11-10T12:56:14.183568+00:00
 Expected per process system memlock (soft) limit to lock
 instance MAX SHARED GLOBAL AREA (SGA) into memory: 9218M
2020-11-10T12:56:14.183648+00:00
 Available system pagesizes:
  4K, 2048K
2020-11-10T12:56:14.183726+00:00
 Supported system pagesize(s):
2020-11-10T12:56:14.183775+00:00
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
2020-11-10T12:56:14.183818+00:00
        4K       Configured               7         1749997        NONE
2020-11-10T12:56:14.183918+00:00
     2048K             1200            4609            1191        NONE
2020-11-10T12:56:14.183959+00:00
RECOMMENDATION:
2020-11-10T12:56:14.184101+00:00
 1. For optimal performance, configure system with expected number
 of pages for every supported system pagesize prior to the next
 instance restart operation.
2020-11-10T12:56:14.189604+00:00

As you can see I have a report that tells me my memlock (/etc/security/limits.conf) setting is fine, but I’ve had to allocate 1.7M pages of 4KB (along with almost all the available 2MB pages) to make space for the SGA. The report even shows me how many huge pages I need for this instance.

I’m not an expert at the O/S level but I think that means every process connecting to the SGA will (eventually) need a memory map of around 13.5MB (1.7M addresses * 8 bytes) to convert logical memory addresses to physical memory addresses if it needs to address the whole SGA.

4 Comments »

  1. Jonathan,

    “But you also have a single shared memory TLB for Hugepages.” This part is not clear. Fewer TLB entries is obvious due to the 2MB pages v/s 4KB pages.

    If there is a “single shared memory TLB”, I imagine it could apply for regular page shared memory segments too, as long as it is locked in memory. I guess the related question is: “is it only the huge pages shared memory segments that can be locked into memory?”

    Comment by naresh — February 22, 2018 @ 1:42 pm GMT Feb 22,2018 | Reply

    • Naresh,

      It’s been quite a long time since I last looked at these details, but the shared memory map “intimate shared memory” (and for Solaris DISM – dynamic intimate shared memory) has been around for a long time and can be used independently of huge pages. I don’t think I had been aware that it came into play automatically with huge pages, but maybe that’s O/S and version dependent anyway; and I have to plead ignorance of any direct connection between shared memory maps and locking memory.

      Comment by Jonathan Lewis — February 28, 2018 @ 10:11 am GMT Feb 28,2018 | Reply

  2. How do Huge Pages work in a VM environment where memory is normally over-allocated?

    Thank you

    Comment by Amir — February 22, 2018 @ 7:43 pm GMT Feb 22,2018 | Reply

    • Amir,

      I see that in a follow-up email to the list (links added to the article), Stefan Koehler has already commented on this topic, and I can’t add to what he says.

      Comment by Jonathan Lewis — February 28, 2018 @ 10:15 am GMT Feb 28,2018 | Reply


RSS feed for comments on this post. TrackBack URI

Comments and related questions are welcome.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Website Powered by WordPress.com.