RAM capacity and speed
Consider the total RAM capacity of your servers. This is not nearly the quandary it once was, but it is still vital to plan carefully. If your server model accepts say at most 16 memory modules and your initial calculations are that 64 GB of RAM is what you need, resist the urge to save a few bucks on 4 GB DIMMs. Any future upgrades would require removing installed modules to make way for newer, larger DIMMs. Situations that could increase your RAM needs include:
- Adding more storage drives per server
- Swapping in larger storage drives for smaller
- Cluster expansion via additional servers may increase per-process memory requirements
- Future use of caching software
- Operating system or Ceph setting tweaks
- Conversion to a converged storage/compute architecture
It is often the case that the largest available memory modules carry a hefty per-GB price premium, so one step down may be the sweet spot of preserving both CapEx and expandability.
Servers today are typically able to use memory of multiple speeds, though mixing modules will generally step operation down to the lowest common denominator. Your server vendor may offer memory at multiple speeds, technologies, and price points, and the choice of module size, count, and layout may influence the speed at which the entire memory system is able to run. Certain CPU choices within a product line may also have memory or QPI channel differences that affect configuration options, total configurable memory, and speed. These concerns may increase with larger system configurations, for example, a 72-bay four-socket system where meticulous tuning may be required to extract the last 20% of performance.
Many factors influence RAM requirements. A suggested rule-of-thumb was once a base of 16 GB plus at least 2 GB for each Ceph daemon (MON, RGW, MDS), and at least 2 GB per OSD. With 16 GB DIMMs common and affordable today, we suggest that for any Ceph server role you provision a minimum of 64 GB, with dense OSD nodes adding 2 GB for each OSD after the first ten. Again, when selecting memory module sizes ensure that you count available slots and provision DIMMs in a way that allows future expansion without throwing away your initial investment. However, at any given time the very latest and densest module size is often disproportionately expensive, making the modules a step down the sweet spot. Take care on multi-socket systems that each CPU socket has its own memory slots that are ideally filled in a balanced fashion.