極大化vsphere的效能,DELL perc6 加上 hp smartarray P410

最近一直在思考,如何可以極大化單台主機虛擬化的效能,這樣就算是資料中心,機器數量可以再減少,尤其是VMWARE的授權是算CPU的,節省主機就是節省費用,所以有了這個實驗

平台目前鎖定為AMD平台,原因在於,目前可購置到可以安裝48G以上的系統,價錢可以裝在家裡的XEON系統仍然太貴

分別由CPU , RAM , DISK IO , NETWOEK 等子系統來做強化


以下為配置的目標

1.CPU:

目前使用opteron,2路的機器是實驗的目標,opteron系列的機子之後還可以升級為六核心,CPU無論為雙核或是4核價錢都遠低於XEON,本次使用的主機板為supermicro H8DME-2

2.RAM:

目前預計目標為裝滿該平台最大數量,如以目前普遍可以買的的4G ECC REGISTERED的記憶體來說,該主機板可裝64G

3.網路:

除內建網路之外,另外加裝PCI-X 單卡雙PORT的PRO 1000 MT,由於該主機上還有兩根插槽,可以再做擴充,除了網路卡之外並且加裝QLOGIC的光纖卡,這樣可以同時使用 NFS , ISCSI , SAN

4.RAID卡

IO一向為最大的問題,除了直接掛載各種儲存設備以外,如何在本身的機器上極大化處理能力,變成本次實驗的重點,目前裝上 DELL PERC 6I 和 HP P410,裝滿主機最大數量硬碟開出2個1.9T的datastore作為儲存空間,圖下為RAID卡與軟體中的照片

5.電力消耗
單台主機的消耗在350w內,開機只有300w出頭

最後配置的設備清單

Case: supermicro 4U 745TQ-800B 加上 3BAY 5插槽的硬碟抽取盒
CPU: opteron 8347HE *2 1.9G*8
RAM : 4G *8 = 32G
RAID : HP P410 500G*7 (RAID50+ hot spare) , dell PERC 6i 500G*6 (RAID5+hot spare)
NETWORK : intel PRO 1000MT
FIBRE: Q-LOGIC 2462



後記
目前PCI-X的2G光纖卡和2PORT的網卡由於新主機的規格不再支援舊的介面,導致於3大廠的備品於市面上可以很便宜的買的到,opteron在拍賣中,雙核心的價錢已經慘烈到不到1000元,4U的機器於空間配置中能極大化的部份很多,比起很多主機買了之後插槽閒置,最好的配置就是,將其全部用完

而CPU可以雙插槽,其實RAID卡也可以,目前來說,三大廠都還沒有類似的作法,由於目前主流的三大廠機器,SAS的背板都是鎖定在一個RAID卡上面,導致於真的裝兩張RAID時,必須採取一內一外藉助類似DELL MD1000或是IBM System Storage EXP3000 的櫃子才能處理,仔細思考了一下,3U 16BAY的機器配上兩個Raid50是目前最好的配置,但目前應該沒有類似規格的東西的套裝伺服器可以買,其實仔細想想大部分的X3650或是DL380G5,都沒有將其記憶體插槽或是擴充槽用完,那些空間,如果能在配置與加裝,應該就不會有兩個XEON 5420的使用率只有不到50%的問題,以這樣的配置,在不藉助外加的儲存設備下如果加滿64G的記憶體,應該可以開出至少40-50台LAB的主機,或是將所有的CPU效能消耗完畢,單機只有350W-400W的電力,實在是物超所值

必須還是要提一下,supermicro的opteron 2000系列的主機板,無論於擴充.相容性,都是vshpere的上上之選,HP的P410根據某位苦主的測試,連HP自家的DL160G5都不能用,說來也實在是好笑,三大廠的東西在獨立伺服器的廠商平台上大和解,讓我們替supermicro拍拍手,好東西還是有他的價值

電力消耗一覽

測試時隨手註記,參考如下,待機時電力看來ＡＭＤ略低一些（忽略ＣＰＵ核心數差距）
ESX vsphere 4.0

AMD
CPU: opteron 2214 *2
RAM: 4G*8
I/O: DELL PERC6i+7*WD 500G AAKS

約270W

INTEL
CPU : E5335*2
RAM : 4G*6
I/O: DELL PERC6i+7*WD 500G AAKS

約360W

shell 小指令

CTRL-A 移到列的開頭
CTRL-E 移到列的結尾

esx所用的port列表

不翻譯了,直接看比較快

port 80
vCenter Server requires port 80 for direct HTTP connections. Port 80 redirects requests to HTTPS port443. This is useful if you accidentally use http://server instead of https://server.

port 389
This port must be open on the local and all remote instances of vCenter Server. This is the LDAP portnumber for the Directory Services for the vCenter Server group. The vCenter Server system needs tobind to port 389, even if you are not joining this vCenter Server instance to a Linked Mode group. Ifanother service is running on this port, it might be preferable to remove it or change its port to differentport. If needed, you can run the LDAP service on any port from 1025 through 65535.If this instance is serving as the Microsoft Windows Active Directory, change the port number from 389to an available port from 1025 through 65535.

port 443
The default port that the vCenter Server system uses to listen for connections from the vSphere Client.To enable the vCenter Server system to receive data from the vSphere Client, open port 443 in thefirewall.The vCenter Server system also uses port 443 to listen for data transfer from the vSphere Web AccessClient and other SDK clients.If you use another port number for HTTPS, you must use : when you log in to thevCenter Server system.

port 636
For vCenter Linked Mode, this is the SSL port of the local instance. If another service is running on thisport, it might be preferable to remove it or change its port to different port. If needed, you can run theSSL service on any port from 1025 through 65535.


port 902
The default port that the vCenter Server system uses to send data to managed hosts. Managed hostsalso send a regular heartbeat over UDP port 902 to the vCenter Server system. This port must not beblocked by firewalls between the server and the hosts, or between hosts.

port 902/903
Ports 902 and 903 must not be blocked between the vSphere Client and the hosts. These ports are usedby the vSphere Client to display virtual machine consoles.

port 8080
Web Services HTTP. Used for the VMware VirtualCenter Management Webservices.

port 8443
Web Services HTTPS. Used for the VMware VirtualCenter Management Webservices.

網頁上對於iscsi的參數說明

Could you please give me the detailed parameter of the 'IPfrag tuning', 'Jumbo Frames Config', 'NFS Daemon tuning' , 'Read ahead disk tuning' and 'iSCSI daemon option ' in the 'Tuning options'.
IPfrag Tuning - it si apcket fragmentation section. You can set arnage of it in this option

Jumbo Frames Config also called MTU(Maximum Transmission Unit) - refers to the size (in bytes) of the largest packet that a given layer of a communications protocol can pass onwards.

NFS Daemon tuning - you can set how many NFS daemons you want to have run in the system. On some system NFS causes NFS timeouts and chnging this value then helps. It also can improve NFS performance.

Read ahead disk tuning - with this option you can increase for better performance size of cache. In some cases it is rquierd to decrese it for better compatiblity.

iSCSI daemon option - in this option you can set values of iSCSI target:

a) MaxRecvDataSegmentLength - Sets the maximum data segment length that can be received. This value should be set to multiples of PAGE_SIZE. Currently the maximum supported value is 64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB. Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 8192.

b) MaxBurstLength - Sets the maximum amount of either unsolicited or solicited data the initiator may send in a single burst. Any amount of data exceeding this value must be explicitly solicited by the target. This value should be set to multiples of PAGE_SIZE. Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 262144.

c) MaxXmitDataSegmentLength - Sets the maximum data segment length that can be sent. This value actually used is the minimum of MaxXmitDataSegmentLength and the MaxRecvDataSegmentLength announced by the initiator. It should be set to multiples of PAGE_SIZE. Currently the maximum supported value is 64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB. Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 8192.

d) DataDigest - If set to "CRC32C" and the initiator is configured accordingly, the integrity of an iSCSI PDU's data segment will be protected by a CRC32C checksum. The default is "None". Note that data digests are not supported during discovery sessions.

e) MaxOutstandingR2T - Controls the maximum number of data transfers the target may request at once, each of up to MaxBurstLength bytes. The default is 1.

f) InitialR2T - If set to "Yes" (default), the initiator has to wait for the target to solicit SCSI data before sending it. Setting it to "No" allows the initiator to send a burst of FirstBurstLength bytes unsolicited right after and/or (depending on the setting of ImmediateData together with the command. Thus setting it to "No" may improve performance.

g) ImmediateData - This allows the initiator to append unsolicited data to a command. To achieve better performance, this should be set to "Yes". The default is "No".

h) DataPDUInOrder - It tells initiator if data has to be sent in order. Defualt is "Yes", which is also recommended.i) DataSequencerInOrder - It tells initiator if data has to be sent in order. Defualt is "Yes", which is also recommended.

j) HeaderDigest - If set to "CRC32C" and the initiator is configured accordingly, the integrity of an iSCSI PDU's header segments will be protected by a CRC32C checksum. The default is "None". Note that header digests are not supported during discovery sessions.

k) Wthreads - The iSCSI target employs several threads to perform the actual block I/O to the device. Depending on your hardware and your (expected) workload, the number of these threads may be carefully adjusted. The default value of 8 should be sufficient for most purposes.

open-e iscsi參數

open-e 建議的esx 參數

maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes

新版vSphere4的iscsi

之前在3.5,說實在的i-scsi的小毛病實在很多,這次的實驗計畫如下:

1.參數的改變效果

2.不同的target軟體影響會有多大

3.網卡真的會有神奇功能嗎

以下為官方文件對於參數檔案設置的說明

Header Digest

Increases data integrity. When header digest is enabled, the systemperforms a checksum over each iSCSI Protocol Data Unit’s (PDU’s)header part and verifies using the CRC32C algorithm.

Data Digest

Increases data integrity. When data digest is enabled, the systemperforms a checksum over each PDU's data part and verifies usingthe CRC32C algorithm.NOTE Systems that use Intel Nehalem processors offload the iSCSIdigest calculations for software iSCSI, thus reducing the impact onperformance.

MaximumOutstanding R2T

Defines the R2T (Ready to Transfer) PDUs that can be in transitionbefore an acknowledge PDU is received.

First Burst Length

Specifies the maximum amount of unsolicited data an iSCSI initiatorcan send to the target during the execution of a single SCSI command,in bytes.

Maximum BurstLength

Maximum SCSI data payload in a Data-In or a solicited Data-OutiSCSI sequence, in bytes.

Maximum ReceiveData Segment Length

Maximum data segment length, in bytes, that can be received in aniSCSI PDU.

Delayed ACK

Allows systems to delay acknowledgment of received data packets.