Minio S3 small setup -22.8 Gb/sec read.
Small S3 minio setup for benchmarking Iceberg/Delta on prem with S3.
This is the non clustred minio setup gives 22.8 Gigabyte/Sec sustained read which is better than mediocre ! Tested with Wasabi's s3 performance bench in Go.
(base) rickard@Allfadern:./s3-benchmark -a minioadmin -b data2 -s bajsanka -u http://localhost:9000 -z 16M -t 32
Wasabi benchmark program v2.0
Parameters: url=http://localhost:9000, bucket=data2, region=us-east-1, duration=60, threads=32, loops=1, size=16M
status code: 409, request id: 175CE2A0FF5F183D, host id:
Loop 1: PUT time 60.0 secs, objects = 22838, speed = 5.9GB/sec, 380.3 operations/sec. Slowdowns = 0
Loop 1: GET time 60.0 secs, objects = 87707, speed = 22.8GB/sec, 1461.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.7 secs, 13700.5 deletes/sec. Slowdowns = 0Ingredients workstation (32 core AMD Threadripper pro):
- Minio Single mode
- 4x samsung 1735 NVME pci 4.0
- ASUS Pro WS WRX80E-SAGE SE
My installation script with settings can be found here https://github.com/Ignalina/racksetup/tree/main/dist/rocky_9_1/installation_kits/singleminio
Doing this with 4x Samsung 9a3 NVME PCI 4.0 on a normal server platform.
s3-benchmark]# ./s3-benchmark -a minioadmin -b data2 -s changeme -u http://localhost:9000 -z 16M -t 48
Wasabi benchmark program v2.0
Parameters: url=http://localhost:9000, bucket=data2, region=us-east-1, duration=60, threads=48, loops=1, size=16M
Loop 1: PUT time 60.1 secs, objects = 23566, speed = 6.1GB/sec, 392.3 operations/sec. Slowdowns = 0
Loop 1: GET time 60.0 secs, objects = 82305, speed = 21.4GB/sec, 1371.0 operations/sec. Slowdowns = 0
Loop 1: DELETE time 2.0 secs, 12032.6 deletes/sec. Slowdowns = 0Ingredients Server (24 core Supermicro):
- Minio Single mode
- 4x samsung 9A3 NVME pci 4.0
- Supermicro H12ssw-nt
- Retimer AOC-SLG4-4E4T connecting NVME's drive
My installation script with settings can be found here https://github.com/Ignalina/racksetup/tree/main/dist/rocky_9_1/installation_kits/singleminio
Each Samsung disk has the following setup (4096 bytes blocks)
root@Allfadern:/var# nvme id-ns -H /dev/nvme1n1
NVME Identify Namespace 1:
....
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x1 Better
LBA Format 1 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0x3 Degraded
LBA Format 2 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best (in use)
LBA Format 3 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format 4 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0x3 Degraded
You can change (WARNING DELETES DATA!!!!!!!) between drives suggested optimal lba (block size) by "nvme format --lbaf=X /dev/nvme1n1 --force" where X is given by prev command (nvme id-ns -H /dev/nvme1n1)
The /etc/fstab showing xfs filesystem per disk
LABEL=DISK1 /mnt/disk1 xfs defaults,noatime 0 2
LABEL=DISK2 /mnt/disk2 xfs defaults,noatime 0 2
LABEL=DISK3 /mnt/disk3 xfs defaults,noatime 0 2
LABEL=DISK4 /mnt/disk4 xfs defaults,noatime 0 2
The etc/default/minio
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=bajsanka
MINIO_OPTS="--console-address :9001 --address :9000"
MINIO_VOLUMES="/mnt/disk{1...4}"
Here is me attempting to install last Samsung 1735 Drive while filming and listening to MBR (Master Boot Record).