Offsite backups are important but Round Trip Time should be taken into consideration for particular transfer protocols. TCP connections can be severely hampered under certain conditions from what is known as “Bandwidth – Delay Product” and Wikipedia has the following descriptive article:

In data communications, bandwidth-delay product refers to the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, an amount of data measured in bits (or bytes), is equivalent to the maximum amount of data on the network circuit at any given time, i.e., data that has been transmitted but not yet acknowledged.

To use an extreme example, a tiny 64kB TCP buffer size would constrain a 100Mibit/s interface to 2.6Mibit/s if there is a consistent 200ms latency whereas a 50ms latency would allow 10.5 Mibit/s.

Fortunately modern systems are no longer defaulted with a small TCP buffer size but tweaks were required prior to kernel 2.6. The following system parameters can be explored:

• proc.sys.net.core.wmem_max (Maximum send window size)
• proc.sys.net.core.rmem_max (Maximum receive window size)
• proc.sys.net.ipv4.tcp_wmem (Reserved memory for TCP send buffer)
• proc.sys.net.ipv4.tcp_rmem (Reserved memory for TCP receive buffer)

Always be overly cautious when increasing buffers on production servers because it can potentially cause instability. Improvements can be seen immediately for FTP (more importantly FTPS) over long inflight time but there will still be circumstances where changes won’t seem to show any effect even with the best of configurations in place, such as when SO_SNDBUF and SO_RCVBUF are static for setsockopt().

Sender side autotuning has been present for quite some time but receiver limits are now usually included and this can be verified by reading net.ipv4.tcp_window_scaling and net.ipv4.tcp_moderate_rcvbuf:

sysctl net.ipv4.tcp_window_scaling net.ipv4.tcp_moderate_rcvbuf
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_moderate_rcvbuf = 1

Interestingly SSH2 multiplexes sessions over a TCP connection, which means patches to OpenSSH such as High Performance SSH/SCP are necessary to attain greater transfer rates for the likes of SCP.

I would like to take this opportunity to wish everyone a wonderful festive season and upcoming New Year!

A developer friend of mine doing work for a mobile operator in West Africa needed advice because their database setup experienced severe congestion over Valentine’s Day. I could recall us discussing the same problem back when we studied together and my advice then was to replace MySQL with Percona. Eventually MariaDB was the chosen replacement, which is also a great choice.

Talks with government officials are underway to migrate their databases to a specialist PaaS (Platform as a Service) which should ultimately ensure ongoing scalability for the future but in the meantime developers will each be provisioned with a virtual system installed with MariaDB 5.5. Workloads and tweaks can then be isolated but resource allocation is scarce because of limited hardware.

My personal memory allocator of choice has always been jemalloc simply because it operates with leaner memory utilisation. Memory consumption can be difficult to measure because the reading can differ greatly from one extremely short interval to another when an optimised memory allocator is used. My recommendation was jemalloc because the developer servers won’t be single purpose but he wished to know my opinion of TCMalloc.

TCMalloc is a seriously fast memory allocator which is part of gperftools but there are common inaccuracies surrounding its memory handling. Many mention that memory is never released back to the system by TCMalloc but this is not factual information and can even be controlled by adjusting the TCMALLOC_RELEASE_RATE and TCMALLOC_HEAP_LIMIT_MB environment variables as outlined in page_heap.cc. The default TCMALLOC_RELEASE_RATE value is 1.0 which means memory is released, albeit slowly.

There is no denying that TCMalloc can overpower jemalloc in overall performance but only by a small margin in many cases. It has been some time since last I checked, so I decided to conduct some benchmarks on very limited virtual servers to test the benefits when compared with glibc.

The usage of an alternative memory allocator can be determined in a variety of different ways such as checking the MariaDB log file after startup or perhaps manually with pmap, lsof or in /proc such as in the examples below:

[root@benchmark ~]# pmap $(pidof mysqld) | grep malloc
00007f7e1404f000 196K r-x-- libjemalloc.so.1
00007f7e14080000 2044K ----- libjemalloc.so.1
00007f7e1427f000 8K r---- libjemalloc.so.1
00007f7e14281000 4K rw--- libjemalloc.so.1

[root@benchmark ~]# grep malloc /proc/$(pidof mysqld)/maps
7f7e1404f000-7f7e14080000 r-xp 00000000 fd:01 27453423 /usr/lib64/libjemalloc.so.1
7f7e14080000-7f7e1427f000 ---p 00031000 fd:01 27453423 /usr/lib64/libjemalloc.so.1
7f7e1427f000-7f7e14281000 r--p 00030000 fd:01 27453423 /usr/lib64/libjemalloc.so.1
7f7e14281000-7f7e14282000 rw-p 00032000 fd:01 27453423 /usr/lib64/libjemalloc.so.1

Each benchmark is based on a default MariaDB installation with sysbench 0.4.12 and is executed four times. The standard complex OLTP (On Line Transaction Processing) benchmarks are used instead of the newer customisable Lua workloads. The following arguments are specified for each test, proceeded with a table drop / recreation and reboot:

–oltp-table-size=2000000
–max-time=300

Two separate CentOS 7 virtual servers with matching E5645 CPU clockspeeds were used but with differing memory and core count assignments:

• 1 core at 2.4 GHz with 2 GiB of memory
• 4 cores of 2.4 GHz with 1 GiB of memory

The results show a benefit for InnoDB with either TCMalloc or jemalloc over glibc even for low end specifications. There’s a narrowed performance gap between the server with constrained memory versus the server with a limited processor configuration as thread counts increase when glibc is replaced.

MariaDB 10 is now built with jemalloc by default.

UI (User Interface) as well as UX (User Experience) formed the foundation of my first real world development experience and will always remain a passion of mine.

An earlier Music for a Good Cause post detailed my interest to promote publicity for local musicians with the added benefit to aid animals who need help.

I dedicate effort for this noble cause when able, which has lead to a number of notably positive experiences with AngularJS.

Node.js has always fascinated me and I find it increasingly impressive the more I work with it. Thanks to amazing technology like this, we can help work towards a project so that animals like this little guy can be all smiles again.

Consider an example which involves a music trackbar. A trackbar was displayed for each track and this proved to not only be resource inefficient but also displeasing in appearance:

Separate Trackbars:

Unified Trackbar:

Changes to the play button styling aside, the only necessary adjustment was to create an additional card which contains the trackbar. I didn’t even need a second cup of coffee since only two lines of code were needed to be inserted.

Have a safe and wonderful Festive Season ahead with a great New Year everyone! See you next year.

The objective of hidepid is to ensure privacy concerning process information for standard users and its presence can prove beneficial for a multi tenant environment.

Wikipedia’s article related to procfs describes it as follows:

procfs (or the proc filesystem) is a special filesystem in Unix-like operating systems that presents information about processes and other system information in a hierarchical file-like structure, providing a more convenient and standardized method for dynamically accessing process data held in the kernel than traditional tracing methods or direct access to kernel memory.

Linux Kernel version 3.3 initially introduced the hidepid mount option for procfs quite some time ago but its usage isn’t always implemented. CentOS 6.3 and above have since offered full support for its usage. In the past the kernel source fs/proc/base.c needed to manually be patched with the line below to achieve the same capability:

inode->i_mode = S_IFDIR|S_IRUSR|S_IXUSR;

The following options can be defined for hidepid:

hidepid=0: Disable hidepid.
hidepid=1: Users other than root are only capable of realising information regarding their own processes but can still manually inspect /proc to gather references such as Process IDs.
hidepid=2: The ideal setting which will ensure privacy of /proc amongst standard users.

One factor to keep in mind is hidepid will not protect actual processes which do exist – it serves to only make process information more private.

Aside from modifying the system’s fstab, hidepid can be activated at runtime by remounting /proc as follows:

mount -o remount,hidepid=2 /proc