Author: roscoe

How Distance can Affect Throughput

slothOffsite backups are important but Round Trip Time should be taken into consideration for particular transfer protocols. TCP connections can be severely hampered under certain conditions from what is known as “Bandwidth – Delay Product” and Wikipedia has the following descriptive article:

In data communications, bandwidth-delay product refers to the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, an amount of data measured in bits (or bytes), is equivalent to the maximum amount of data on the network circuit at any given time, i.e., data that has been transmitted but not yet acknowledged.

To use an extreme example, a tiny 64kB TCP buffer size would constrain a 100Mibit/s interface to 2.6Mibit/s if there is a consistent 200ms latency whereas a 50ms latency would allow 10.5 Mibit/s.

Fortunately modern systems are no longer defaulted with a small TCP buffer size but tweaks were required prior to kernel 2.6. The following system parameters can be explored:

  • proc.sys.net.core.wmem_max (Maximum send window size)
  • proc.sys.net.core.rmem_max (Maximum receive window size)
  • proc.sys.net.ipv4.tcp_wmem (Reserved memory for TCP send buffer)
  • proc.sys.net.ipv4.tcp_rmem (Reserved memory for TCP receive buffer)

Always be overly cautious when increasing buffers on production servers because it can potentially cause instability. Improvements can be seen immediately for FTP (more importantly FTPS) over long inflight time but there will still be circumstances where changes won’t seem to show any effect even with the best of configurations in place, such as when SO_SNDBUF and SO_RCVBUF are static for setsockopt().

Sender side autotuning has been present for quite some time but receiver limits are now usually included and this can be verified by reading net.ipv4.tcp_window_scaling and net.ipv4.tcp_moderate_rcvbuf:

sysctl net.ipv4.tcp_window_scaling net.ipv4.tcp_moderate_rcvbuf
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_moderate_rcvbuf = 1

Interestingly SSH2 multiplexes sessions over a TCP connection, which means patches to OpenSSH such as High Performance SSH/SCP are necessary to attain greater transfer rates for the likes of SCP.

I would like to take this opportunity to wish everyone a wonderful festive season and upcoming New Year!

prague-winter

Sweet Greetings from Prague!

gingerbread-museum-a

praguePrague leaves me speechless – probably in the same way that it inspired Walt Disney. Some places are great to live in while others are better to visit as a tourist but in my experience, few can be entrusted to offer a quality of life better than the cities of Prague or Paris.

Prague has exceeded my every hopeful expectation. The residents are friendly and there are regular events which are interesting and unique for all tastes ranging from Rihanna and Justin Bieber performances to temporary gallery exhibits.

Transit is efficient with an incredible metro and tram network which is consistently punctual.

park
This might be the fabled gingerbread house.

The sheer variety of nature within the city is astonishing. It would be challenging to explore without discovering expansive parks and nature trails seemingly around every corner – many of which look like a scene from a fairy tale.

Recently Britain voted for the unthinkable, which complicates matters for me since it represents one of my nationalities. Overall, my investment and travel lifestyle intentions remain mostly unchanged.

Whatever the outcome, I still haven’t been everywhere but it is on my list.

Benefits of an Efficient Memory Allocator

A developer friend of mine doing work for a mobile operator in West Africa needed advice because their database setup experienced severe congestion over Valentine’s Day. I mariadbcould recall us discussing the same problem back when we studied together and my advice then was to replace MySQL with Percona. Eventually MariaDB was the chosen replacement, which is also a great choice.

Talks with government officials are underway to migrate their databases to a specialist PaaS (Platform as a Service) which should ultimately ensure ongoing scalability for the future but in the meantime developers will each be provisioned with a virtual system installed with MariaDB 5.5. Workloads and tweaks can then be isolated but resource allocation is scarce because of limited hardware.

My personal memory allocator of choice has always been jemalloc simply because it operates with leaner memory utilisation. Memory consumption can be difficult to measure because the reading can differ greatly from one extremely short interval to another when an optimised memory allocator is used. My recommendation was jemalloc because the developer servers won’t be single purpose but he wished to know my opinion of TCMalloc.

TCMalloc is a seriously fast memory allocator which is part of gperftools but there are common inaccuracies surrounding its memory handling. Many mention that memory is never released back to the system by TCMalloc but this is not factual information and can even be controlled by adjusting the TCMALLOC_RELEASE_RATE and TCMALLOC_HEAP_LIMIT_MB environment variables as outlined in page_heap.cc. The default TCMALLOC_RELEASE_RATE value is 1.0 which means memory is released, albeit slowly.

There is no denying that TCMalloc can overpower jemalloc in overall performance but only by a small margin in many cases. It has been some time since last I checked, so I decided to conduct some benchmarks on very limited virtual servers to test the benefits when compared with glibc.

The usage of an alternative memory allocator can be determined in a variety of different ways such as checking the MariaDB log file after startup or perhaps manually with pmap, lsof or in /proc such as in the examples below:

[root@benchmark ~]# pmap $(pidof mysqld) | grep malloc
00007f7e1404f000 196K r-x-- libjemalloc.so.1
00007f7e14080000 2044K ----- libjemalloc.so.1
00007f7e1427f000 8K r---- libjemalloc.so.1
00007f7e14281000 4K rw--- libjemalloc.so.1

[root@benchmark ~]# grep malloc /proc/$(pidof mysqld)/maps
7f7e1404f000-7f7e14080000 r-xp 00000000 fd:01 27453423 /usr/lib64/libjemalloc.so.1
7f7e14080000-7f7e1427f000 ---p 00031000 fd:01 27453423 /usr/lib64/libjemalloc.so.1
7f7e1427f000-7f7e14281000 r--p 00030000 fd:01 27453423 /usr/lib64/libjemalloc.so.1
7f7e14281000-7f7e14282000 rw-p 00032000 fd:01 27453423 /usr/lib64/libjemalloc.so.1

Each benchmark is based on a default MariaDB installation with sysbench 0.4.12 and is executed four times. The standard complex OLTP (On Line Transaction Processing) benchmarks are used instead of the newer customisable Lua workloads. The following arguments are specified for each test, proceeded with a table drop / recreation and reboot:

–oltp-table-size=2000000
–max-time=300

Two separate CentOS 7 virtual servers with matching E5645 CPU clockspeeds were used but with differing memory and core count assignments:

  • 1 core at 2.4 GHz with 2 GiB of memory
  • 4 cores of 2.4 GHz with 1 GiB of memory

glibcjemalloctcmallocThe results show a benefit for InnoDB with either TCMalloc or jemalloc over glibc even for low end specifications. There’s a narrowed performance gap between the server with constrained memory versus the server with a limited processor configuration as thread counts increase when glibc is replaced.

MariaDB 10 is now built with jemalloc by default.

My 2016 Local Financial Markets Playbook

gingerbread-aA few days ago I read an article that contained a sentence which I consider to be profoundly disturbing written by a prominent economist:

“I am convinced the rand’s undervaluation will correct but I am unsure how it will correct.”

In August of last year I predicted a US Dollar / Rand rate of at least R15 when it was valued at R12.50. Market commentators often refer to the Rand as unjustifiably depreciated but the end of year rate was R15.49.

At least one trading desk ran the joke “sell EM if you have, sell EM if you don’t” two weeks ago in the midst of a huge selloff. Well done to all who saw the opportunity.

It is my belief that the Rand’s value will deteriorate further during 2016. I have prepared a list of key risks which South Africa will face this year:

  • A credit rating downgrade regarding our foreign debt as sub investment grade. Local currency credit should remain investment grade.
  • Confirmation if Jacob Zuma’s intention is to serve as state president for a further term come 2017. This would be oppositional to our Constitution.
  • Additional rate hikes by both the Federal Reserve as well as Bank of England which cause their currencies to become more attractive.
  • Increased nationwide food and water shortages with El Niño conditions estimated to extend well into 2017.
  • The introduction of winter season with arguably less electricity generation by Eskom, a situation which has not shown improvement since 2013.
  • Further slowdown of China’s economy which adversely affects demand for commodities.
  • Unforeseen expenditure to bail out state owned enterprises such as an effectively insolvent South African Airways.

South Africa’s contingency reserve is already depleted and we are only just starting a new year with these problems. On the other side of the spectrum are a few scenarios which can greatly help the Rand:

  • The replacement of President Jacob Zuma.
  • A significant increase regarding the price of oil could result in the return of investor appetite to emerging markets as a whole.
  • India could replace China’s previous demand for commodities due to infrastructure renewal.
  • Strength as a result from a local credit rating downgrade being avoided.

USD / ZAR Forecasts:

  • Good Case: R19 by the first half of 2016.
  • Good Case: R21 by the end of 2016.
  • Excellent Case: R18 by the end of 2016.
  • Bad Case: R28 by the end of 2016.

Excellent Case = Replacement of President Jacob Zuma with a more capable successor.
Good Case = Avoidance of a foreign credit ratings downgrade.
Bad Case = Foreign credit ratings downgrade.

Economic Forecasts:

  • GDP growth of between 0% and 0.5% with the start of a recession in the second half of the year.
  • Retailers will experience tremendous pressure from rising interest rates and lack of disposable income available for consumers. Higher levels of inflation will cause sales volume to decrease.
  • Banks should encounter increased non performing loans.

Should spare capital need to be allocated then an ideal financial advisor would make offshore products available as an option as well as possess multi asset capabilities. Actions by the Reserve Bank can be taken advantage of but be wary because carry trades here are no longer effective, as Japanese investors have recently discovered. Interest rate increases at these levels will likely harm economic growth and cause no swing to reach our inflation target.

Usually a backdrop of rising interest rates would dictate otherwise but offshore real estate is still the asset class of choice in my opinion. A REIT structure can form the composition of the asset holding thereby avoiding the need to be a landlord. One good balance would be to also tilt towards shorter duration T – Bills.

Global equity indicies have fallen quite substantially but are only slightly lower than fairly valued. Should the trend continue then a tentative entry towards developed world indicies is warranted.

This would not be a time to panic because the sale of securities can often initiate a Capital Gains Tax event which can erode returns. Always seek advice from a suitably qualified and independent financial advisor before making any decisions.

IWjGk9z

The Brilliance of Node.js with AngularJS

UI (User Interface) as well as UX (User Experience) formed the foundation of my first real world development experience and will always remain a passion of mine.

An earlier Music for a Goodangularjs-logo Cause post detailed my interest to promote publicity for local musicians with the added benefit to aid animals who need help.

sealion
I dedicate effort for this noble cause when able, which has lead to a number of notably positive experiences with AngularJS.

Node.js has always fascinated me and I find it increasingly impressive the more I work with it. Thanks to amazing technology like this, we can help work towards a project so that animals like this little guy can be all smiles again.

Consider an example which involves a music trackbar. A trackbar was displayed for each track and this proved to not only be resource inefficient but also displeasing in appearance:

Separate Trackbars:

separate-trackbars

Unified Trackbar:

unified-trackbar

 

Changes to the play button styling aside, the only necessary adjustment was to create an additional card which contains the trackbar. I didn’t even need a second cup of coffee since only two lines of code were needed to be inserted.

Have a safe and wonderful Festive Season ahead with a great New Year everyone! See you next year.

Efficient Incident Responsiveness

Monitoring service availability is perhaps the most fundamental aspect which every web hosting service provider should be deeply cognisant of. There are many different technologies which can fulfill this role, notably Nagios but I prefer one of my personal favourites, Icinga, for this example.

An organisation can possess a great service monitoring implementation but incidents should still be responded to quickly and efficiently in a manner which respect company procedures. The easiest means of ensuring these concerns are adhered to would be to develop a frontend which utilises the monitoring system as a backend. Integration can leverage JSON or even SQL and I have pieced together a basic example which portrays the principal requirements which would commonly be needed.

Icinga offers out of the box JSON support (although Nagios can also be configured in a similar manner) and included below is a screenhot which depicts this functionality with reports concerning two Google checks which have been collecting data for quite some time:

json

This data can then be presented within the context of an organisation’s appropriate procedures and I have a handful included in the example:

  • The ability to “lock” a host so that other staff members can immediately be aware that the problem is being worked on and by who. It can also serve as a metric to track productivity and follow up times.
  • Creation of a comment. This is important because useful observations or attempts for resolution can be recorded.
  • Acknowledge the issue to be resolved.
  • Refresh the particular entry.
  • Escalate to senior staff.

Since the frontend is entirely customised by the organisation, various buttons and areas can be accessible according to user level criteria. Customer information can also be represented should it also integrate with a billing system’s database. The following screenshots portray an example with important aspects pieced together:

service-monitor-a service-monitor-b

 

 

This is merely an example and by no means a finished product. The code also needs complete restructuring with the consideration of NodeJS so that action updates, table content and pagination can be dynamic without the need for a full pageload.

The use of a third party service monitor can also be used so that customers can have peace of mind regarding statistical integrity.

I would be very interested to learn if your organisation has a similar setup in production and if any immediate benefits were thereof derived.

Music for a Good Cause

G58s806Recently I chose to dedicate a portion of my spare time to help charities in need through musical performances offered to up and coming music artists with very pleasing results.

The interest which was expressed proved to be substantial with many having dreamed of a live audience or stage performance. The concept was simple: if there was a willingness to perform for a good cause, then I would get behind them (admittedly with a bias towards helping animal organisations).

If I were to give any advice for applicants, it would be to ensure professionalism while at the same time grasp a solid understanding of the criteria. I wish it had been otherwise for many respondents and I do hope that they come across the opportunity again. Just a few of many examples are illustrated below:

Bad enquiries:

d

e

Slightly more acceptable:

b

c

And lastly, a very cute reply:

z

I hope to offer my assistance again in the future because South Africa has so much undiscovered musical talent which can be creatively leveraged to benefit everyone involved.

Buy The Rumour, Sell The Fact

bullbear I almost never need to change the television channel, it is either showing Bloomberg, BusinessDay TV or CNBC. There are many regular interviews whereby money managers as well as financial advisors can express their views and opinion.

It’s very important for most people to seek guidance from a financial advisor / planner since many individuals don’t have the time nor interest regarding markets. What holds even more importance is the candidate’s knowledge because it is alarming for me to witness professionals from various highly esteemed financial service providers issue recommendations which are based purely on price and not fundamentals.

Over the past few months, many experts had answered that the South African Rand had depreciated to such an extent that it would not be wise to convert their discretionary savings to a hard currency such as US Dollar or Pound Sterling. On more than one occasion, no basis or factual information was provided other than the conversion rate seeming too high. These individuals clearly choose to instead be oblivious to many critical facts which would oppose their view:

  • Rumour: Imminent United States interest rate hike cycle causing capital flight to safety from emerging markets.
  • Fact: Falling commodity prices which negatively affect commodity driven economies such as South Africa and Australia.
  • Fact: Conversion of China’s economy to further dampen demand for commodities.
  • Fact: Rapid deindustrialisation of South Africa’s economy due to labour demands which result in an uncompetitive cost of production.

Most of these points are facts, which together with the concerning rumour should cause the South African Rand to be extremely undesirable. I can only hope viewers did not consider the advice because they would have experienced a severe loss of their initial capital’s buying power. I have illustrated the long term as well as recently accelerating currency depreciation below:

08072015

Their advice was to rather invest on the local stock exchange with a bias towards businesses which earn their income offshore. This approach holds some merit as per David Shapiro‘s “The Great Trek” but if you are investing on a stock exchange to primarily gain international exposure, then you’re most likely approaching it the wrong way – unless of course you purchase a counter such as Naspers to gain Tencent with their subsidiaries for free. Why limit yourself to a small amount of companies when there are so many greater opportunities elsewhere?

If your financial advisor ever issues advice simply based on price, then please seek a second opinion. The price of any investment is composed of the underlying future value dependent on broader sentiment. The current price means little before the sum of parts and future prospects are taken into careful consideration.

In my opinion, a more prudent piece of advice would have perhaps been to suggest that global listed property be considered either in the form of an index or even physically as residential with a fixed interest rate, especially at incredibly low levels currently. With most equities expensive world wide (or at the very least fair – valued) and diminishing bond yields from developed nations outside of the United States, it would have been refreshing to hear an alternative view such as this but very few financial advisors would suggest a better suited product if the company they represent don’t offer it.

Regardless of currency moves, one should always review and periodically rebalance based on logically sound expectations. The US Dollar won’t sustain growing strength indefinitely but for the time being I would follow the cycle rather than look through the cycle. There is a reason why the saying “don’t fight the Fed” is considered wise.

EDIT:

mzls1YXc

Congratulations all who seized the uptrend! My personal target for USD/ZAR is easily R15.00 as a minimum by year end and am buying the dips produced by a better than expected narrowed current account gap. I will continue to buy into any strength followed by profit taking after each double digit reversal.

Best of luck for everyone involved!

Harden procfs Security using hidepid

securityThe objective of hidepid is to ensure privacy concerning process information for standard users and its presence can prove beneficial for a multi tenant environment.

Wikipedia’s article related to procfs describes it as follows:

procfs (or the proc filesystem) is a special filesystem in Unix-like operating systems that presents information about processes and other system information in a hierarchical file-like structure, providing a more convenient and standardized method for dynamically accessing process data held in the kernel than traditional tracing methods or direct access to kernel memory.

Linux Kernel version 3.3 initially introduced the hidepid mount option for procfs quite some time ago but its usage isn’t always implemented. CentOS 6.3 and above have since offered full support for its usage. In the past the kernel source fs/proc/base.c needed to manually be patched with the line below to achieve the same capability:

inode->i_mode = S_IFDIR|S_IRUSR|S_IXUSR;

The following options can be defined for hidepid:

hidepid=0: Disable hidepid.
hidepid=1: Users other than root are only capable of realising information regarding their own processes but can still manually inspect /proc to gather references such as Process IDs.
hidepid=2: The ideal setting which will ensure privacy of /proc amongst standard users.

One factor to keep in mind is hidepid will not protect actual processes which do exist – it serves to only make process information more private.

Aside from modifying the system’s fstab, hidepid can be activated at runtime by remounting /proc as follows:

mount -o remount,hidepid=2 /proc

Rethinking Charity into Gifts

donations

Animal welfare is a subject which is very important to me. Recently it was my birthday and I wanted to share my personal feelings about supporting a good cause, especially since a celebratory period of gift giving is fast approaching.

People often wonder what gifts to purchase for others. They debate if the gift will be appreciated or unique enough to impress the recipient. I recently came to my own conclusion:

Gifts in the form of personalised donations to support a good cause is not only almost guaranteed to be unique but also appreciated, especially if it relates to the individual’s pet.

An example would be that some time ago I was planning someone’s birthday present for later in the year. I wanted it to be special but also personal at the same time so I decided on organising a photo of their pet as the front cover of a charity related magazine concerning animals. Sadly I couldn’t complete the project but this would have been a powerful and amazing present which the person would have cherished forever.

Please consider a personalised donation to a charitable cause if you are unable to decide on a gift this December. Here are a few which I have come across:

  • “Adoption” of a kennel. A personal photo and choice of wording is attached to the front of the kennel in honour of someone or their pet.
  • Purchase of a brick to construct an animal sanctuary. Each brick is labeled with a person’s name or pet.
  • A “donators” area on the organisation’s website which can list their name. Admittedly this isn’t as personal as a physical gift.

There are so many avenues for charities and animal welfare organisations to leverage personalised gifts to encourage donations. What’s vitally important is that their online presence be seamlessly integrated for both standard donations and more importantly their personalised donations offerings. Given time, personalised donations can become far more popular than standard donations. Many people feel burdened with the prospect of donating but this can change in the form of a worthwhile and heartwarming gift.

If you operate such an organisation and need ideas, then please get in contact with me and I will gladly help.