Support for Multibyte Strings

[raamdev]

Forked from #635.

@jaswsinc writes...

We should do a better job (even better than WP itself) in dealing with UTF-8. Given the amount of string manipulation that occurs in Comet Cache, we should:

• Adopt the use of all mb_*() functions where applicable.
• Always pass the /u flag to any preg_*() functions that we use.
• Ensure that WP PHP RV requires the mbstring extension to prevent issues with sites that don't have it installed.

This work can probably be moved to a separate issue though; i.e., we should have a separate issue that will be for UTF-8 enhancements. See this article.

Related GitHub issue: #640 (comment)

---

Also noting here at an announcement was made in Comet Cache v160222 to any site owner that did not have the mbstring extension installed that the PHP mbstring extension would be required by Comet Cache after March 1st.

[jaswrks]

• [x]

A first and very easy step here would be to search for all calls to:

• preg_match
• preg_replace
• preg_replace_callback

and on the end of the regex pattern add a u modifier.

e.g., /pattern/ becomes /pattern/u
e.g., /pattern/i becomes /pattern/ui

[jaswrks]

• [x]

Next, we need to search for instances of:

• strpos
• stripos

and a decision needs to be made that is based on the following.

• If strpos is being used to detect the number of bytes, leave it as-is.
• If strpos is being used to detect the number of characters (or to search for another string, most prevalent in Comet Cache), change it to mb_strpos or mb_stripos respectively.

[jaswrks]

• [x]

Next, search for:

• strtolower
• strtoupper

And change all of those to mb_strtolower() or mb_strtoupper() respectively.

[jaswrks]

• [x]

Same thing for substr; i.e., should become mb_substr.

[jaswrks]

• [x]

Next, we will need to see which other UTF-8-incompatible functions we are using in Comet Cache; i.e., see which functions we use in PHP core for which there is no UTF-8 mb_* variation available. In those cases, we will need to build a replacement of our own as a polyfill.

e.g., There is no mb_str_ireplace, and there is no mb_str_pad() either. PHP is also missing mb_substr_replace(). I'm not sure if Comet Cache needs any of these (or others), but we need to dig through and see which might need polyfills.

Examples of a UTF-8 compatible polyfills:

• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/SubstrReplace.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/StrCaseCmp.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/StrPad.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/StrRev.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/StrSplit.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/Trim.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/UcFirst.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/UcWords.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/MbLcFirst.php
• https://github.com/websharks/core/blob/000000-dev/src/includes/classes/Core/Utils/ReplaceOnce.php

[jaswrks]

Referencing full list of existing mb_ functions provided by the mbstring extension.
http://php.net/manual/en/ref.mbstring.php

[jaswrks]

• Noting that the trim(), rtrim() and ltrim() PHP functions are generally safe to use on multibyte data. However, if we find instances of these functions where we are attempting to trim characters outside of the ASCII range, we will need to use preg_replace() with the /u modifier instead.

[raamdev]

If strpos is being used to detect the number of bytes, leave it as-is.
If strpos is being used to detect the number of characters (or to search for another string, most prevalent in Comet Cache), change it to mb_strpos or mb_stripos respectively.

Can you give an example of those two? I'm not seeing anything that looks like strpos() is being used to detect the number of bytes...

[jaswrks]

• [x]

Sorry, I stated that incorrectly. What I should have been talking about there was strlen() instead of strpos(). All instances of strpos() and stripos() should be updated to their mb_ counterparts.

---

Examples of strlen() looking for (or testing) the number of bytes:

if(!strlen($variable)) {
   // Empty.
}

header('Content-Length: '.strlen($export));

Examples of strlen() looking for the total number of characters.

if (strlen($string) > $max_length) {

$hook = GLOBAL_NS.substr($hook, strlen('zencache'));