Locale issues with Zend Lucene search

We use Zend Lucene Search on a PHP ecommerce website and ran into some issues where records added to the search index from the website interface weren’t the same as when created from the command line. It turned out to be a locale issue and setting the locale fixed the problem.

Error messages

A clue that something wasn’t working correctly was this error message:

iconv(): Detected an illegal character in input string

There are a variety of conversions that can be done which fix this error, but our problem appears to have been caused by a non-configured locale, defaulting to “C” and which does not support UTF8.

Check what locale is currently being used

In PHP, you can call “setlocale(LC_ALL, 0)” to find out what the locale is currently set as. Running through Nginx with PHP-FPM, it output this:

C

and from the command line this:

en_NZ.UTF-8

Running “locale” from a SSH terminal session output this:

LANG=en_NZ.UTF-8
 LANGUAGE=
 LC_CTYPE="en_NZ.UTF-8"
 LC_NUMERIC="en_NZ.UTF-8"
 LC_TIME="en_NZ.UTF-8"
 LC_COLLATE="en_NZ.UTF-8"
 LC_MONETARY="en_NZ.UTF-8"
 LC_MESSAGES="en_NZ.UTF-8"
 LC_PAPER="en_NZ.UTF-8"
 LC_NAME="en_NZ.UTF-8"
 LC_ADDRESS="en_NZ.UTF-8"
 LC_TELEPHONE="en_NZ.UTF-8"
 LC_MEASUREMENT="en_NZ.UTF-8"
 LC_IDENTIFICATION="en_NZ.UTF-8"
 LC_ALL=

which would indicate the CLI script is picking the locale from the system, but Nginx/PHP-FPM is not.

How to fix

It’s possible to set the default locale in the php.ini file, although it might not be a good idea if you run many websites on your server as it could cause issues.

Instead use the setlocale() to set it specifically for your website. Check out the www.php.net/setlocale manual page for more information about the function.

In my case, we did this:

setlocale(LC_ALL, 'en_NZ.UTF-8');

This then solved the issue with adding documents to the Lucene index.

Categories PHP

PHP: Class XsltProcessor not found

A PHP install on Debian 5 by default does not include the Xslt extension which means the following error will occur if trying to use the XsltProcessor class: " Fatal error: Class ‘XsltProcessor’ not found…". This post shows how to install the extension on Debian 5.

Categories PHP

PHP: Write and append to files with file_put_contents

The PHP file_put_contents() function is a useful shortcut for dumping a string into a file (compared with using fopen, fputs etc). By default it will create a new file if it does not exist, or overwrite the file if it does exist. Recently I re-read the documentation for this function and discovered it is also possible to append to a file with file_put_contents.

Categories PHP