Log message:
xapian: update to 1.4.30. Changes:

API:

* Stem: 1.4.28 changed the English stemmer code to remove an exception
  for "skis" which we incorrectly thought wasn't needed.  This was not
  intended to be a functional change, so we've restored the exception.

* Stem: Fix a bug in Snowball's runtime code for decoding 4-byte UTF-8
  sequences.  This affects grouping checks against Unicode codepoints U+40000
  to U+7FFFF and U+80000 to U+FFFFF, and we would decode a higher codepoint
  value.  The largest codepoint used in a grouping in any stemmer we ship is
  U+0585 so this seems to be a latent bug.  Reported in Snowball by Stephane
  Carrez.

documentation:

* The API docs still said DatabaseError was thrown for operations on a closed
  database, but since 1.4.10 we throw its subclass DatabaseClosedError instead.

* INSTALL: List the more commonly used variable-like configure options and
  reorder the sections to present the information in a more logical order.

portability:

* Fix instance of undefined behaviour in Database::check() for glass databases.
  We would end up deleting a pointer to a non-virtual class via a pointer to
  the base class, so delete ended up called with the wrong size.  In practice
  this probably doesn't cause problems (it was detected by checks added to
  recent valgrind versions).

* Fix probe for -Bsymbolic-function on AIX.  The AIX linker doesn't support
  this flag, but it only emits a warning when it is used and that warning
  didn't match any of the patterns we already check for so we were detecting it
  as supported with the result that the build would use this option and AIX ld
  would emit a warning about ignoring it as unsupported.

* Fix compiler warnings in testsuite when building on AIX.

Log message:
xapian: update to 1.4.29. Changes:

API:

* Stem:

  + Optimise stemming algorithms.  The biggest improvement is for Tamil where
    there's a 47% reduction in the time taken to stem all the words in our test
    vocabulary.  Less dramatic improvements for Arabic, Danish, Dutch, English,
    French, Finnish, Irish, Kraaij-Pohlmann, Lithuanian, Norwegian, Swedish and
    Turkish.

  + Add dutch_porter as alias for current dutch stemmer for forward
    compatibility with the next release series where we will have
    kraaij_pohlmann as the default dutch stemmer.

testsuite:

* stemtest: Add support for gzipped vocabulary lists, which means we
  no longer skip testcase stemdict for Arabic.

glass backend:

* Improve exception message when asked to store too large a tag.  Reported by
  Jean-Francois Dockes on xapian-discuss.

* Optimise reading tag.  A special case for handling the first chunk of an item
  gives a measurable speed improvement.  Reported by uis.

portability:

* Work around autoconf 2.72 AC_SYS_LARGEFILE bug.  This configure probe always
  fails when run with the C++ compiler, like we do, which leads to large file
  support not being enabled when it isn't the default (e.g. on 32-bit Linux
  platforms).  Reported by Adrian Bunk.

* Fix C23 compatibility issues.  GCC 15 fails with compile errors due to C23
  dropping support for old-style function declarations and function
  definitions.

* Avoid calling frexp() on infinity or NaN.  The specification for frexp() says
  it returns an "unspecified" exponent for these cases.  Reported by \ 
awilfox.

* Fix GCC warning with 32-bit off_t.  Reported by Adrian Bunk.

* Fix compile error with newer emscripten.

build system:

* 1.4.28 had its library version information incorrectly set.  This resulted in
  the shared library having an incorrect SONUMBER - e.g. on Linux,
  libxapian.so.29 instead of libxapian.so.30.  This release has been made to
  fix this problem.  We have added a check to configure that the SONUMBER has
  the expected value to avoid repeats of this (it happened previously in 1.2.11
  as well).  Thanks to Vitaly Chikunov and Sven Joachim for reporting this.

Log message:
xapian-core: update to 1.4.26. Changes:

API:

* Weight: Document that Weight statistics DOC_LENGTH_MIN, DOC_LENGTH_MAX and
  WDF_MAX are for the shard rather than the whole database.  Usually this is
  what we want as with a sharded database it gives tighter bounds and so better
  match optimisation, but it does make them unsuitable for uses such as
  calculating a suitable offset to add to every get_sumextra() to allow
  implementing a weighting formula which can give a negative term independent
  weight contribution.  This case will be addressed in the next release series
  which also provides bounds such as DB_DOC_LENGTH_MIN which are for the whole
  database.

* LMWeight: This class was meant to implement the "Language Model" \ 
Weighting
  scheme, but we've discovered the implementation was incorrect and fixing it
  requires ABI-incompatible changes.  For 1.4.x we need to leave it in place so
  as not to break existing code, but it's now deprecated and we recommend
  avoiding using it.  It will be removed in the next release series and
  replaced with new separate classes implementing Language Model weighting with
  each smoothing.  Thanks to Sourav Saha for reporting this problem.

* PL2PlusWeight: Fix bug in implementation of formula.  Our variable mean is
  1/lambda_t from the PL2+ paper, so we need to check mean>1 for lambda_t<1 but
  we were actually checking mean<1 instead.  The result of this is that PL2+
  actually returned a zero weight unless the term occurred frequently enough in
  the collection.

* TradWeight::get_maxpart() no longer forces the wdf_max value to be at least
  one.  We used to do this so that a non-existent term in the query would cause
  it not to achieve 100%, but now we calculate percentages based on the number
  of matching subqueries, and it is more natural for a non-existent term to get
  zero weight (ditto for a term which always has wdf 0).  This was already
  addressed for BM25Weight in 1.2.1 back in 2010.

* Enquire::set_expansion_scheme(): Add "prob" as new preferred name for
  probabilistic query expansion, with the previous "trad" still being \ 
accepted
  for now.

* QueryParser::set_prefix() and set_boolean_prefix(): Allow an optional
  trailing `:` on the field name.  This makes the API here more consistent with
  ranges, where you need to include the `:` if you want one.  See #720.

testsuite:

* Catch and report if a testcase causes signal SIGPIPE.

* Suppress valgrind errors about calling memmove() with overlapping source and
  destination (which is valid, valgrind is just confused when memcpy() and
  memmove() share an implementation).

* Add more testing of weighting schemes.

* Mark checkstatsweight3 with a sharded database as XFAIL (expected to fail).
  This testcase was previous not run for sharded databases, with a FIXME
  comment noting this.  Investigating shows it's due to a bug where we use the
  shard's termfreqs rather than those for the whole database for an expanded
  wildcard, but this seems complex to fix.

matcher:

* Fix minor wildcard weighting bug spotted while reading the code.  We were
  returning too high a value from the first call to get_maxpart() in some
  cases.  Mostly this just means the matcher continue working when it could
  have stopped, but it will also cause MSet::get_termweight() to return a
  higher value than the actual known upper bound.

glass backend:

* Simplify file descriptor handling for lock files on Unix-like platforms
  which don't support OFD locks.  This eliminates corner cases where we
  could end up with file descriptors without close-on-exec set in the main
  process.

remote backend:

* xapian-tcpsrv: Use _exit() instead of exit() to end child processes which
  avoids the risk of duplicated output from stdio buffers getting copied by
  fork() then flushed in both processes.

* Simplify file descriptor handling when launching prog remote.

inmemory backend:

* Fix bug adding posting entries.

build system:

* Improve probe for -Bsymbolic-functions.  MSVC doesn't support this flag, but
  it only emits a warning when it is used and that warning didn't match any of
  the patterns we already check for so we were detecting it as supported.

* Report result of probe to determine compiler support for -Werror or
  equivalent.

documentation:

* Improve MSVC build instructions.  Thanks to Baran Demir for feedback.

* Improve formatting of stat_flags API documentation.

* sorting.rst: Replace custom weighting scheme documentation with a link to the
  more complete equivalent in "Getting Started with Xapian".

* remote.rst: Update to reflect that user metadata is fully supported (since
  1.2.4).

portability:

* Fix to compile as C++20 and C++23.

* Resolve SIGPIPE issues on NetBSD, which were causing testcase keepalive1 to
  fail.  These seem to be due to SO_NOSIGPIPE not working correctly there so
  we now use MSG_NOSIGNAL instead for NetBSD.

* Include <errno.h> for sys_errlist.  We already do this for the configure
  check but were failing to when actually using sys_errlist, which probably
  affects at least NetBSD.

* configure: Fix clang detection which wasn't working when configure determined
  a -std=X option was needed to get C++11 support.  The obvious symptom was
  that --enable-werror wouldn't add -Werror.

* configure: NetBSD automatically pulls in library dependencies, so set
  link_all_deplibs_CXX=no there.

* Avoid using sprintf() if snprintf() is available, even in cases where the
  output size is bounded, to avoid deprecation warnings on macOS.  For 1.4.x
  we still fall back to sprintf() to avoid a point release breaking support
  for any platform still lacking snprintf().

* Stop linking with --enable-runtime-pseudo-reloc.  We were requiring this for
  cygwin and mingw, but from the documentation it should only be needed for a
  library which exports data symbols, which we don't do, and the build works
  without it.

* Use `override` for subclassing functors.  This is good practice as it gives a
  clear compile error if we have to change the signature of an virtual method
  on such a functor.  See #830.

* Avoid redefining MSVC-specific macros if they are already defined.  This
  avoids an MSVC warning and potential for the code to be ill-formed if the
  user defines these macros with a value other than 1.  Patch from A. Jiang
  (https://github.com/xapian/xapian/pull/334).

Log message:
xapian: update to 1.4.25. Changes:

API:

* MSet::get_eset(): Don't fetch the collection frequency for each term unless
  we're using the Bo1EWeight expansion scheme which actually needs it.  In a
  simple test this reduced the time taken to do a search and generate expand
  terms by a third.  Partly addresses #264.

* QueryParser::parse_query(): Fix parse error when using FLAG_CJK_NGRAM (aka
  FLAG_NGRAMS) with a query string which has non-CJK followed by whitespace,
  CJK, and more non-CJK.  Patch from Robert Stepanek
  (https://github.com/xapian/xapian/pulls/331).

testsuite:

* unittest: Improve sparse file detection by using SEEK_HOLE, which is
  specified by POSIX and seems to be widely supported.  On platforms without it
  or on an FS with a > 128K block size we will skip the tests involving a 4GB
  file, but that's acceptable.  On ZFS st_blocks reports the number of blocks
  after compression and also lags behind when data has only been committed to
  the journal, which means our previous check based on st_blocks couldn't be
  made to work without potentially falsely detecting sparse file support.
  Fixes #823, reported by someplaceguy.

* apitest: Enable adddoc2 and adddoc5 testcases for sharded databases.  We
  now just skip the TermIterator::get_termfreq() checks in this case.

glass backend:

* Check Btree level value from disk is in range, which avoids potential out of
  range access on corrupt database.  Fixes #824, reported by group13.

* Reject invalid blocksize read from corrupted version file.  Throw
  DatabaseCorruptError if value is out of range or not a power of two.

* Optimise allterms iteration.  Most terms don't contain any zero bytes, and
  for such terms the key for the first chunk in the termlist table is just the
  termname so no decoding is needed when advancing the iterator.  This optimisation
  is 8.4% faster in a simple test of iterating allterms via xapian-delve.

* Compaction of an empty non-optional table now gives an empty output, whereas
  previous it was one block in size (8K by default).  This isn't important in
  general as the non-optional tables are not likely to be empty in a real
  database, but it's helpful for making small test database and it seems weird
  that compaction would make a database much larger in percentage terms in this
  edge case.

chert backend:

* Check Btree level value from disk is in range, which avoids potential out of
  range access on corrupt database.  Fixes #824, reported by group13.

build system:

* configure: DragonflyBSD automatically pulls in library dependencies, so set
  link_all_deplibs_CXX=no there.

documentation:

* Document allterms_begin() and termlist_begin() iteration order.  Thanks to
  Eric Wong for querying this.

* Document TermIterator::get_termfreq() quirk.  In the case of a TermIterator
  from termlist_begin() on a Document from a sharded database, you get term
  frequencies from just the shard.  Fixes #423

portability:

* Support building on platforms without AI_NUMERICSERV (e.g. macOS 10.5).
  Patch from Sergey Fedorov.

Log message:
Update to 1.4.23. From the changelog:

API:

* Add `FLAG_NGRAMS` as a preferred new alias for `FLAG_CJK_NGRAM` and
  `SNIPPET_NGRAMS` as a preferred new alias for `SNIPPET_CJK_NGRAM`.
  In the next release series these features have been expanded to cover many
  more languages so the "CJK" in the name has become inaccurate as it \ 
stands
  for "Chinese, Japanese and Korean").

* Database::check(): Improve the exception message for an empty filename from
  "Failed to rewind file descriptor -1 (Bad file descriptor)" to \ 
"Couldn't find
  Xapian database or table to check (No such file or directory)".

testsuite:

* The test harness now supports generated databases for all backends and
  all testcases which can use generated databases now do, which together
  improve test coverage as we now run many testcases under more backends.
  Generated databases can also be cached, which helps speed up the test
  suite.

* Remove some testcases which are now redundant with other testcases or
  no longer useful for other reasons.

* Add regression test for #781, reported by Germán M. Bravo.  This bug was
  fixed 4 years in 1.4.12 but a clean regression test only became possible
  thanks to changes in this release.

* The test harness would attempt to throw a Xapian::Database (rather than
  Xapian::DatabaseError) if it failed to rename a temporary database to its
  final name.  This would fail because there wasn't a database called "rename
  failed".  Reported by stark3y on #xapian.

* apitest: Fix exceed32bitcombineddb1 testcase (which only runs with
  --enable-64bit-docid).

* apitest: Fix nomoredocids1 when configured with --enable-64bit-docid.

* Add testcase for removal of positions for replaced doc (regression test for
  bug in git master not present in 1.4.x).

* unittest: Unit test block file functions

* Stop using std::endl in tests since this seems to be C++ best practice
  as it causes a flush of the stream, which is rarely actually wanted.  Also
  often the replacement \n can be combined with a string literal.

* Use Xapian::docid for document ids in testcases.  Using `int` or `size_t`
  works, but is less correct and can trigger compiler warnings because not all
  values are representable.

matcher:

* BM25PlusWeight: With some parameter combinations we were requesting stats we
  didn't actually need.  Confirmed with Vivek Pal on IRC.

glass backend:

* Avoid unnecessary copying of data when adding an item to a B-tree table which
  should speed up indexing a bit.

* xapian-check: If the first chunk for a term is a continuation chunk, the
  affected term was missing from the error message.

* xapian-check: We now report the document id as context in many more error
  messages.

chert backend:

* Fix wdf upper bounds used when search a modified chert-format
  WritableDatabase object.  A bug introduced in 1.4.19 meant we were using the
  wdf upper bounds from the last committed version of the database.  This bug
  could cause assertion failures when configured with --enable-assertions.

remote backend:

* Fix thread-safety issue starting remote prog server.  Previously we did some
  memory allocation in the child process after fork() but before we exec-ed the
  specified program, but in a multi-threaded program (which libxapian might be
  used in) it's only safe to call async-signal-safe functions in the child
  process after fork() until exec, and malloc, etc aren't async-signal-safe.

* If we failed to open /dev/null in the child process while starting a remote
  prog server, we would try to throw an exception.  That's not going to work
  well so now we just call `_exit(-1)`.

inmemory backend:

* Throw exception on docid overflow like we do in other backends.

build system:

* configure: When probing for socklen_t or equivalent include the same headers
  as we do in the code to reduce the risk of configure deciding to use
  socklen_t but it not getting defined in the code.

* configure: When probing if a particular compiler or linker option is
  supported, we were relying on the compiler exit status but some compilers
  only warn about unknown options.  We now inspect the compiler's stderr
  output to try to detect such cases.

* configure: Avoid compiler warning during GCC version check when compiler
  needs an option to enable C++11 support.

* Avoid running pwd since the directories we need are available in automake
  variables.

documentation:

* Suggest protocol buffers for structured document data.  Fixes #53.

* Clarify documentation for release() methods.

* Fix typo "shared database" to "sharded database" in API docs.

* Document that transactions aren't atomic across shards.

tools:

* xapian-progsrv,xapian-tcpsrv: Support multiple DBs with --writable.

* xapian-tcpsrv: Fix default timeouts in --help output which have been reported
  as the name of the constant rather than its value since 1.3.3.

* xapian-tcpsrv: When --one-shot is specified, don't do the usual test open of
  the specified database(s) as it doesn't really seem useful in this case.  The
  test harness uses --one-shot so this reduces overhead when running remote
  tests.

* Stop using std::endl in tools since this seems to be C++ best practice
  as it causes a flush of the stream, which is rarely actually wanted.  Also
  often the replacement \n can be combined with a string literal.

portability:

* Don't pass mode to open when it's unused as this triggers "missing O_CREAT or
  O_TMPFILE?" warnings when compiling for Android.

* Stop using INFINITY macro.  If the implementation supports floating point
  infinities then HUGE_VAL gives us infinity as a double directly.  If not,
  then it's the maximum finite value of a double.

* Don't auto-enable _FORTIFY_SOURCE on mingw or mingw-w64.  Enabling
  _FORTIFY_SOURCE on newer mingw-w64 requires linking with -lssp so we
  attempted to stop automatically enabling it there in 1.4.19 but this fix
  didn't actually work.  Trying to get this to work automatically has proved
  difficult and I couldn't find evidence that _FORTIFY_SOURCE was actually
  supported on mingw.  If it is, enabling manually will still work.

* Fix mingw32 build with C++17 compiler, which ends up with std::byte
  conflicting with byte typedefs in system headers, due to us having `using
  namespace std;` in some of our internal headers.  Switch the ones which
  are causing problems to more targetted `using std::string;`, etc instead.

* Fix WIN32 build with --disable-backend-remote which was failing because we
  were always trying to build common/socket_utils.cc which fails because
  SOCKLEN_T hasn't been probed.  Fixes #821, reported by mgautier.

* soaktest: Use C++11 <random> which is more portable than random().

* In WIN32 builds, pass the program pathname separately to CreateProcess()
  which is apparently more robust if the program pathname contains spaces.

* Stop trying to set Microsoft-specific SO_EXCLUSIVEADDRUSE option on our
  listening sockets.  It's not possible to set both SO_REUSEADDR and
  SO_EXCLUSIVEADDRUSE so this call will always fail with WSAEINVAL, but we
  were ignoring these errors because SO_EXCLUSIVEADDRUSE required admin
  privileges in older OS versions.

* Suppress MSVC deprecation warning for GetVersionEx since none of the
  suggested replacements seems to actually provide the functionality we are
  using from it.

* Fix some warnings from MSVC in the fallback code for overflow-checked
  arithmetic.

* Support Enquire::set_time_limit() on GNU Hurd since Hurd now implements
  timer_create().

* Remove lingering traces of IRIX support as it's been dead for many years.

debug code:

* Fix build failure with --enable-assertions due to incorrect variable name in
  assertion.  Patch from Alexei Kharchev in
  https://github.com/xapian/xapian/pull/327.

* Fix GCC warning with --enable-log.

* Fix debug logging for 3 GlassDatabase methods to log their parameter.

* Add a mechanism to support debug logging templated return types containing
  commas.  This was added to fix a build failure on master with --enable-log
  reported by ttyS3.

Bindings:

Documentation:

* Where we document how to build the bindings, link to the download page so
  the reader can easily find the source to download.  Reported by John G. Heim.

* Update lists of debian packages to install for bindings development to
  show be for the latest stable release and the one before (bookworm and
  bullseye).

* HACKING: Fix typos in deprecation warning section

* HACKING: Update link to Pike bindings

* HACKING: Document issues caused by macOS SIP.  Fixes #732.

General:

* Add `FLAG_NGRAMS` as a preferred new alias for `FLAG_CJK_NGRAM` and
  `SNIPPET_NGRAMS` as a preferred new alias for `SNIPPET_CJK_NGRAM`.
  In the next release series these features have been expanded to cover many
  more languages so the "CJK" in the name has become inaccurate as it \ 
stands
  for "Chinese, Japanese and Korean").

* Avoid running pwd in makefiles since the directories we need are available in
  automake variables.

CSharp:

* SmokeTest.cs: Fix some incorrect messages if testcases fail.

* Smoketest.cs: Test stemmer description.

Lua:

* Update docs for supported Lua versions.

Perl:

* Generate HTML docs for each class.

* Document all QueryParser FLAG_* constants.

* Update one place which didn't correctly indicate the current status of the
  Perl bindings.  Fixes #523.

Python:

* Fix bug in wrapper for xapian.Query when constructing OP_WILDCARD queries.
  A workaround for this bug which works with existing releases is to pass
  0 for the third and fourth optional parameters like so:

  xapian.Query(xapian.Query.OP_WILDCARD, "xyz", 0, 0)

Python3:

* Fix bug in wrapper for xapian.Query when constructing OP_WILDCARD queries.
  A workaround for this bug which works with existing releases is to pass
  0 for the third and fourth optional parameters like so:

  xapian.Query(xapian.Query.OP_WILDCARD, "xyz", 0, 0)

Ruby:

* Improve documentation about __call__ methods.

Log message:
xapian: remove comment about csharp-xapian

Log message:
xapian: note that updating the version WILL break csharp-xapian

Log message:
Update to 1.4.21. From the changelog:

glass backend:

* Avoid throwing InvalidArgument when searching for overlong terms in some
  cases.  Such terms can't match, but are valid to query for.  Patch from
  Robert Stepanek in https://github.com/xapian/xapian/pull/313.

chert backend:

* Avoid throwing InvalidArgument when searching for overlong terms in some
  cases.  Such terms can't match, but are valid to query for.

build system:

* Clean up cygwin and mingw configure checks.  When we check $host_os, always
  anchor at the start (e.g. cygwin* not *cygwin*), and check for msys* as well
  since that's a derivative of cygwin and behaves similarly for the things
  we're checking here.

* Update to use AX_CXX_COMPILE_STDCXX which is a replacement for
  AX_CXX_COMPILE_STDCXX_11 (which we were using) which also supports newer C++
  standards versions which will be useful.  For C++11 the only difference seems
  to be that the macro now checks for attribute support - we use C++11
  attributes so that seems a good thing.

documentation:

* INSTALL: Restructure MSVC section for clarity.

* INSTALL: Mention -D_FORTIFY_SOURCE=3 too (fairly new - requires GCC 12).

* Consistently say "macOS" not "Mac OS X", "OS X", etc.

* admin_notes.rst:

  + Update locking section to cover Open File Description locks

  + Add some discussion of block sizes (fixes #819, reported by mgautier)

  + Fix cut-and-paste error - we were suggesting that the docdata table only
    exists if there's positional data.  Noted by Gaurav Arora.

  + Improve markup

examples:

* Stop using std::endl in examples since this seems to be C++ best practice
  as it causes a flush of the stream, which is rarely actually wanted.  Also
  often the replacement \n can be combined with a string literal.

portability:

* Stop trying to check for incompatible C++ ABI between the compiler used to
  build xapian-core and the compiler used to build code using xapian-core.

  This check was helpful in the GCC 3 days, but ABI versions 2 and up are
  compatible aside from obscure corner cases, and GCC now defaults to using the
  latest ABI version it supports.  The result is that this check is no longer
  useful enough to justify the noise.

  We still check for incompatible _GLIBCXX_DEBUG between the library and
  application builds, since that will cause things not to work, and the normal
  error message doesn't make it clear what's wrong.

  Reported by David Bremner.

* Fix new warnings from GCC 12.

* Avoid undefined value use when unpacking a key in a corrupted glass docdata
  table.  We now skip further checks on the entry in this case.

* Merge allocations in MSVC directory reading compatibility code so we can
  allocate in a single malloc() call.

* Add accept() wrapper which checks an assumption that Microsoft's SOCKET type
  only actually holds 32 bit values even in 64 bit platforms and throws an
  exception if violated.

* Eliminate a use of sprintf.

* Squash some unhelpful MSVC deprecation warnings.

* Declare dummy invalid parameter handler noexcept to fix a warning from MSVC.

* Include <stdlib.h> in configure check for sys_errlist as that's where it is
  with mingw and MSVC.

debug code:

* Fix debug logging for Xapian::Internal::intrusive_ptr<const T>.

  GCC 12.2 warned about infinite recursion in the old version, and this seems
  to be the case.  This code has been there a long time, but is only used when
  debug logging is enabled.  It does seem to be used, so presumably nobody's
  tried to log something which triggered it.

Updating during the freeze for the bug and portability fixes.