MonetDB Nov2009-SP2 release

Release "Nov2009-SP2" consists of the following updated components

Name	Version	CVS module	CVS branch	Note/Comment
MonetDB Common	1.34.4	MonetDB	Nov2009
MonetDB Clients	1.34.4	clients	Nov2009
MonetDB4 Server	4.34.4	MonetDB4	Nov2009
MonetDB4/XQuery	0.34.4	pathfinder	Nov2009
MonetDB5 Server	5.16.4	MonetDB5	Nov2009
MonetDB5/SQL	2.34.4	sql	Nov2009
MonetDB5/SQL/GIS	0.14.4	geom	Nov2009
MonetDB Java	1.34.4	java	Nov2009
MonetDB Testing	1.34.4	testing	Nov2009	(developers only)

NOTE:
Before upgrading to the latest "Nov2009-SP2" release from a release of MonetDB older than the Nov2008 release, please make a dump (SQL, XQuery) of your database, and rename/move/backup the database directories (dbfarm, sql_logs, xquery_logs). After the upgrade, restore (SQL, XQuery) your database, again. Once you have checked and confirmed the correctness of the restore, you can delete the above renamed/moved/backed-up old database directories.

When upgrading from a release of MonetDB older than the May2009 release, you need to move your write-ahead log following the "recommended" instructions for SQL and/or XQuery.

Contents

• Nov 2009 SP2 bug-fix release
• Nov 2009 SP1 bug-fix release
• Nov 2009 feature release
  • Packaging
  • MonetDB Common
  • MonetDB Server 5
  • SQL
  • XQuery
  • Client Packages
  • Bug fixes

Nov 2009 SP2 bug-fix release

This is the second bug fix release of the Nov2009 version. This version can be installed "on top of" the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• No need to require bash for running the *-config scripts. (SF bug #2914563)
• MonetDB5 development package now contains all development include files. (SF RFE #2929558)
• Added support for compiling on OpenBSD (contributed, i.e. not tested by us).
• General code improvements, partly inspired by running Coverity Scan.
• mclient -D will exit with a non-zero exit code if dumping failed. (SF bug #2925674)
• The Python interface now uses the same socket options as the C version, giving a boost in performance. (SF bug #2925750)
• Fixed a bug in the Python interface to deal with fields with embedded newline characters. (SF bug #2917219)
• Added initial support to the Python client interface for specifying the encoding.
• Fixed a bug in execute argument parsing in Python interface.
• Added missing .sql files to the Windows installer.
• In MonetDB/SQL, fixed bug in not(invalidtable). (SF bug #2927174)

The following bug tracker items were fixed since the Nov2009-SP1 release:

• [ 2911008 ] Cannot start because lib_replication.dll was not found
• [ 2914558 ] Modify MonetDB.m4 to support OpenBSD
• [ 2917219 ] Python client doesn't handle fields with newline characters
• [ 2925674 ] mclient dump fails on UTF-8 char and gives no error
• [ 2925750 ] setsockopt in Python client greatly improves performace
• [ 2927174 ] SQL: not(invalidtable.bla) segfaults

Nov 2009 SP1 bug-fix release

This is a bug fix release of the Nov2009 version. This version can be installed "on top of" the Nov2009 release. The database format is fully compatible, so there is no need to do a dump and restore. (Although it is *always* a good idea to make backups.)

Some of the bugs that were fixed:

• The Windows installers for MonetDB/SQL in the Nov2009 release were incomplete. The missing files have been added.
• Fixed performance issue with loading complex schemas in readonly mode in MonetDB/SQL.
• Fixed performance issue with grouping in MonetDB/SQL when one of the columns is sorted.
• Fixed problems with LIKE joins in MonetDB/SQL.
• Load all files in a multi-file COPY INTO query.
• Fix error reporting of unexpected input in XQuery queries to be Unicode aware.
• Correctly handle circular dependencies in FD property check. (SF bug #2908615.)
• Fixes throughout the code base to better cope with unexpected situations (such as memory shortage).

The following bug tracker items were fixed since the Nov2009 release:

• [ 1681244 ] Mapi ports and patfhinder/pftijah
• [ 1893448 ] PF: xrpcConvert2 fails on Windows
• [ 2686008 ] SQL: multi-attribute GROUP BY may ignore sortedness
• [ 2879008 ] SQL: WHERE A.x LIKE A.y misses results
• [ 2879011 ] SQL: leftjoin undefined
• [ 2884050 ] Broken link to VOC tutorial
• [ 2902320 ] SQL: COPY multiple files
• [ 2903111 ] Wrong error / no check for new database dir
• [ 2908615 ] Not clause breaks result

Nov 2009 feature release

Ubuntu

While the Ubuntu version of the Aug2009 release was built for Ubuntu 9.04, the Jaunty Jackalope, the Ubuntu version of the Nov2009 release has now been built for Ubuntu 9.10, the Karmic Koala.

Fedora

In addition to Fedora 10 & 11, we now also provide RPMs for Fedora 12.

Python

Packaging of some Python support files which are only used for testing was changed. The MonetDB-python RPM has been removed, the files are now packaged in the MonetDB-testing RPM. Equally, the python-monetdb Debian/Ubuntu package has been removed.

Windows

On Windows, the bzip2 library is now also used. The server (e.g., using SQL's COPY INTO) can read files that are compressed using gzip and bzip2.

The Coverity service has been used to uncover potential errors, such as dereference through NULL pointers and leakage. The warnings produced in the first run have been dealt with for >90%.

• The M5 code underwent code hardening to better deal with rare events that would materialize mostly on 32bit systems. Mostly minor changes, e.g. display of debugging output has been improved to be more informative.

• There is a new implementation of the SQL COPY INTO statement which uses multiple CPUs/cores when available. This version is a bit stricter when parsing the CSV file than the old version. E.g., when strings are quoted in the CSV file, the string delimiter must be specified in the COPY INTO statement, and to let empty field be treated as NULL values, the COPY INTO statement must specify NULL AS ''.
• The TRACE statement modifier has been changed to collect the events in BATs first and returning them after the result set. The support library has been adjusted accordingly. See the documentation: SQL runtime features.
• Several experimental optimizer pipelines are researched and being worked upon. They are not (yet) meant as the default pipeline for production applications. See the documentation: SQL runtime features.
• Changing usernames is now possible from SQL, including the monetdb adminstrator user. Feature Request #2830257. The SQL syntax is: ALTER USER "monetdb" RENAME TO "newname";.

Merovingian

The following features were implemented in the merovingian and monetdb programs:

• monetdb: Created databases are now always put under maintenance and as such they have to be explicitly released. This allows the DBA to initialise the new database without any client interference.
• merovingian: Neighbour discovery now no longer announces databases under maintenance, such that they are not shown as being available to peers. This allows to pull a database out of the cluster easily.
• monetdb: No operations can be performed with monetdb any more without a merovingian instance running.
• merovingian: Implemented Feature Request #2830754. monetdb can now control a remote merovingian using the -h -p and -P global options to monetdb. The password used is stored at the merovingian site in the .merovingian_pass file in the controlled dbfarm. An example of how this can be used can be found in the Assets section under the VOC loader script.
• merovingian: Remote controllable merovingians now announce their controlport to peers, which are marked with an asterisk in the output of monetdb discover.
• monetdb: Implemented Feature Request #2855846. The global -q option to monetdb makes it suppress any status messages, causing only error messages to be printed when they occurr.
• merovingian: The outside URI that Merovingian is available on is now announced to each mserver5 started by merovingian via the merovingian_uri GDK environment variable.
• monetdb: For each database now the property optpipe can be set which allows per database control of the SQL optimiser pipeline. Valid values can be found in the monetdb5.conf configuration file under the sql_optimizer section.
• monetdb: Fixed bug where the uptime of databases with an uptime longer than 7 days was reported incorrectly.

• Use of value indices on comparisions between a value and an XPath ending in text() or attribute access (e.g., doc(x)//x/y[z/text() = "foo"] and for $x in doc//x/y where $x/@z = "bar" return $x); such queries do not need scanning data, but start at the index and navigate backwards from there.

  NOTE: This required a change in the physical database storage. While the Nov2009 release of MonetDB/XQuery can still read older (pre-Nov2009) databases, it is recommended to re-shred all documents to benefit from the new indices and achieve optimal performance.

• A session expression cache, where you can run multiple queries in the same session and within that session keep intermediate results present (somewhat like a database cursor). One use is quickly paging in a GUI through a large result, another is acceleration of series of subsequent queries that overlap (e.g. zoom in to a result set). See: Session Expression Cache

mclient

• Result rendering has been changed to try harder to find an optimal rendering of results wider than the terminal allows. Unlike previous releases, mclient now emits results wider than the terminal width if it is impossible or unreasonable to fit the results, thereby aiming to avoid wrapping columns.
• EXPLAIN, TRACE and PROFILE output are now correctly rendered.
• Timer support for single statements was implemented, Bug #2872884.
• Added support for .monetdb file with defaults for mclient. The DOTMONETDBFILE environment file can be used to point to a different location or file name. The file layout of a .monetdb file is a key=value file, where keys include user, password and language.
• Data fed into mclient using a pipe is no longer read line by line, but per block, like files given as arguments are. This improves performance considerably, but changes the behaviour regarding error messages and understood commands. The internal \x commands of mclient are no longer respected when used with a pipe. To obtain the previous behaviour, add the -i option to mclient when using data fed from a pipe.
• The default username (the name of the currently logged in user) is no longer used without asking. The -u option now always requires an argument, when this option is absent, mclient will ask for a username, unless a default was set in the .monetdb file. Note: mclient no longer logs in as monetdb/monetdb by default.
• The start_with_help option was removed. --help now always gives the general help of mclient and exits afterwards. Instead, mclient now gives a welcome message with some diagnostic information and a hint how to get (language specific) help.
• The \d command now describes all tables by reporting whether it is a TABLE or VIEW. Rendering mode of database object describe mode has been improved.
• Autocommit status and changes therein are now reported by mclient. Empty responses are no longer reported as "0 tuples", instead proper status messages are returned, giving e.g. information about a generated identity key.
• Error messages no longer include the leading !.

Python & Ruby

• The native implementations have been matured further.

PHP

• The SWIG-based PHP interface (leveraging the C MAPI library) has been replaced by a native PHP implementation. For the time being, the old SWIG-based interface is still in the code base, but only the new native interface is activated and distributed in the installers.

JDBC

• JDBC received a workaround for a longstanding problem when running the driver using Java 6.
• A bug was fixed which caused Connection warning messages to disappear when a connection to Merovingian was made.

The following bug tracker items were fixed since the previous "Aug 2009 SP2" release:

• ID: 2903111: Wrong error / no check for new database dir
• ID: 2902320: SQL: COPY multiple files
• ID: 2898944: pf takes 'forever' (> 1 hour) to compile particular query
• ID: 2898378: M4: tests/by_Peter fail
• ID: 2897126: pf fatal error: duplicate column in equi-join
• ID: 2896012: Filter using same field in OR and AND throws typeexception
• ID: 2895791: ORDER BY with Row_number() return wrong results
• ID: 2895290: M5: segfault for tst019 when run with mclient
• ID: 2894927: Mx: @include does not write any "#line" directives
• ID: 2894150: test fail due to error message format/protocol changes
• ID: 2893878: SQL: more tests fail with Nov2009 that worked with Aug2009
• ID: 2893875: PF: 82 tests fail on Auf2009_NFI/Nov2009 but work on Aug2009
• ID: 2893859: SQL: various VIA-M tests fail
• ID: 2893846: SQL: set_history_and_drop_table.SF-2607045 fails
• ID: 2893823: SQL: src/backends/monet5/Tests/optimizers.sql fails
• ID: 2893821: SQL: src/test//Tests/median_stdev.sql fails
• ID: 2893815: PF: xrpcConvert tests segfault since 2009/10/08
• ID: 2893484: All CASE branches are evaluated
• ID: 2893247: SQL: QOTupdateStatistics segfaults
• ID: 2891718: M5/MAL: io.print() yields no output
• ID: 2891191: unconscious redirect
• ID: 2890914: StartOfHeaderParser.java#getNextAsString() Err: IndexOutOfB
• ID: 2890702: M5: pma.mx fails to compile on various platforms
• ID: 2890035: SQL: optimizer confused by concurrency
• ID: 2887282: union bug
• ID: 2879011: SQL: leftjoin undefined
• ID: 2879008: SQL: WHERE A.x LIKE A.y misses results
• ID: 2879005: SQL: duplicate column name
• ID: 2878994: SQL: WHERE A.x LIKE B.x does not work
• ID: 2877044: PF: binding sequence in a for clause has static type `empty'
• ID: 2876948: PF: error "We do not support the value NaN"
• ID: 2874045: SQL: segv for unknown identifier in subquery
• ID: 2873564: Cannot create unique constraint on char column
• ID: 2872884: mclient -t -s... does not show "Timer" info
• ID: 2864179: function call not handled
• ID: 2863458: PF/TIJAH: tokenize in a for-loop
• ID: 2837050: SQL: src/test/xquery/Tests/q01.sql fails
• ID: 2834654: PHP+XQUERY: multiple results not returned correctly
• ID: 2817414: XQ: MonetDB allocates >5GB mem while querying small document
• ID: 2791361: SQL: Division by zero error
• ID: 2791356: SQL: Floating point exception
• ID: 2771052: M5/SQL: parallel bulk-load (copy into) incorrect
• ID: 2716723: PF: child-steps + replace() fails to compile
• ID: 2686008: SQL: multi-attribute GROUP BY may ignore sortedness
• ID: 2627137: PF: tests fail with BATpropcheck error
• ID: 2211565: PF: type disjointness incorrect
• ID: 2015135: PF: pf:mil() can't deal with nil values in MIL results
• ID: 1991738: XQ: compilation or runtime choice
• ID: 1981735: PF/alg: some error messages differ from PF/mps
• ID: 1981735: PF/alg: some error messages differ from PF/mps
• ID: 1976341: XQ: leftovers after deleting document
• ID: 1964365: PF/Alg: insert..into inserts in wrong order
• ID: 1959269: XQ+JDBC: protocol violation
• ID: 1911209: XQ: MonetDB 'hangs' after requesting non-existing doc
• ID: 1893448: PF: xrpcConvert2 fails on Windows
• ID: 1809586: PF: busy port crashes Mserver
• ID: 1747068: XQ: windows de-install should warn and empty bat/ dirs
• ID: 1739353: EXPLAIN support from JDBC
• ID: 1730547: XQ: Mserver crashes on concatenated query
• ID: 1726599: XQuery: collection management broken
• ID: 1681244: Mapi ports and patfhinder/pftijah
• ID: 1637867: XQ: seqcast int? to numeric unsupported
• ID: 1607210: XQ: server-side compilation crash (member benchmark)
• ID: 1595129: XQ: property error on 64-bit Darwin - big-endian related?
• ID: 987304: SQL: div by zero results in ascii art
• ID: 921173: SQL: extremely long and complex query causes SEGFAULT

The detailed list of changes can be found in the source code CVS logs.